2024-09-21T17:51:32.334 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T17:51:32.358 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T17:51:32.449 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219 branch: wip-rf-qprio-w-qa description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/cache} email: null first_in_suite: false job_id: '7914219' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-rf-qprio-w-qa ceph: conf: global: mon election default strategy: 1 ms bind msgr2: false ms type: async osd debug inject dispatch delay duration: 0.1 osd debug inject dispatch delay probability: 0.1 osd_async_recovery_min_cost: 1 osd_object_clean_region_max_num_intervals: 1000 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd beacon report interval: 30 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd inject bad map crc probability: 0.1 osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd max scrubs: 5 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd_max_pg_log_entries: 3000 osd_min_pg_log_entries: 3000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - failed to encode map - but it is still running - objects unfound and apparently lost - osd_map_cache_size - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - must scrub before tier agent can activate - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: c69a6b39fef6dd04706886608793361ef510a986 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c69a6b39fef6dd04706886608793361ef510a986 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rf-qprio-w-qa sha1: c69a6b39fef6dd04706886608793361ef510a986 owner: scheduled_rfriedma@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.4 - osd.8 - osd.12 - - mon.b - osd.1 - osd.5 - osd.9 - osd.13 - - mon.c - osd.2 - osd.6 - osd.10 - osd.14 - - mgr.x - osd.3 - osd.7 - osd.11 - osd.15 - client.0 seed: 9109 sha1: c69a6b39fef6dd04706886608793361ef510a986 sleep_before_teardown: 0 subset: 3/444 suite: rados suite_branch: wip-rf-qprio-w-qa suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c69a6b39fef6dd04706886608793361ef510a986/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c69a6b39fef6dd04706886608793361ef510a986 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDfGfx3GUkiesxylWtdDqhksjAWAN584U+jEQ+xyMJMNmqLTC5GVWGmS2rzo1qEUkvlZJbsQS6s5o+OZY9nrdL4= smithi040.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDeKb+ps/cSLHfthzbjB9j+EEoGZXGJunEMA8nMsHCDjWV2k4tvyLn8r28ZASRK87Y2dyAoXMyQWLhgJFP7i2Ao= smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOqXSJyMZSCAiEgvt5zfxAeLr3tvqXmfCKl35I1JIElFTvkBDMI86gFiTxvrl3NkF8k6z+9bkyjH3tUd2jiXxkg= smithi160.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwo1yNojRWJw3dt7SfwCf53/m0v1TwVITZWQ8uldbKo3Bei0al/m6+12EfZ8jJEkNqOYqnMrjMKps/FTKPnaGU= tasks: - install: null - ceph: conf: osd: debug monc: 20 - exec: mon.a: - ceph config set mgr mgr/balancer/log_level debug - ceph balancer status - ceph osd set-require-min-compat-client reef - ceph balancer mode upmap-read - ceph balancer on - ceph balancer status - ceph balancer status detail - thrashosds: chance_pgnum_grow: 0.25 chance_pgnum_shrink: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 map_discontinuity_sleep_time: 200 min_in: 2 thrash_hosts: true timeout: 1800 - exec: client.0: - sudo ceph osd pool create base 4 - sudo ceph osd pool application enable base rados - sudo ceph osd pool create cache 4 - sudo ceph osd tier add base cache - sudo ceph osd tier cache-mode cache writeback - sudo ceph osd tier set-overlay base cache - sudo ceph osd pool set cache hit_set_type bloom - sudo ceph osd pool set cache hit_set_count 8 - sudo ceph osd pool set cache hit_set_period 3600 - sudo ceph osd pool set cache min_read_recency_for_promote 0 - sudo ceph osd pool set cache min_write_recency_for_promote 0 - rados: clients: - client.0 objects: 500 op_weights: cache_evict: 50 cache_flush: 50 cache_try_flush: 50 copy_from: 50 delete: 50 read: 100 write: 100 ops: 4000 pools: - base teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_17:35:20 tube: smithi user: rfriedma verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T17:51:32.449 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c69a6b39fef6dd04706886608793361ef510a986/qa; will attempt to use it 2024-09-21T17:51:32.450 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c69a6b39fef6dd04706886608793361ef510a986/qa/tasks 2024-09-21T17:51:32.450 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T17:51:32.452 INFO:teuthology.task.internal:Checking packages... 2024-09-21T17:51:32.473 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c69a6b39fef6dd04706886608793361ef510a986' 2024-09-21T17:51:32.474 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T17:51:32.474 INFO:teuthology.packaging:ref: None 2024-09-21T17:51:32.474 INFO:teuthology.packaging:tag: None 2024-09-21T17:51:32.474 INFO:teuthology.packaging:branch: wip-rf-qprio-w-qa 2024-09-21T17:51:32.474 INFO:teuthology.packaging:sha1: c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T17:51:32.474 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-rf-qprio-w-qa 2024-09-21T17:51:32.618 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T17:51:32.620 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T17:51:32.633 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T17:51:32.633 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T17:51:32.709 INFO:teuthology.task.internal:Saving configuration 2024-09-21T17:51:32.726 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T17:51:32.751 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T17:51:32.775 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 17:44:50.375656', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINtpwvrR9d2KMHXfn9mDOIQuTxbimNW2nizWlFlIYCcH'} 2024-09-21T17:51:32.801 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi040.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 17:44:50.370455', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIsdKAK7+PE43LXD42atXOb590GHhc75uiL1Yv0bp/37'} 2024-09-21T17:51:32.822 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 17:44:50.374130', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOIfVHT+9/1rd2z2nCfuqIFqJtrOAePdekextFqxZhOJ'} 2024-09-21T17:51:32.840 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi160.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 17:44:50.372418', 'locked_by': 'scheduled_rfriedma@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINVPub4XjSpJTCIiPCJDht5qJ7makYDbkTaH3Um6rTK4'} 2024-09-21T17:51:32.841 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T17:51:32.847 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'] 2024-09-21T17:51:32.847 INFO:teuthology.task.internal:roles: ubuntu@smithi040.front.sepia.ceph.com - ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'] 2024-09-21T17:51:32.847 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'] 2024-09-21T17:51:32.847 INFO:teuthology.task.internal:roles: ubuntu@smithi160.front.sepia.ceph.com - ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0'] 2024-09-21T17:51:32.847 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T17:51:32.967 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd13acc8160>, signals=[15]) 2024-09-21T17:51:32.967 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T17:51:32.973 INFO:teuthology.task.internal:Opening connections... 2024-09-21T17:51:32.973 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T17:51:32.974 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:51:33.055 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T17:51:33.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:51:33.133 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T17:51:33.133 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:51:33.209 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T17:51:33.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:51:33.287 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T17:51:33.294 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-09-21T17:51:33.300 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-09-21T17:51:33.301 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-09-21T17:51:33.347 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T17:51:33.347 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="Ubuntu" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="22.04" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_CODENAME=jammy 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:ID=ubuntu 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE=debian 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T17:51:33.348 INFO:teuthology.orchestra.run.smithi029.stdout:UBUNTU_CODENAME=jammy 2024-09-21T17:51:33.349 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-09-21T17:51:33.369 DEBUG:teuthology.orchestra.run.smithi040:> uname -m 2024-09-21T17:51:33.375 INFO:teuthology.orchestra.run.smithi040.stdout:x86_64 2024-09-21T17:51:33.375 DEBUG:teuthology.orchestra.run.smithi040:> cat /etc/os-release 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:NAME="Ubuntu" 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_ID="22.04" 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:VERSION_CODENAME=jammy 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:ID=ubuntu 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:ID_LIKE=debian 2024-09-21T17:51:33.422 INFO:teuthology.orchestra.run.smithi040.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T17:51:33.423 INFO:teuthology.orchestra.run.smithi040.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T17:51:33.423 INFO:teuthology.orchestra.run.smithi040.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T17:51:33.423 INFO:teuthology.orchestra.run.smithi040.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T17:51:33.423 INFO:teuthology.orchestra.run.smithi040.stdout:UBUNTU_CODENAME=jammy 2024-09-21T17:51:33.423 INFO:teuthology.lock.ops:Updating smithi040.front.sepia.ceph.com on lock server 2024-09-21T17:51:33.444 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-09-21T17:51:33.449 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-09-21T17:51:33.450 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-09-21T17:51:33.495 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="Ubuntu" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="22.04" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_CODENAME=jammy 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:ID=ubuntu 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE=debian 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T17:51:33.496 INFO:teuthology.orchestra.run.smithi070.stdout:UBUNTU_CODENAME=jammy 2024-09-21T17:51:33.496 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-09-21T17:51:33.519 DEBUG:teuthology.orchestra.run.smithi160:> uname -m 2024-09-21T17:51:33.524 INFO:teuthology.orchestra.run.smithi160.stdout:x86_64 2024-09-21T17:51:33.525 DEBUG:teuthology.orchestra.run.smithi160:> cat /etc/os-release 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:NAME="Ubuntu" 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_ID="22.04" 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:VERSION_CODENAME=jammy 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:ID=ubuntu 2024-09-21T17:51:33.572 INFO:teuthology.orchestra.run.smithi160.stdout:ID_LIKE=debian 2024-09-21T17:51:33.573 INFO:teuthology.orchestra.run.smithi160.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T17:51:33.573 INFO:teuthology.orchestra.run.smithi160.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T17:51:33.573 INFO:teuthology.orchestra.run.smithi160.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T17:51:33.573 INFO:teuthology.orchestra.run.smithi160.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T17:51:33.573 INFO:teuthology.orchestra.run.smithi160.stdout:UBUNTU_CODENAME=jammy 2024-09-21T17:51:33.573 INFO:teuthology.lock.ops:Updating smithi160.front.sepia.ceph.com on lock server 2024-09-21T17:51:33.594 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T17:51:33.603 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T17:51:33.638 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T17:51:33.638 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-09-21T17:51:33.641 DEBUG:teuthology.orchestra.run.smithi040:> test '!' -e /home/ubuntu/cephtest 2024-09-21T17:51:33.643 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-09-21T17:51:33.646 DEBUG:teuthology.orchestra.run.smithi160:> test '!' -e /home/ubuntu/cephtest 2024-09-21T17:51:33.651 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T17:51:33.683 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T17:51:33.683 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-09-21T17:51:33.688 DEBUG:teuthology.orchestra.run.smithi040:> test -z $(ls -A /var/lib/ceph) 2024-09-21T17:51:33.691 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-09-21T17:51:33.693 DEBUG:teuthology.orchestra.run.smithi160:> test -z $(ls -A /var/lib/ceph) 2024-09-21T17:51:33.774 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T17:51:33.865 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T17:51:33.878 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T17:51:33.879 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T17:51:33.879 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T17:51:33.879 DEBUG:teuthology.orchestra.run.smithi040:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T17:51:33.880 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T17:51:33.880 DEBUG:teuthology.orchestra.run.smithi160:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T17:51:33.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T17:51:33.885 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-21T17:51:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T17:51:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T17:51:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T17:51:33.887 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-21T17:51:33.887 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-21T17:51:33.888 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-09-21T17:51:33.932 INFO:teuthology.orchestra.run.smithi070.stdout:5.15.0-56-generic 2024-09-21T17:51:33.932 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.15.0-56-generic 2024-09-21T17:51:33.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get clean 2024-09-21T17:51:33.934 INFO:teuthology.orchestra.run.smithi160.stdout:5.15.0-56-generic 2024-09-21T17:51:33.934 INFO:teuthology.task.kernel:Running kernel on smithi160: 5.15.0-56-generic 2024-09-21T17:51:33.934 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2024-09-21T17:51:33.935 INFO:teuthology.orchestra.run.smithi040.stdout:5.15.0-56-generic 2024-09-21T17:51:33.936 INFO:teuthology.orchestra.run.smithi029.stdout:5.15.0-56-generic 2024-09-21T17:51:33.936 INFO:teuthology.task.kernel:Running kernel on smithi040: 5.15.0-56-generic 2024-09-21T17:51:33.936 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2024-09-21T17:51:33.936 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.15.0-56-generic 2024-09-21T17:51:33.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get clean 2024-09-21T17:51:34.061 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2024-09-21T17:51:34.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2024-09-21T17:51:34.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-09-21T17:51:34.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2024-09-21T17:51:34.200 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T17:51:34.218 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:51:34.236 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T17:51:34.242 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:51:34.242 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T17:51:34.242 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T17:51:34.303 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T17:51:34.311 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:51:34.311 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T17:51:34.330 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T17:51:34.366 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:51:34.369 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T17:51:34.393 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T17:51:34.420 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T17:51:34.424 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T17:51:34.471 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T17:51:34.484 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T17:51:34.536 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T17:51:34.540 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T17:51:34.542 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T17:51:34.542 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T17:51:34.546 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T17:51:34.557 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T17:51:34.563 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T17:51:34.563 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T17:51:34.566 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T17:51:34.566 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T17:51:34.573 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T17:51:34.577 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T17:51:34.579 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T17:51:34.579 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T17:51:34.579 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T17:51:34.582 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T17:51:34.586 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T17:51:34.586 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T17:51:34.587 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T17:51:34.594 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T17:51:34.595 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T17:51:34.595 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T17:51:34.600 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T17:51:34.603 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T17:51:34.603 INFO:teuthology.orchestra.run.smithi070.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T17:51:34.603 INFO:teuthology.orchestra.run.smithi070.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T17:51:34.609 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T17:51:34.613 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T17:51:34.613 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T17:51:34.618 INFO:teuthology.orchestra.run.smithi070.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T17:51:34.623 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T17:51:34.625 INFO:teuthology.orchestra.run.smithi070.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T17:51:34.628 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T17:51:34.630 INFO:teuthology.orchestra.run.smithi070.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T17:51:34.630 INFO:teuthology.orchestra.run.smithi070.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T17:51:34.630 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T17:51:34.636 INFO:teuthology.orchestra.run.smithi070.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T17:51:34.646 INFO:teuthology.orchestra.run.smithi070.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T17:51:34.648 INFO:teuthology.orchestra.run.smithi070.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T17:51:34.667 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T17:51:34.668 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T17:51:34.668 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T17:51:34.673 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T17:51:34.673 INFO:teuthology.orchestra.run.smithi160.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T17:51:34.673 INFO:teuthology.orchestra.run.smithi160.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T17:51:34.674 INFO:teuthology.orchestra.run.smithi160.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T17:51:34.674 INFO:teuthology.orchestra.run.smithi160.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T17:51:34.685 INFO:teuthology.orchestra.run.smithi160.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T17:51:34.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T17:51:34.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T17:51:34.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T17:51:34.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T17:51:34.686 INFO:teuthology.orchestra.run.smithi070.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T17:51:34.690 INFO:teuthology.orchestra.run.smithi070.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T17:51:34.691 INFO:teuthology.orchestra.run.smithi070.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T17:51:34.702 INFO:teuthology.orchestra.run.smithi160.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T17:51:34.703 INFO:teuthology.orchestra.run.smithi160.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T17:51:34.703 INFO:teuthology.orchestra.run.smithi160.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T17:51:34.724 INFO:teuthology.orchestra.run.smithi070.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T17:51:34.725 INFO:teuthology.orchestra.run.smithi070.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T17:51:34.730 INFO:teuthology.orchestra.run.smithi070.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T17:51:34.730 INFO:teuthology.orchestra.run.smithi070.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T17:51:34.737 INFO:teuthology.orchestra.run.smithi070.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T17:51:34.746 INFO:teuthology.orchestra.run.smithi070.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T17:51:34.748 INFO:teuthology.orchestra.run.smithi070.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T17:51:34.749 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T17:51:34.749 INFO:teuthology.orchestra.run.smithi070.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T17:51:34.749 INFO:teuthology.orchestra.run.smithi070.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T17:51:34.749 INFO:teuthology.orchestra.run.smithi070.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T17:51:34.761 INFO:teuthology.orchestra.run.smithi070.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T17:51:34.761 INFO:teuthology.orchestra.run.smithi070.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T17:51:34.761 INFO:teuthology.orchestra.run.smithi070.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T17:51:34.762 INFO:teuthology.orchestra.run.smithi070.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T17:51:34.776 INFO:teuthology.orchestra.run.smithi160.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T17:51:34.776 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T17:51:34.778 INFO:teuthology.orchestra.run.smithi070.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T17:51:34.778 INFO:teuthology.orchestra.run.smithi070.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T17:51:34.778 INFO:teuthology.orchestra.run.smithi070.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T17:51:34.779 INFO:teuthology.orchestra.run.smithi070.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T17:51:34.779 INFO:teuthology.orchestra.run.smithi070.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T17:51:34.802 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T17:51:34.887 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T17:51:34.915 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T17:51:35.082 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T17:51:35.202 INFO:teuthology.orchestra.run.smithi160.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T17:51:35.218 INFO:teuthology.orchestra.run.smithi160.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T17:51:35.227 INFO:teuthology.orchestra.run.smithi160.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T17:51:35.228 INFO:teuthology.orchestra.run.smithi160.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T17:51:35.235 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T17:51:35.246 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T17:51:35.248 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T17:51:35.260 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T17:51:35.261 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T17:51:35.265 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T17:51:35.270 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T17:51:35.274 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T17:51:35.275 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T17:51:35.276 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T17:51:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T17:51:35.285 INFO:teuthology.orchestra.run.smithi029.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T17:51:35.325 INFO:teuthology.orchestra.run.smithi160.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T17:51:35.326 INFO:teuthology.orchestra.run.smithi160.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T17:51:35.331 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T17:51:35.335 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T17:51:35.337 INFO:teuthology.orchestra.run.smithi160.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T17:51:35.337 INFO:teuthology.orchestra.run.smithi160.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T17:51:35.349 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T17:51:35.350 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T17:51:35.355 INFO:teuthology.orchestra.run.smithi029.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T17:51:35.359 INFO:teuthology.orchestra.run.smithi029.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T17:51:35.365 INFO:teuthology.orchestra.run.smithi160.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T17:51:35.371 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T17:51:35.372 INFO:teuthology.orchestra.run.smithi029.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T17:51:35.373 INFO:teuthology.orchestra.run.smithi029.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T17:51:35.374 INFO:teuthology.orchestra.run.smithi029.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T17:51:35.380 INFO:teuthology.orchestra.run.smithi160.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T17:51:35.387 INFO:teuthology.orchestra.run.smithi160.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T17:51:35.387 INFO:teuthology.orchestra.run.smithi160.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T17:51:35.387 INFO:teuthology.orchestra.run.smithi160.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T17:51:35.399 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T17:51:35.404 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T17:51:35.410 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T17:51:35.411 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T17:51:35.417 INFO:teuthology.orchestra.run.smithi029.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T17:51:35.417 INFO:teuthology.orchestra.run.smithi029.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T17:51:35.417 INFO:teuthology.orchestra.run.smithi029.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T17:51:35.429 INFO:teuthology.orchestra.run.smithi029.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T17:51:35.438 INFO:teuthology.orchestra.run.smithi029.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T17:51:35.443 INFO:teuthology.orchestra.run.smithi029.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T17:51:35.443 INFO:teuthology.orchestra.run.smithi029.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T17:51:35.445 INFO:teuthology.orchestra.run.smithi029.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T17:51:35.445 INFO:teuthology.orchestra.run.smithi029.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T17:51:35.448 INFO:teuthology.orchestra.run.smithi029.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T17:51:35.448 INFO:teuthology.orchestra.run.smithi029.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T17:51:35.449 INFO:teuthology.orchestra.run.smithi029.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T17:51:35.449 INFO:teuthology.orchestra.run.smithi029.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T17:51:35.467 INFO:teuthology.orchestra.run.smithi160.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T17:51:35.467 INFO:teuthology.orchestra.run.smithi160.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T17:51:35.482 INFO:teuthology.orchestra.run.smithi029.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T17:51:35.482 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T17:51:35.483 INFO:teuthology.orchestra.run.smithi029.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T17:51:35.483 INFO:teuthology.orchestra.run.smithi029.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T17:51:35.484 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T17:51:35.484 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T17:51:35.484 INFO:teuthology.orchestra.run.smithi029.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T17:51:35.485 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T17:51:35.486 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T17:51:35.491 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T17:51:35.492 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T17:51:35.492 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T17:51:35.515 INFO:teuthology.orchestra.run.smithi029.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T17:51:35.527 INFO:teuthology.orchestra.run.smithi029.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T17:51:35.528 INFO:teuthology.orchestra.run.smithi029.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T17:51:35.528 INFO:teuthology.orchestra.run.smithi029.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T17:51:35.529 INFO:teuthology.orchestra.run.smithi029.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T17:51:35.530 INFO:teuthology.orchestra.run.smithi029.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T17:51:35.563 INFO:teuthology.orchestra.run.smithi029.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T17:51:35.572 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T17:51:35.573 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T17:51:41.048 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 16.3 MB in 2s (7,415 kB/s) 2024-09-21T17:51:41.067 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 16.3 MB in 2s (7,352 kB/s) 2024-09-21T17:51:41.273 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 16.3 MB in 2s (7,153 kB/s) 2024-09-21T17:51:42.235 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 16.3 MB in 3s (5,527 kB/s) 2024-09-21T17:51:42.403 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T17:51:42.421 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:51:42.483 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T17:51:42.627 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T17:51:42.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:51:42.695 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-09-21T17:51:42.695 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-09-21T17:51:42.710 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T17:51:42.760 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T17:51:42.778 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:51:42.844 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T17:51:42.885 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:51:42.885 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:51:42.885 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:51:42.886 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:51:42.886 INFO:teuthology.orchestra.run.smithi070.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T17:51:42.886 INFO:teuthology.orchestra.run.smithi070.stdout: python2.7-minimal 2024-09-21T17:51:42.887 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:51:42.888 INFO:teuthology.orchestra.run.smithi070.stdout:The following additional packages will be installed: 2024-09-21T17:51:42.888 INFO:teuthology.orchestra.run.smithi070.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:42.888 INFO:teuthology.orchestra.run.smithi070.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T17:51:42.888 INFO:teuthology.orchestra.run.smithi070.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:42.890 INFO:teuthology.orchestra.run.smithi070.stdout:Suggested packages: 2024-09-21T17:51:42.890 INFO:teuthology.orchestra.run.smithi070.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T17:51:42.890 INFO:teuthology.orchestra.run.smithi070.stdout:Recommended packages: 2024-09-21T17:51:42.890 INFO:teuthology.orchestra.run.smithi070.stdout: thermald 2024-09-21T17:51:42.920 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2024-09-21T17:51:42.921 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2024-09-21T17:51:42.945 INFO:teuthology.orchestra.run.smithi070.stdout:The following NEW packages will be installed: 2024-09-21T17:51:42.945 INFO:teuthology.orchestra.run.smithi070.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:42.945 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T17:51:42.946 INFO:teuthology.orchestra.run.smithi070.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:42.947 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages will be upgraded: 2024-09-21T17:51:42.948 INFO:teuthology.orchestra.run.smithi070.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T17:51:42.997 INFO:teuthology.orchestra.run.smithi070.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:51:42.998 INFO:teuthology.orchestra.run.smithi070.stdout:Need to get 113 MB of archives. 2024-09-21T17:51:42.998 INFO:teuthology.orchestra.run.smithi070.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T17:51:42.998 INFO:teuthology.orchestra.run.smithi070.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T17:51:43.057 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2024-09-21T17:51:43.058 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2024-09-21T17:51:43.111 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:51:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:51:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:51:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:51:43.111 INFO:teuthology.orchestra.run.smithi160.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T17:51:43.112 INFO:teuthology.orchestra.run.smithi160.stdout: python2.7-minimal 2024-09-21T17:51:43.112 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:51:43.113 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2024-09-21T17:51:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T17:51:43.113 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:43.115 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2024-09-21T17:51:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T17:51:43.115 INFO:teuthology.orchestra.run.smithi160.stdout:Recommended packages: 2024-09-21T17:51:43.115 INFO:teuthology.orchestra.run.smithi160.stdout: thermald 2024-09-21T17:51:43.168 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2024-09-21T17:51:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T17:51:43.168 INFO:teuthology.orchestra.run.smithi160.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:43.170 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages will be upgraded: 2024-09-21T17:51:43.170 INFO:teuthology.orchestra.run.smithi160.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T17:51:43.221 INFO:teuthology.orchestra.run.smithi160.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:51:43.222 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 113 MB of archives. 2024-09-21T17:51:43.222 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T17:51:43.222 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T17:51:43.256 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:51:43.257 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:51:43.257 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:51:43.257 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:51:43.257 INFO:teuthology.orchestra.run.smithi040.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T17:51:43.257 INFO:teuthology.orchestra.run.smithi040.stdout: python2.7-minimal 2024-09-21T17:51:43.258 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:51:43.259 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2024-09-21T17:51:43.259 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:43.259 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T17:51:43.260 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:43.261 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2024-09-21T17:51:43.261 INFO:teuthology.orchestra.run.smithi040.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T17:51:43.261 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2024-09-21T17:51:43.262 INFO:teuthology.orchestra.run.smithi040.stdout: thermald 2024-09-21T17:51:43.315 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2024-09-21T17:51:43.315 INFO:teuthology.orchestra.run.smithi040.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:43.315 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T17:51:43.315 INFO:teuthology.orchestra.run.smithi040.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:43.317 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages will be upgraded: 2024-09-21T17:51:43.318 INFO:teuthology.orchestra.run.smithi040.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T17:51:43.371 INFO:teuthology.orchestra.run.smithi040.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:51:43.371 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 113 MB of archives. 2024-09-21T17:51:43.371 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T17:51:43.371 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T17:51:43.432 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T17:51:43.543 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T17:51:43.646 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T17:51:43.740 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T17:51:43.824 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T17:51:43.937 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T17:51:44.360 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T17:51:44.360 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T17:51:44.360 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T17:51:44.480 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T17:51:44.502 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T17:51:44.516 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T17:51:44.516 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T17:51:44.516 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T17:51:44.631 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T17:51:44.636 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T17:51:44.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:51:44.657 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T17:51:44.708 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T17:51:44.782 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T17:51:44.782 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T17:51:44.782 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T17:51:44.911 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T17:51:44.923 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2024-09-21T17:51:44.924 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2024-09-21T17:51:44.933 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T17:51:44.965 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 113 MB in 2s (70.4 MB/s) 2024-09-21T17:51:45.115 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:51:45.115 INFO:teuthology.orchestra.run.smithi029.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:51:45.115 INFO:teuthology.orchestra.run.smithi029.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:51:45.116 INFO:teuthology.orchestra.run.smithi029.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:51:45.116 INFO:teuthology.orchestra.run.smithi029.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T17:51:45.116 INFO:teuthology.orchestra.run.smithi029.stdout: python2.7-minimal 2024-09-21T17:51:45.116 INFO:teuthology.orchestra.run.smithi029.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:51:45.117 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2024-09-21T17:51:45.118 INFO:teuthology.orchestra.run.smithi029.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:45.118 INFO:teuthology.orchestra.run.smithi029.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T17:51:45.118 INFO:teuthology.orchestra.run.smithi029.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:45.119 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2024-09-21T17:51:45.119 INFO:teuthology.orchestra.run.smithi029.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T17:51:45.120 INFO:teuthology.orchestra.run.smithi029.stdout:Recommended packages: 2024-09-21T17:51:45.120 INFO:teuthology.orchestra.run.smithi029.stdout: thermald 2024-09-21T17:51:45.123 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 113 MB in 2s (73.4 MB/s) 2024-09-21T17:51:45.140 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T17:51:45.175 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2024-09-21T17:51:45.176 INFO:teuthology.orchestra.run.smithi029.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T17:51:45.176 INFO:teuthology.orchestra.run.smithi029.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T17:51:45.176 INFO:teuthology.orchestra.run.smithi029.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T17:51:45.177 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages will be upgraded: 2024-09-21T17:51:45.177 INFO:teuthology.orchestra.run.smithi029.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T17:51:45.228 INFO:teuthology.orchestra.run.smithi029.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:51:45.228 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 113 MB of archives. 2024-09-21T17:51:45.228 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T17:51:45.228 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T17:51:45.337 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T17:51:45.413 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 113 MB in 2s (67.7 MB/s) 2024-09-21T17:51:45.597 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T17:51:45.712 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T17:51:45.830 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T17:51:46.966 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T17:51:46.967 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T17:51:46.967 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T17:51:47.091 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T17:51:47.096 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:47.097 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T17:51:47.120 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T17:51:47.238 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:47.314 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T17:51:47.318 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:47.467 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:47.601 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T17:51:47.605 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:47.670 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:47.687 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 113 MB in 2s (54.3 MB/s) 2024-09-21T17:51:47.898 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T17:51:49.240 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T17:51:49.262 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:49.357 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:49.684 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T17:51:49.705 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:49.789 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:49.868 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T17:51:49.889 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:49.926 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:50.030 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T17:51:50.051 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:50.145 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:50.262 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T17:51:50.282 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:50.312 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:50.625 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T17:51:50.646 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:50.683 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:50.864 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T17:51:50.868 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:51.182 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:53.955 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T17:51:53.976 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:54.104 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:54.645 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T17:51:54.665 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:51:54.711 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:51:56.529 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:56.657 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:57.032 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:57.160 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:57.268 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:57.396 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:57.437 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:57.438 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T17:51:57.449 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T17:51:57.479 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:51:57.545 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:57.746 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:57.871 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:51:57.874 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:57.998 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:51:58.227 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T17:51:58.249 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T17:51:58.293 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:51:58.302 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T17:51:58.324 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T17:51:58.368 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:03.064 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:03.188 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:03.235 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T17:52:03.272 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:52:03.309 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:03.533 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:03.662 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:03.673 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T17:52:03.692 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:52:03.731 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:04.026 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T17:52:04.048 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T17:52:04.084 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:04.490 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T17:52:04.527 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:52:04.565 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:05.840 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:05.956 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:06.289 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:06.385 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:06.486 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T17:52:06.616 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:06.718 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:06.843 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:07.169 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:07.298 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:07.426 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T17:52:07.561 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:07.852 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:07.948 INFO:teuthology.orchestra.run.smithi070.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:07.948 INFO:teuthology.orchestra.run.smithi070.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:07.948 INFO:teuthology.orchestra.run.smithi070.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:07.948 INFO:teuthology.orchestra.run.smithi070.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:08.002 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:08.110 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:08.377 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:08.524 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:08.674 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T17:52:08.778 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:08.935 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:08.935 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:08.935 INFO:teuthology.orchestra.run.smithi040.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:08.936 INFO:teuthology.orchestra.run.smithi040.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:09.088 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:09.477 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T17:52:09.578 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T17:52:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:10.159 INFO:teuthology.orchestra.run.smithi160.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:10.647 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:10.738 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:10.889 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T17:52:11.007 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T17:52:11.158 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:12.049 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:12.111 INFO:teuthology.orchestra.run.smithi070.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T17:52:12.112 INFO:teuthology.orchestra.run.smithi070.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:12.147 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T17:52:12.290 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T17:52:12.441 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:12.576 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:12.631 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T17:52:12.631 INFO:teuthology.orchestra.run.smithi040.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:13.067 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T17:52:13.105 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T17:52:13.150 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:13.818 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:13.874 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T17:52:13.875 INFO:teuthology.orchestra.run.smithi160.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:15.496 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T17:52:15.655 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T17:52:16.060 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T17:52:16.191 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:16.328 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T17:52:17.631 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:19.515 INFO:teuthology.orchestra.run.smithi029.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:19.516 INFO:teuthology.orchestra.run.smithi029.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:19.692 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:21.128 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T17:52:21.296 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T17:52:21.423 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:22.862 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T17:52:22.921 INFO:teuthology.orchestra.run.smithi029.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T17:52:22.921 INFO:teuthology.orchestra.run.smithi029.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:26.807 INFO:teuthology.orchestra.run.smithi070.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T17:52:26.808 INFO:teuthology.orchestra.run.smithi070.stdout:Sourcing file `/etc/default/grub' 2024-09-21T17:52:26.824 INFO:teuthology.orchestra.run.smithi070.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:26.839 INFO:teuthology.orchestra.run.smithi070.stdout:Generating grub configuration file ... 2024-09-21T17:52:27.102 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:27.112 INFO:teuthology.orchestra.run.smithi040.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T17:52:27.112 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub' 2024-09-21T17:52:27.122 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:27.123 INFO:teuthology.orchestra.run.smithi040.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:27.161 INFO:teuthology.orchestra.run.smithi040.stdout:Generating grub configuration file ... 2024-09-21T17:52:27.411 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:27.417 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:27.431 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:27.457 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:27.485 INFO:teuthology.orchestra.run.smithi070.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:27.490 INFO:teuthology.orchestra.run.smithi070.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:27.667 INFO:teuthology.orchestra.run.smithi070.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:27.703 INFO:teuthology.orchestra.run.smithi070.stdout:done 2024-09-21T17:52:27.747 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:27.753 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:27.822 INFO:teuthology.orchestra.run.smithi040.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:27.828 INFO:teuthology.orchestra.run.smithi040.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:27.985 INFO:teuthology.orchestra.run.smithi040.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:28.028 INFO:teuthology.orchestra.run.smithi040.stdout:done 2024-09-21T17:52:28.167 DEBUG:teuthology.orchestra.run.smithi070:> dpkg -s linux-image-generic 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Package: linux-image-generic 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Status: install ok installed 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Priority: optional 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Section: kernel 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Installed-Size: 21 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:28.191 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: amd64 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Source: linux-meta 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Recommends: thermald 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:28.192 INFO:teuthology.orchestra.run.smithi070.stdout: available. 2024-09-21T17:52:28.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T17:52:28.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T17:52:28.192 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T17:52:28.193 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-09-21T17:52:28.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T17:52:28.193 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get clean 2024-09-21T17:52:28.347 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-09-21T17:52:28.409 INFO:teuthology.orchestra.run.smithi160.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T17:52:28.409 INFO:teuthology.orchestra.run.smithi160.stdout:Sourcing file `/etc/default/grub' 2024-09-21T17:52:28.442 INFO:teuthology.orchestra.run.smithi160.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:28.455 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T17:52:28.471 INFO:teuthology.orchestra.run.smithi160.stdout:Generating grub configuration file ... 2024-09-21T17:52:28.478 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 21 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2024-09-21T17:52:28.499 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:28.500 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2024-09-21T17:52:28.500 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T17:52:28.500 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T17:52:28.500 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T17:52:28.500 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T17:52:28.500 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi040.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T17:52:28.501 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get clean 2024-09-21T17:52:28.577 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:52:28.660 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T17:52:28.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2024-09-21T17:52:28.756 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T17:52:28.795 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:28.816 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:28.938 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:52:28.938 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T17:52:29.004 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T17:52:29.085 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T17:52:29.167 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:29.173 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:29.243 INFO:teuthology.orchestra.run.smithi160.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:29.249 INFO:teuthology.orchestra.run.smithi160.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:29.407 INFO:teuthology.orchestra.run.smithi160.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:29.407 INFO:teuthology.orchestra.run.smithi160.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:29.408 INFO:teuthology.orchestra.run.smithi160.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:29.449 INFO:teuthology.orchestra.run.smithi160.stdout:done 2024-09-21T17:52:29.966 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T17:52:29.978 DEBUG:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2024-09-21T17:52:29.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:52:30.000 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2024-09-21T17:52:30.000 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2024-09-21T17:52:30.000 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2024-09-21T17:52:30.000 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 21 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:30.001 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2024-09-21T17:52:30.001 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T17:52:30.001 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T17:52:30.002 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T17:52:30.002 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2024-09-21T17:52:30.002 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi160.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T17:52:30.002 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get clean 2024-09-21T17:52:30.046 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T17:52:30.155 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2024-09-21T17:52:30.258 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-09-21T17:52:30.258 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-09-21T17:52:30.404 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:52:30.404 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T17:52:30.415 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T17:52:30.434 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:52:30.452 INFO:teuthology.orchestra.run.smithi070.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T17:52:30.452 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:52:30.453 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:52:30.453 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:52:30.453 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:52:30.453 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T17:52:30.454 INFO:teuthology.orchestra.run.smithi070.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T17:52:30.454 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:52:30.477 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T17:52:30.488 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T17:52:30.495 INFO:teuthology.orchestra.run.smithi070.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:52:30.497 DEBUG:teuthology.orchestra.run.smithi070:> dpkg -s linux-image-generic 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Package: linux-image-generic 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Status: install ok installed 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Priority: optional 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Section: kernel 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Installed-Size: 21 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: amd64 2024-09-21T17:52:30.509 INFO:teuthology.orchestra.run.smithi070.stdout:Source: linux-meta 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:Recommends: thermald 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:30.510 INFO:teuthology.orchestra.run.smithi070.stdout: available. 2024-09-21T17:52:30.510 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-21T17:52:30.554 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.gcYOoHY5RB 2024-09-21T17:52:30.554 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub/grub.cfg /tmp/tmp.gcYOoHY5RB 2024-09-21T17:52:30.558 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T17:52:30.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.gcYOoHY5RB 2024-09-21T17:52:30.684 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2024-09-21T17:52:30.685 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2024-09-21T17:52:30.737 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.gcYOoHY5RB is 10KB 2024-09-21T17:52:30.752 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.gcYOoHY5RB 2024-09-21T17:52:30.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:30.809 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T17:52:30.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T17:52:30.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T17:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:52:30.862 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T17:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T17:52:30.863 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:52:30.873 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T17:52:30.873 DEBUG:teuthology.orchestra.run.smithi070:> sudo update-grub 2024-09-21T17:52:30.904 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:52:30.906 DEBUG:teuthology.orchestra.run.smithi040:> dpkg -s linux-image-generic 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Package: linux-image-generic 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Status: install ok installed 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Priority: optional 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Section: kernel 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Installed-Size: 21 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Architecture: amd64 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Source: linux-meta 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:30.965 INFO:teuthology.orchestra.run.smithi040.stdout:Recommends: thermald 2024-09-21T17:52:30.966 INFO:teuthology.orchestra.run.smithi040.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:30.966 INFO:teuthology.orchestra.run.smithi040.stdout: available. 2024-09-21T17:52:30.966 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-09-21T17:52:31.009 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.o5bodbGBqG 2024-09-21T17:52:31.009 DEBUG:teuthology.orchestra.run.smithi040:> sudo cp /boot/grub/grub.cfg /tmp/tmp.o5bodbGBqG 2024-09-21T17:52:31.065 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.o5bodbGBqG 2024-09-21T17:52:31.182 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.o5bodbGBqG is 10KB 2024-09-21T17:52:31.197 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.o5bodbGBqG 2024-09-21T17:52:31.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:31.257 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T17:52:31.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T17:52:31.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:31.325 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T17:52:31.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo update-grub 2024-09-21T17:52:31.848 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T17:52:31.867 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:52:31.928 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T17:52:32.065 INFO:teuthology.orchestra.run.smithi070.stderr:Sourcing file `/etc/default/grub' 2024-09-21T17:52:32.066 INFO:teuthology.orchestra.run.smithi070.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:32.068 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-09-21T17:52:32.128 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2024-09-21T17:52:32.129 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2024-09-21T17:52:32.236 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:32.242 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:32.300 INFO:teuthology.orchestra.run.smithi160.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T17:52:32.300 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:52:32.300 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:52:32.300 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:52:32.300 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:52:32.301 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T17:52:32.301 INFO:teuthology.orchestra.run.smithi160.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T17:52:32.301 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:52:32.342 INFO:teuthology.orchestra.run.smithi160.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:52:32.343 DEBUG:teuthology.orchestra.run.smithi160:> dpkg -s linux-image-generic 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Package: linux-image-generic 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Status: install ok installed 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Priority: optional 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Section: kernel 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Installed-Size: 21 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Architecture: amd64 2024-09-21T17:52:32.359 INFO:teuthology.orchestra.run.smithi160.stdout:Source: linux-meta 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout:Recommends: thermald 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:32.360 INFO:teuthology.orchestra.run.smithi160.stdout: available. 2024-09-21T17:52:32.360 DEBUG:teuthology.orchestra.run.smithi160:> mktemp 2024-09-21T17:52:32.403 INFO:teuthology.orchestra.run.smithi160.stdout:/tmp/tmp.OiCAh5NYRt 2024-09-21T17:52:32.404 DEBUG:teuthology.orchestra.run.smithi160:> sudo cp /boot/grub/grub.cfg /tmp/tmp.OiCAh5NYRt 2024-09-21T17:52:32.458 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:32.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0666 /tmp/tmp.OiCAh5NYRt 2024-09-21T17:52:32.464 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:32.504 INFO:teuthology.orchestra.run.smithi070.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:32.509 INFO:teuthology.orchestra.run.smithi070.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:32.541 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub' 2024-09-21T17:52:32.542 INFO:teuthology.orchestra.run.smithi040.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:32.544 INFO:teuthology.orchestra.run.smithi040.stderr:Generating grub configuration file ... 2024-09-21T17:52:32.573 DEBUG:teuthology.orchestra.remote:smithi160:/tmp/tmp.OiCAh5NYRt is 10KB 2024-09-21T17:52:32.585 DEBUG:teuthology.orchestra.run.smithi160:> rm -fr /tmp/tmp.OiCAh5NYRt 2024-09-21T17:52:32.590 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:32.626 INFO:teuthology.orchestra.run.smithi070.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:32.626 INFO:teuthology.orchestra.run.smithi070.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:32.626 INFO:teuthology.orchestra.run.smithi070.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:32.639 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-09-21T17:52:32.647 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T17:52:32.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T17:52:32.647 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:32.648 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-09-21T17:52:32.731 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T17:52:32.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo update-grub 2024-09-21T17:52:32.740 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:32.746 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:32.977 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:32.982 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:33.028 INFO:teuthology.orchestra.run.smithi040.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:33.033 INFO:teuthology.orchestra.run.smithi040.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:33.164 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:33.164 INFO:teuthology.orchestra.run.smithi040.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:33.164 INFO:teuthology.orchestra.run.smithi040.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:33.177 INFO:teuthology.orchestra.run.smithi040.stderr:done 2024-09-21T17:52:33.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo shutdown -r now 2024-09-21T17:52:34.267 INFO:teuthology.orchestra.run.smithi160.stderr:Sourcing file `/etc/default/grub' 2024-09-21T17:52:34.268 INFO:teuthology.orchestra.run.smithi160.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:34.271 INFO:teuthology.orchestra.run.smithi160.stderr:Generating grub configuration file ... 2024-09-21T17:52:34.507 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:34.513 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:34.775 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:34.781 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:34.821 INFO:teuthology.orchestra.run.smithi160.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:34.826 INFO:teuthology.orchestra.run.smithi160.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:34.943 INFO:teuthology.orchestra.run.smithi160.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:34.943 INFO:teuthology.orchestra.run.smithi160.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:34.943 INFO:teuthology.orchestra.run.smithi160.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:34.952 INFO:teuthology.orchestra.run.smithi160.stderr:done 2024-09-21T17:52:34.954 DEBUG:teuthology.orchestra.run.smithi160:> sudo shutdown -r now 2024-09-21T17:52:40.951 INFO:teuthology.orchestra.run.smithi029.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T17:52:40.952 INFO:teuthology.orchestra.run.smithi029.stdout:Sourcing file `/etc/default/grub' 2024-09-21T17:52:41.017 INFO:teuthology.orchestra.run.smithi029.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:41.039 INFO:teuthology.orchestra.run.smithi029.stdout:Generating grub configuration file ... 2024-09-21T17:52:41.385 INFO:teuthology.orchestra.run.smithi029.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:41.422 INFO:teuthology.orchestra.run.smithi029.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:41.735 INFO:teuthology.orchestra.run.smithi029.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:41.742 INFO:teuthology.orchestra.run.smithi029.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:41.826 INFO:teuthology.orchestra.run.smithi029.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:41.832 INFO:teuthology.orchestra.run.smithi029.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:42.145 INFO:teuthology.orchestra.run.smithi029.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:42.145 INFO:teuthology.orchestra.run.smithi029.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:42.145 INFO:teuthology.orchestra.run.smithi029.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:42.248 INFO:teuthology.orchestra.run.smithi029.stdout:done 2024-09-21T17:52:42.847 DEBUG:teuthology.orchestra.run.smithi029:> dpkg -s linux-image-generic 2024-09-21T17:52:42.908 INFO:teuthology.orchestra.run.smithi029.stdout:Package: linux-image-generic 2024-09-21T17:52:42.908 INFO:teuthology.orchestra.run.smithi029.stdout:Status: install ok installed 2024-09-21T17:52:42.908 INFO:teuthology.orchestra.run.smithi029.stdout:Priority: optional 2024-09-21T17:52:42.908 INFO:teuthology.orchestra.run.smithi029.stdout:Section: kernel 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Installed-Size: 21 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: amd64 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Source: linux-meta 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Recommends: thermald 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:42.909 INFO:teuthology.orchestra.run.smithi029.stdout: available. 2024-09-21T17:52:42.909 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T17:52:42.910 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T17:52:42.910 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T17:52:42.910 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T17:52:42.910 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T17:52:42.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get clean 2024-09-21T17:52:43.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2024-09-21T17:52:43.327 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T17:52:43.327 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T17:52:43.387 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T17:52:43.469 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T17:52:44.900 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T17:52:44.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T17:52:44.974 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T17:52:45.165 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2024-09-21T17:52:45.165 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T17:52:45.342 INFO:teuthology.orchestra.run.smithi029.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T17:52:45.343 INFO:teuthology.orchestra.run.smithi029.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T17:52:45.343 INFO:teuthology.orchestra.run.smithi029.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T17:52:45.387 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T17:52:45.389 DEBUG:teuthology.orchestra.run.smithi029:> dpkg -s linux-image-generic 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Package: linux-image-generic 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Status: install ok installed 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Priority: optional 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Section: kernel 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Installed-Size: 21 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: amd64 2024-09-21T17:52:45.402 INFO:teuthology.orchestra.run.smithi029.stdout:Source: linux-meta 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout:Version: 5.15.0.122.122 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout:Recommends: thermald 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout:Description: Generic Linux kernel image 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout: This package will always depend on the latest generic kernel image 2024-09-21T17:52:45.403 INFO:teuthology.orchestra.run.smithi029.stdout: available. 2024-09-21T17:52:45.403 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-09-21T17:52:45.450 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.nkFzJifTB2 2024-09-21T17:52:45.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub/grub.cfg /tmp/tmp.nkFzJifTB2 2024-09-21T17:52:45.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.nkFzJifTB2 2024-09-21T17:52:45.624 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.nkFzJifTB2 is 10KB 2024-09-21T17:52:45.637 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.nkFzJifTB2 2024-09-21T17:52:45.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:45.694 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T17:52:45.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T17:52:45.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T17:52:45.790 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T17:52:45.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo update-grub 2024-09-21T17:52:48.171 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub' 2024-09-21T17:52:48.182 INFO:teuthology.orchestra.run.smithi029.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T17:52:48.205 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-09-21T17:52:48.425 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T17:52:48.431 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T17:52:48.681 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T17:52:48.686 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T17:52:48.730 INFO:teuthology.orchestra.run.smithi029.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T17:52:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T17:52:48.875 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T17:52:48.875 INFO:teuthology.orchestra.run.smithi029.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T17:52:48.875 INFO:teuthology.orchestra.run.smithi029.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T17:52:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-09-21T17:52:48.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-09-21T17:53:02.662 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-09-21T17:53:02.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T17:53:02.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:03.197 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T17:53:03.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T17:53:03.198 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:05.000 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2024-09-21T17:53:05.000 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T17:53:05.000 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:18.930 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T17:53:18.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T17:53:18.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:21.135 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-09-21T17:53:21.647 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-09-21T17:53:23.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-21T17:53:30.142 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T17:53:30.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:30.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T17:53:30.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:32.441 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T17:53:32.442 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:33.199 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-09-21T17:53:33.711 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.40 2024-09-21T17:53:35.503 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.160 2024-09-21T17:53:37.263 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-09-21T17:53:45.207 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T17:53:45.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:45.712 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T17:53:45.712 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:46.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T17:53:46.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:53:47.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T17:53:47.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:54:16.945 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-21T17:54:17.532 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T17:54:17.532 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.15.0-122-generic"... 2024-09-21T17:54:17.532 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-09-21T17:54:17.579 INFO:teuthology.orchestra.run.smithi070.stdout:5.15.0-122-generic 2024-09-21T17:54:17.579 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T17:54:17.579 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T17:54:17.579 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T17:54:18.580 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-09-21T17:54:18.581 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T17:54:18.663 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-09-21T17:54:18.696 DEBUG:teuthology.parallel:result is None 2024-09-21T17:54:45.715 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T17:54:46.266 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T17:54:47.508 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T17:54:58.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T17:54:58.268 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:54:58.623 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-21T17:54:59.541 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T17:54:59.541 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-21T17:54:59.541 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-09-21T17:54:59.588 INFO:teuthology.orchestra.run.smithi029.stdout:5.15.0-122-generic 2024-09-21T17:54:59.589 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T17:54:59.589 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T17:54:59.589 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T17:55:00.589 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T17:55:00.590 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T17:55:00.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T17:55:00.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:55:00.732 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-09-21T17:55:00.798 DEBUG:teuthology.parallel:result is None 2024-09-21T17:55:00.998 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-09-21T17:55:01.535 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T17:55:01.535 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-21T17:55:01.535 DEBUG:teuthology.orchestra.run.smithi040:> uname -r 2024-09-21T17:55:01.582 INFO:teuthology.orchestra.run.smithi040.stdout:5.15.0-122-generic 2024-09-21T17:55:01.582 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T17:55:01.582 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T17:55:01.582 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T17:55:02.509 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T17:55:02.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T17:55:02.583 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T17:55:02.583 DEBUG:teuthology.orchestra.run.smithi040:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T17:55:02.614 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-21T17:55:02.647 INFO:teuthology.orchestra.run.smithi040.stdout:ttyS1 2024-09-21T17:55:02.671 DEBUG:teuthology.parallel:result is None 2024-09-21T17:55:03.183 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T17:55:03.183 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.15.0-122-generic"... 2024-09-21T17:55:03.183 DEBUG:teuthology.orchestra.run.smithi160:> uname -r 2024-09-21T17:55:03.231 INFO:teuthology.orchestra.run.smithi160.stdout:5.15.0-122-generic 2024-09-21T17:55:03.231 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T17:55:03.231 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T17:55:03.231 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T17:55:04.232 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2024-09-21T17:55:04.232 DEBUG:teuthology.orchestra.run.smithi160:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T17:55:04.593 INFO:teuthology.orchestra.run.smithi160.stdout:ttyS1 2024-09-21T17:55:04.624 DEBUG:teuthology.parallel:result is None 2024-09-21T17:55:04.625 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T17:55:04.635 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T17:55:04.635 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T17:55:04.638 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T17:55:04.640 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T17:55:04.643 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T17:55:04.648 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T17:55:04.689 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T17:55:04.723 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T17:55:04.723 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T17:55:04.726 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T17:55:04.728 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T17:55:04.730 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T17:55:04.815 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T17:55:04.821 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T17:55:04.821 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T17:55:04.824 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T17:55:04.826 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T17:55:04.828 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T17:55:04.840 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.847 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.848 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.850 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.851 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.855 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.858 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.859 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T17:55:04.860 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T17:55:04.867 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T17:55:04.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T17:55:04.893 DEBUG:teuthology.orchestra.run.smithi040:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T17:55:04.903 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T17:55:04.906 DEBUG:teuthology.orchestra.run.smithi160:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T17:55:04.919 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T17:55:04.927 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T17:55:04.927 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T17:55:04.949 DEBUG:teuthology.orchestra.run.smithi040:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T17:55:04.959 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T17:55:04.961 DEBUG:teuthology.orchestra.run.smithi160:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T17:55:04.967 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T17:55:05.001 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T17:55:05.053 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T17:55:05.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T17:55:05.114 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T17:55:05.120 DEBUG:teuthology.orchestra.run.smithi040:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T17:55:05.166 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T17:55:05.166 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T17:55:05.230 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T17:55:05.236 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T17:55:05.282 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T17:55:05.282 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T17:55:05.345 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T17:55:05.351 DEBUG:teuthology.orchestra.run.smithi160:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T17:55:05.399 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T17:55:05.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T17:55:05.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-09-21T17:55:05.469 DEBUG:teuthology.orchestra.run.smithi040:> sudo service rsyslog restart 2024-09-21T17:55:05.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-09-21T17:55:05.474 DEBUG:teuthology.orchestra.run.smithi160:> sudo service rsyslog restart 2024-09-21T17:55:05.546 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T17:55:05.555 INFO:teuthology.task.internal:Starting timer... 2024-09-21T17:55:05.556 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T17:55:05.599 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T17:55:05.626 DEBUG:teuthology.task.selinux:Excluding smithi029: OS 'ubuntu' does not support SELinux 2024-09-21T17:55:05.626 DEBUG:teuthology.task.selinux:Excluding smithi040: OS 'ubuntu' does not support SELinux 2024-09-21T17:55:05.626 DEBUG:teuthology.task.selinux:Excluding smithi070: OS 'ubuntu' does not support SELinux 2024-09-21T17:55:05.626 DEBUG:teuthology.task.selinux:Excluding smithi160: OS 'ubuntu' does not support SELinux 2024-09-21T17:55:05.627 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T17:55:05.627 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-21T17:55:05.627 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T17:55:05.627 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T17:55:05.659 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T17:55:05.670 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T17:55:05.671 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi040.front.sepia.ceph.com,smithi070.front.sepia.ceph.com,smithi160.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T18:02:51.970 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi040.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi160.front.sepia.ceph.com')] 2024-09-21T18:02:51.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T18:02:51.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:02:52.080 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-09-21T18:02:52.158 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-09-21T18:02:52.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T18:02:52.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi040.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:02:52.238 DEBUG:teuthology.orchestra.run.smithi040:> true 2024-09-21T18:02:52.318 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi040.front.sepia.ceph.com' 2024-09-21T18:02:52.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T18:02:52.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:02:52.396 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-09-21T18:02:52.470 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-09-21T18:02:52.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T18:02:52.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi160.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T18:02:52.551 DEBUG:teuthology.orchestra.run.smithi160:> true 2024-09-21T18:02:52.630 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi160.front.sepia.ceph.com' 2024-09-21T18:02:52.632 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T18:02:52.642 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T18:02:52.642 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:02:52.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:02:52.644 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:02:52.645 DEBUG:teuthology.orchestra.run.smithi040:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:02:52.647 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:02:52.647 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:02:52.649 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T18:02:52.649 DEBUG:teuthology.orchestra.run.smithi160:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:02:52.672 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T18:02:52.672 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Command line: ntpd -gq 2024-09-21T18:02:52.672 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: ---------------------------------------------------- 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: corporation. Support and training for ntp-4 are 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: available at https://www.nwtime.org/support 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: ---------------------------------------------------- 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: proto: precision = 0.046 usec (-24) 2024-09-21T18:02:52.673 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: basedate set to 2022-02-04 2024-09-21T18:02:52.674 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: gps base set to 2022-02-06 (week 2196) 2024-09-21T18:02:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.674 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.674 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.675 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T18:02:52.675 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T18:02:52.675 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T18:02:52.675 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen normally on 3 ens1f0 172.21.15.29:123 2024-09-21T18:02:52.676 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen normally on 4 lo [::1]:123 2024-09-21T18:02:52.676 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:125a%4]:123 2024-09-21T18:02:52.676 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:02:52 ntpd[18433]: Listening on routing socket on fd #22 for interface updates 2024-09-21T18:02:52.677 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Command line: ntpd -gq 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: ---------------------------------------------------- 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: corporation. Support and training for ntp-4 are 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: available at https://www.nwtime.org/support 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: ---------------------------------------------------- 2024-09-21T18:02:52.678 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: proto: precision = 0.068 usec (-24) 2024-09-21T18:02:52.679 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: basedate set to 2022-02-04 2024-09-21T18:02:52.679 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: gps base set to 2022-02-06 (week 2196) 2024-09-21T18:02:52.679 INFO:teuthology.orchestra.run.smithi070.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Command line: ntpd -gq 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: ---------------------------------------------------- 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: corporation. Support and training for ntp-4 are 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: available at https://www.nwtime.org/support 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: ---------------------------------------------------- 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: proto: precision = 0.066 usec (-24) 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: basedate set to 2022-02-04 2024-09-21T18:02:52.680 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: gps base set to 2022-02-06 (week 2196) 2024-09-21T18:02:52.681 INFO:teuthology.orchestra.run.smithi040.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.681 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.681 INFO:teuthology.orchestra.run.smithi070.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.681 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.682 INFO:teuthology.orchestra.run.smithi040.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen normally on 3 enp3s0f1 172.21.15.70:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen normally on 4 lo [::1]:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3e7%5]:123 2024-09-21T18:02:52.683 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:52 ntpd[18474]: Listening on routing socket on fd #22 for interface updates 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen normally on 3 ens1f0 172.21.15.40:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen normally on 4 lo [::1]:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:12cc%4]:123 2024-09-21T18:02:52.684 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:52 ntpd[18401]: Listening on routing socket on fd #22 for interface updates 2024-09-21T18:02:52.705 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Command line: ntpd -gq 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: ---------------------------------------------------- 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: corporation. Support and training for ntp-4 are 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: available at https://www.nwtime.org/support 2024-09-21T18:02:52.706 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: ---------------------------------------------------- 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: proto: precision = 0.045 usec (-24) 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: basedate set to 2022-02-04 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: gps base set to 2022-02-06 (week 2196) 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T18:02:52.707 INFO:teuthology.orchestra.run.smithi160.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.708 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T18:02:52.708 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T18:02:52.708 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T18:02:52.708 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T18:02:52.708 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen normally on 3 enp3s0f1 172.21.15.160:123 2024-09-21T18:02:52.709 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen normally on 4 lo [::1]:123 2024-09-21T18:02:52.709 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f5b%5]:123 2024-09-21T18:02:52.709 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:52 ntpd[18468]: Listening on routing socket on fd #22 for interface updates 2024-09-21T18:02:53.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:53 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:53 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:53.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:53 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:54.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:54.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:54 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:55 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:55 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:55 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.685 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.686 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:55 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:55.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:55 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:56.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:56 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:56 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:56.678 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:56 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:56.679 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:56 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:57 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.680 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:57 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.683 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:57 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.709 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:57.710 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:57 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:58.675 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:58 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:58.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:58 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:59.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:02:59 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:59.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.678 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stdout:21 Sep 18:02:59 ntpd[18474]: ntpd: time slew -0.002217 s 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stdout:ntpd: time slew -0.002217s 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T18:02:59.679 INFO:teuthology.orchestra.run.smithi070.stderr:21 Sep 18:02:59 ntpd[18474]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T18:02:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.680 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stdout:21 Sep 18:02:59 ntpd[18401]: ntpd: time slew +0.000010 s 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stdout:ntpd: time slew +0.000010s 2024-09-21T18:02:59.681 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T18:02:59.682 INFO:teuthology.orchestra.run.smithi040.stderr:21 Sep 18:02:59 ntpd[18401]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T18:02:59.706 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stdout:21 Sep 18:02:59 ntpd[18468]: ntpd: time slew +0.000223 s 2024-09-21T18:02:59.707 INFO:teuthology.orchestra.run.smithi160.stdout:ntpd: time slew +0.000223s 2024-09-21T18:02:59.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T18:02:59.708 INFO:teuthology.orchestra.run.smithi160.stderr:21 Sep 18:02:59 ntpd[18468]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T18:02:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:02:59.728 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================== 2024-09-21T18:02:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.730 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================== 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:02:59.764 INFO:teuthology.orchestra.run.smithi160.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:03:00.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:03:00.673 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stdout:21 Sep 18:03:00 ntpd[18433]: ntpd: time slew +0.000352 s 2024-09-21T18:03:00.674 INFO:teuthology.orchestra.run.smithi029.stdout:ntpd: time slew +0.000352s 2024-09-21T18:03:00.675 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T18:03:00.675 INFO:teuthology.orchestra.run.smithi029.stderr:21 Sep 18:03:00 ntpd[18433]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T18:03:00.721 INFO:teuthology.orchestra.run.smithi029.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:03:00.721 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================== 2024-09-21T18:03:00.722 INFO:teuthology.orchestra.run.smithi029.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:03:00.722 INFO:teuthology.orchestra.run.smithi029.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:03:00.722 INFO:teuthology.orchestra.run.smithi029.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:03:00.722 INFO:teuthology.orchestra.run.smithi029.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T18:03:00.723 INFO:teuthology.run_tasks:Running task install... 2024-09-21T18:03:00.738 DEBUG:teuthology.task.install:project ceph 2024-09-21T18:03:00.738 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}} 2024-09-21T18:03:00.738 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'} 2024-09-21T18:03:00.738 INFO:teuthology.task.install:Using flavor: default 2024-09-21T18:03:00.743 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T18:03:00.743 INFO:teuthology.task.install:extra packages: [] 2024-09-21T18:03:00.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-key list | grep Ceph 2024-09-21T18:03:00.744 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-key list | grep Ceph 2024-09-21T18:03:00.744 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-key list | grep Ceph 2024-09-21T18:03:00.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-key list | grep Ceph 2024-09-21T18:03:00.808 INFO:teuthology.orchestra.run.smithi070.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T18:03:00.809 INFO:teuthology.orchestra.run.smithi160.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T18:03:00.809 INFO:teuthology.orchestra.run.smithi040.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T18:03:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T18:03:00.835 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T18:03:00.836 INFO:teuthology.orchestra.run.smithi160.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T18:03:00.836 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-21T18:03:00.836 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:03:00.840 INFO:teuthology.orchestra.run.smithi070.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T18:03:00.840 INFO:teuthology.orchestra.run.smithi070.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T18:03:00.841 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-21T18:03:00.841 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:03:00.844 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T18:03:00.844 INFO:teuthology.orchestra.run.smithi040.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T18:03:00.844 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-21T18:03:00.845 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:03:00.857 INFO:teuthology.orchestra.run.smithi029.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T18:03:00.857 INFO:teuthology.orchestra.run.smithi029.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T18:03:00.858 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-21T18:03:00.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:03:00.976 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default/ 2024-09-21T18:03:00.977 INFO:teuthology.task.install.deb:Package version is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:00.981 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default/ 2024-09-21T18:03:00.982 INFO:teuthology.task.install.deb:Package version is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:00.991 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default/ 2024-09-21T18:03:00.991 INFO:teuthology.task.install.deb:Package version is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:01.016 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default/ 2024-09-21T18:03:01.016 INFO:teuthology.task.install.deb:Package version is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:01.085 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:03:01.086 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T18:03:01.099 DEBUG:teuthology.orchestra.run.smithi160:> sudo apt-get update 2024-09-21T18:03:01.107 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:03:01.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T18:03:01.109 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:03:01.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T18:03:01.123 DEBUG:teuthology.orchestra.run.smithi070:> sudo apt-get update 2024-09-21T18:03:01.124 DEBUG:teuthology.orchestra.run.smithi040:> sudo apt-get update 2024-09-21T18:03:01.127 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:03:01.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T18:03:01.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo apt-get update 2024-09-21T18:03:01.296 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T18:03:01.296 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T18:03:01.305 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T18:03:01.321 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T18:03:01.322 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T18:03:01.343 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-21T18:03:01.354 INFO:teuthology.orchestra.run.smithi070.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T18:03:01.354 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T18:03:01.367 INFO:teuthology.orchestra.run.smithi160.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T18:03:01.368 INFO:teuthology.orchestra.run.smithi029.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T18:03:01.375 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-21T18:03:01.375 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-21T18:03:01.392 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-09-21T18:03:01.396 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T18:03:01.396 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T18:03:01.397 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-21T18:03:01.397 INFO:teuthology.orchestra.run.smithi040.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T18:03:01.398 INFO:teuthology.orchestra.run.smithi070.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T18:03:01.415 INFO:teuthology.orchestra.run.smithi070.stdout:Hit:7 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T18:03:01.416 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T18:03:01.416 INFO:teuthology.orchestra.run.smithi029.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T18:03:01.420 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-21T18:03:01.420 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-21T18:03:01.425 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-21T18:03:01.441 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T18:03:01.441 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:5 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T18:03:01.442 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-21T18:03:01.446 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-21T18:03:01.449 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,415 B] 2024-09-21T18:03:01.468 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T18:03:01.474 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-21T18:03:01.497 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T18:03:01.519 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T18:03:01.549 INFO:teuthology.orchestra.run.smithi160.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T18:03:01.579 INFO:teuthology.orchestra.run.smithi040.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T18:03:01.601 INFO:teuthology.orchestra.run.smithi029.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T18:03:01.642 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 28.6 kB in 0s (84.9 kB/s) 2024-09-21T18:03:01.722 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 28.6 kB in 0s (68.9 kB/s) 2024-09-21T18:03:01.786 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 28.6 kB in 1s (56.8 kB/s) 2024-09-21T18:03:01.822 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 28.6 kB in 0s (57.5 kB/s) 2024-09-21T18:03:02.628 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T18:03:02.645 DEBUG:teuthology.orchestra.run.smithi070:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5073-gc69a6b39-1jammy cephadm=19.3.0-5073-gc69a6b39-1jammy ceph-mds=19.3.0-5073-gc69a6b39-1jammy ceph-mgr=19.3.0-5073-gc69a6b39-1jammy ceph-common=19.3.0-5073-gc69a6b39-1jammy ceph-fuse=19.3.0-5073-gc69a6b39-1jammy ceph-test=19.3.0-5073-gc69a6b39-1jammy ceph-volume=19.3.0-5073-gc69a6b39-1jammy radosgw=19.3.0-5073-gc69a6b39-1jammy python3-rados=19.3.0-5073-gc69a6b39-1jammy python3-rgw=19.3.0-5073-gc69a6b39-1jammy python3-cephfs=19.3.0-5073-gc69a6b39-1jammy python3-rbd=19.3.0-5073-gc69a6b39-1jammy libcephfs2=19.3.0-5073-gc69a6b39-1jammy libcephfs-dev=19.3.0-5073-gc69a6b39-1jammy librados2=19.3.0-5073-gc69a6b39-1jammy librbd1=19.3.0-5073-gc69a6b39-1jammy rbd-fuse=19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:02.699 INFO:teuthology.orchestra.run.smithi070.stdout:Reading package lists... 2024-09-21T18:03:02.763 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T18:03:02.780 DEBUG:teuthology.orchestra.run.smithi040:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5073-gc69a6b39-1jammy cephadm=19.3.0-5073-gc69a6b39-1jammy ceph-mds=19.3.0-5073-gc69a6b39-1jammy ceph-mgr=19.3.0-5073-gc69a6b39-1jammy ceph-common=19.3.0-5073-gc69a6b39-1jammy ceph-fuse=19.3.0-5073-gc69a6b39-1jammy ceph-test=19.3.0-5073-gc69a6b39-1jammy ceph-volume=19.3.0-5073-gc69a6b39-1jammy radosgw=19.3.0-5073-gc69a6b39-1jammy python3-rados=19.3.0-5073-gc69a6b39-1jammy python3-rgw=19.3.0-5073-gc69a6b39-1jammy python3-cephfs=19.3.0-5073-gc69a6b39-1jammy python3-rbd=19.3.0-5073-gc69a6b39-1jammy libcephfs2=19.3.0-5073-gc69a6b39-1jammy libcephfs-dev=19.3.0-5073-gc69a6b39-1jammy librados2=19.3.0-5073-gc69a6b39-1jammy librbd1=19.3.0-5073-gc69a6b39-1jammy rbd-fuse=19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:02.815 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T18:03:02.832 DEBUG:teuthology.orchestra.run.smithi160:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5073-gc69a6b39-1jammy cephadm=19.3.0-5073-gc69a6b39-1jammy ceph-mds=19.3.0-5073-gc69a6b39-1jammy ceph-mgr=19.3.0-5073-gc69a6b39-1jammy ceph-common=19.3.0-5073-gc69a6b39-1jammy ceph-fuse=19.3.0-5073-gc69a6b39-1jammy ceph-test=19.3.0-5073-gc69a6b39-1jammy ceph-volume=19.3.0-5073-gc69a6b39-1jammy radosgw=19.3.0-5073-gc69a6b39-1jammy python3-rados=19.3.0-5073-gc69a6b39-1jammy python3-rgw=19.3.0-5073-gc69a6b39-1jammy python3-cephfs=19.3.0-5073-gc69a6b39-1jammy python3-rbd=19.3.0-5073-gc69a6b39-1jammy libcephfs2=19.3.0-5073-gc69a6b39-1jammy libcephfs-dev=19.3.0-5073-gc69a6b39-1jammy librados2=19.3.0-5073-gc69a6b39-1jammy librbd1=19.3.0-5073-gc69a6b39-1jammy rbd-fuse=19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:02.844 INFO:teuthology.orchestra.run.smithi040.stdout:Reading package lists... 2024-09-21T18:03:02.870 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T18:03:02.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-5073-gc69a6b39-1jammy cephadm=19.3.0-5073-gc69a6b39-1jammy ceph-mds=19.3.0-5073-gc69a6b39-1jammy ceph-mgr=19.3.0-5073-gc69a6b39-1jammy ceph-common=19.3.0-5073-gc69a6b39-1jammy ceph-fuse=19.3.0-5073-gc69a6b39-1jammy ceph-test=19.3.0-5073-gc69a6b39-1jammy ceph-volume=19.3.0-5073-gc69a6b39-1jammy radosgw=19.3.0-5073-gc69a6b39-1jammy python3-rados=19.3.0-5073-gc69a6b39-1jammy python3-rgw=19.3.0-5073-gc69a6b39-1jammy python3-cephfs=19.3.0-5073-gc69a6b39-1jammy python3-rbd=19.3.0-5073-gc69a6b39-1jammy libcephfs2=19.3.0-5073-gc69a6b39-1jammy libcephfs-dev=19.3.0-5073-gc69a6b39-1jammy librados2=19.3.0-5073-gc69a6b39-1jammy librbd1=19.3.0-5073-gc69a6b39-1jammy rbd-fuse=19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:03:02.893 INFO:teuthology.orchestra.run.smithi070.stdout:Building dependency tree... 2024-09-21T18:03:02.894 INFO:teuthology.orchestra.run.smithi070.stdout:Reading state information... 2024-09-21T18:03:02.896 INFO:teuthology.orchestra.run.smithi160.stdout:Reading package lists... 2024-09-21T18:03:02.948 INFO:teuthology.orchestra.run.smithi029.stdout:Reading package lists... 2024-09-21T18:03:03.060 INFO:teuthology.orchestra.run.smithi040.stdout:Building dependency tree... 2024-09-21T18:03:03.061 INFO:teuthology.orchestra.run.smithi040.stdout:Reading state information... 2024-09-21T18:03:03.087 INFO:teuthology.orchestra.run.smithi070.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T18:03:03.087 INFO:teuthology.orchestra.run.smithi070.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T18:03:03.087 INFO:teuthology.orchestra.run.smithi070.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T18:03:03.087 INFO:teuthology.orchestra.run.smithi070.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T18:03:03.088 INFO:teuthology.orchestra.run.smithi070.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T18:03:03.088 INFO:teuthology.orchestra.run.smithi070.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T18:03:03.088 INFO:teuthology.orchestra.run.smithi070.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T18:03:03.089 INFO:teuthology.orchestra.run.smithi070.stdout:The following additional packages will be installed: 2024-09-21T18:03:03.089 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T18:03:03.089 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T18:03:03.089 INFO:teuthology.orchestra.run.smithi070.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-21T18:03:03.089 INFO:teuthology.orchestra.run.smithi070.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-21T18:03:03.090 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-21T18:03:03.090 INFO:teuthology.orchestra.run.smithi070.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-21T18:03:03.090 INFO:teuthology.orchestra.run.smithi070.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T18:03:03.091 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet zip 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout:Suggested packages: 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T18:03:03.092 INFO:teuthology.orchestra.run.smithi070.stdout:Recommended packages: 2024-09-21T18:03:03.093 INFO:teuthology.orchestra.run.smithi070.stdout: btrfs-tools 2024-09-21T18:03:03.107 INFO:teuthology.orchestra.run.smithi160.stdout:Building dependency tree... 2024-09-21T18:03:03.108 INFO:teuthology.orchestra.run.smithi160.stdout:Reading state information... 2024-09-21T18:03:03.163 INFO:teuthology.orchestra.run.smithi029.stdout:Building dependency tree... 2024-09-21T18:03:03.163 INFO:teuthology.orchestra.run.smithi029.stdout:Reading state information... 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout:The following NEW packages will be installed: 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-21T18:03:03.195 INFO:teuthology.orchestra.run.smithi070.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T18:03:03.196 INFO:teuthology.orchestra.run.smithi070.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T18:03:03.196 INFO:teuthology.orchestra.run.smithi070.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T18:03:03.196 INFO:teuthology.orchestra.run.smithi070.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T18:03:03.196 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T18:03:03.197 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T18:03:03.198 INFO:teuthology.orchestra.run.smithi070.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T18:03:03.198 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T18:03:03.198 INFO:teuthology.orchestra.run.smithi070.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-21T18:03:03.282 INFO:teuthology.orchestra.run.smithi040.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T18:03:03.282 INFO:teuthology.orchestra.run.smithi040.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T18:03:03.282 INFO:teuthology.orchestra.run.smithi040.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T18:03:03.282 INFO:teuthology.orchestra.run.smithi040.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T18:03:03.283 INFO:teuthology.orchestra.run.smithi040.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T18:03:03.283 INFO:teuthology.orchestra.run.smithi040.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T18:03:03.283 INFO:teuthology.orchestra.run.smithi040.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T18:03:03.283 INFO:teuthology.orchestra.run.smithi040.stdout:The following additional packages will be installed: 2024-09-21T18:03:03.284 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T18:03:03.284 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T18:03:03.284 INFO:teuthology.orchestra.run.smithi040.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-21T18:03:03.284 INFO:teuthology.orchestra.run.smithi040.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-21T18:03:03.284 INFO:teuthology.orchestra.run.smithi040.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-21T18:03:03.285 INFO:teuthology.orchestra.run.smithi040.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-21T18:03:03.285 INFO:teuthology.orchestra.run.smithi040.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-21T18:03:03.285 INFO:teuthology.orchestra.run.smithi040.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-21T18:03:03.285 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-21T18:03:03.285 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T18:03:03.286 INFO:teuthology.orchestra.run.smithi040.stdout: xmlstarlet zip 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout:Suggested packages: 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout:Recommended packages: 2024-09-21T18:03:03.287 INFO:teuthology.orchestra.run.smithi040.stdout: btrfs-tools 2024-09-21T18:03:03.317 INFO:teuthology.orchestra.run.smithi160.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T18:03:03.317 INFO:teuthology.orchestra.run.smithi160.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T18:03:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T18:03:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T18:03:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T18:03:03.318 INFO:teuthology.orchestra.run.smithi160.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T18:03:03.318 INFO:teuthology.orchestra.run.smithi160.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T18:03:03.319 INFO:teuthology.orchestra.run.smithi160.stdout:The following additional packages will be installed: 2024-09-21T18:03:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T18:03:03.319 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T18:03:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-21T18:03:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-21T18:03:03.320 INFO:teuthology.orchestra.run.smithi160.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-21T18:03:03.321 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: xmlstarlet zip 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout:Suggested packages: 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T18:03:03.322 INFO:teuthology.orchestra.run.smithi160.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T18:03:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T18:03:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T18:03:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T18:03:03.323 INFO:teuthology.orchestra.run.smithi160.stdout:Recommended packages: 2024-09-21T18:03:03.323 INFO:teuthology.orchestra.run.smithi160.stdout: btrfs-tools 2024-09-21T18:03:03.340 INFO:teuthology.orchestra.run.smithi070.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T18:03:03.340 INFO:teuthology.orchestra.run.smithi070.stdout:Need to get 244 MB of archives. 2024-09-21T18:03:03.340 INFO:teuthology.orchestra.run.smithi070.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-21T18:03:03.340 INFO:teuthology.orchestra.run.smithi070.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5073-gc69a6b39-1jammy [3,624 kB] 2024-09-21T18:03:03.382 INFO:teuthology.orchestra.run.smithi029.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T18:03:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T18:03:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T18:03:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T18:03:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T18:03:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T18:03:03.383 INFO:teuthology.orchestra.run.smithi029.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T18:03:03.384 INFO:teuthology.orchestra.run.smithi029.stdout:The following additional packages will be installed: 2024-09-21T18:03:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T18:03:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T18:03:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-21T18:03:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-21T18:03:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-21T18:03:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-21T18:03:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T18:03:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet zip 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi070.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout:Suggested packages: 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T18:03:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout:Recommended packages: 2024-09-21T18:03:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: btrfs-tools 2024-09-21T18:03:03.398 INFO:teuthology.orchestra.run.smithi040.stdout:The following NEW packages will be installed: 2024-09-21T18:03:03.398 INFO:teuthology.orchestra.run.smithi040.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T18:03:03.398 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T18:03:03.398 INFO:teuthology.orchestra.run.smithi040.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T18:03:03.399 INFO:teuthology.orchestra.run.smithi040.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T18:03:03.399 INFO:teuthology.orchestra.run.smithi040.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-21T18:03:03.399 INFO:teuthology.orchestra.run.smithi040.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T18:03:03.399 INFO:teuthology.orchestra.run.smithi040.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T18:03:03.402 INFO:teuthology.orchestra.run.smithi040.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T18:03:03.402 INFO:teuthology.orchestra.run.smithi040.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T18:03:03.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T18:03:03.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T18:03:03.402 INFO:teuthology.orchestra.run.smithi040.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T18:03:03.403 INFO:teuthology.orchestra.run.smithi040.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout:The following NEW packages will be installed: 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-21T18:03:03.426 INFO:teuthology.orchestra.run.smithi160.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T18:03:03.427 INFO:teuthology.orchestra.run.smithi160.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T18:03:03.427 INFO:teuthology.orchestra.run.smithi160.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T18:03:03.427 INFO:teuthology.orchestra.run.smithi160.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T18:03:03.428 INFO:teuthology.orchestra.run.smithi160.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T18:03:03.429 INFO:teuthology.orchestra.run.smithi160.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T18:03:03.429 INFO:teuthology.orchestra.run.smithi160.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-21T18:03:03.484 INFO:teuthology.orchestra.run.smithi160.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T18:03:03.485 INFO:teuthology.orchestra.run.smithi160.stdout:Need to get 244 MB of archives. 2024-09-21T18:03:03.485 INFO:teuthology.orchestra.run.smithi160.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-21T18:03:03.485 INFO:teuthology.orchestra.run.smithi160.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T18:03:03.496 INFO:teuthology.orchestra.run.smithi029.stdout:The following NEW packages will be installed: 2024-09-21T18:03:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T18:03:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T18:03:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T18:03:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T18:03:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-21T18:03:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T18:03:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T18:03:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-21T18:03:03.543 INFO:teuthology.orchestra.run.smithi070.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5073-gc69a6b39-1jammy [3,335 kB] 2024-09-21T18:03:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T18:03:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:Need to get 244 MB of archives. 2024-09-21T18:03:03.547 INFO:teuthology.orchestra.run.smithi040.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-21T18:03:03.548 INFO:teuthology.orchestra.run.smithi040.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5073-gc69a6b39-1jammy [3,624 kB] 2024-09-21T18:03:03.561 INFO:teuthology.orchestra.run.smithi029.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T18:03:03.561 INFO:teuthology.orchestra.run.smithi029.stdout:Need to get 244 MB of archives. 2024-09-21T18:03:03.561 INFO:teuthology.orchestra.run.smithi029.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-21T18:03:03.561 INFO:teuthology.orchestra.run.smithi029.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T18:03:03.570 INFO:teuthology.orchestra.run.smithi160.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5073-gc69a6b39-1jammy [3,624 kB] 2024-09-21T18:03:03.572 INFO:teuthology.orchestra.run.smithi160.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T18:03:03.575 INFO:teuthology.orchestra.run.smithi160.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T18:03:03.590 INFO:teuthology.orchestra.run.smithi040.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T18:03:03.592 INFO:teuthology.orchestra.run.smithi160.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T18:03:03.616 INFO:teuthology.orchestra.run.smithi070.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5073-gc69a6b39-1jammy [722 kB] 2024-09-21T18:03:03.631 INFO:teuthology.orchestra.run.smithi070.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5073-gc69a6b39-1jammy [356 kB] 2024-09-21T18:03:03.637 INFO:teuthology.orchestra.run.smithi070.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5073-gc69a6b39-1jammy [32.8 kB] 2024-09-21T18:03:03.637 INFO:teuthology.orchestra.run.smithi070.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5073-gc69a6b39-1jammy [180 kB] 2024-09-21T18:03:03.641 INFO:teuthology.orchestra.run.smithi070.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5073-gc69a6b39-1jammy [72.9 kB] 2024-09-21T18:03:03.642 INFO:teuthology.orchestra.run.smithi070.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5073-gc69a6b39-1jammy [339 kB] 2024-09-21T18:03:03.645 INFO:teuthology.orchestra.run.smithi029.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-5073-gc69a6b39-1jammy [3,624 kB] 2024-09-21T18:03:03.650 INFO:teuthology.orchestra.run.smithi029.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T18:03:03.651 INFO:teuthology.orchestra.run.smithi070.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5073-gc69a6b39-1jammy [7,966 kB] 2024-09-21T18:03:03.652 INFO:teuthology.orchestra.run.smithi029.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T18:03:03.653 INFO:teuthology.orchestra.run.smithi160.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T18:03:03.658 INFO:teuthology.orchestra.run.smithi160.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T18:03:03.670 INFO:teuthology.orchestra.run.smithi160.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T18:03:03.670 INFO:teuthology.orchestra.run.smithi029.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T18:03:03.673 INFO:teuthology.orchestra.run.smithi160.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-21T18:03:03.675 INFO:teuthology.orchestra.run.smithi160.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T18:03:03.676 INFO:teuthology.orchestra.run.smithi160.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T18:03:03.676 INFO:teuthology.orchestra.run.smithi160.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T18:03:03.684 INFO:teuthology.orchestra.run.smithi160.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T18:03:03.686 INFO:teuthology.orchestra.run.smithi160.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T18:03:03.688 INFO:teuthology.orchestra.run.smithi160.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T18:03:03.691 INFO:teuthology.orchestra.run.smithi160.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T18:03:03.692 INFO:teuthology.orchestra.run.smithi160.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T18:03:03.708 INFO:teuthology.orchestra.run.smithi160.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T18:03:03.711 INFO:teuthology.orchestra.run.smithi160.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T18:03:03.712 INFO:teuthology.orchestra.run.smithi160.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T18:03:03.713 INFO:teuthology.orchestra.run.smithi160.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T18:03:03.713 INFO:teuthology.orchestra.run.smithi160.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T18:03:03.715 INFO:teuthology.orchestra.run.smithi160.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T18:03:03.715 INFO:teuthology.orchestra.run.smithi160.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T18:03:03.715 INFO:teuthology.orchestra.run.smithi160.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T18:03:03.725 INFO:teuthology.orchestra.run.smithi160.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T18:03:03.726 INFO:teuthology.orchestra.run.smithi160.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T18:03:03.729 INFO:teuthology.orchestra.run.smithi029.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T18:03:03.734 INFO:teuthology.orchestra.run.smithi029.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T18:03:03.742 INFO:teuthology.orchestra.run.smithi160.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T18:03:03.743 INFO:teuthology.orchestra.run.smithi160.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T18:03:03.746 INFO:teuthology.orchestra.run.smithi160.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T18:03:03.746 INFO:teuthology.orchestra.run.smithi160.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T18:03:03.746 INFO:teuthology.orchestra.run.smithi160.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T18:03:03.747 INFO:teuthology.orchestra.run.smithi029.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T18:03:03.747 INFO:teuthology.orchestra.run.smithi160.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T18:03:03.748 INFO:teuthology.orchestra.run.smithi160.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T18:03:03.748 INFO:teuthology.orchestra.run.smithi160.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T18:03:03.751 INFO:teuthology.orchestra.run.smithi029.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-21T18:03:03.752 INFO:teuthology.orchestra.run.smithi029.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T18:03:03.753 INFO:teuthology.orchestra.run.smithi029.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T18:03:03.754 INFO:teuthology.orchestra.run.smithi029.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T18:03:03.759 INFO:teuthology.orchestra.run.smithi160.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T18:03:03.761 INFO:teuthology.orchestra.run.smithi160.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T18:03:03.762 INFO:teuthology.orchestra.run.smithi029.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T18:03:03.765 INFO:teuthology.orchestra.run.smithi029.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T18:03:03.767 INFO:teuthology.orchestra.run.smithi029.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T18:03:03.769 INFO:teuthology.orchestra.run.smithi029.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T18:03:03.769 INFO:teuthology.orchestra.run.smithi029.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T18:03:03.776 INFO:teuthology.orchestra.run.smithi160.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T18:03:03.777 INFO:teuthology.orchestra.run.smithi160.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T18:03:03.782 INFO:teuthology.orchestra.run.smithi160.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T18:03:03.782 INFO:teuthology.orchestra.run.smithi160.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T18:03:03.783 INFO:teuthology.orchestra.run.smithi160.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T18:03:03.783 INFO:teuthology.orchestra.run.smithi160.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T18:03:03.785 INFO:teuthology.orchestra.run.smithi160.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T18:03:03.785 INFO:teuthology.orchestra.run.smithi029.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T18:03:03.787 INFO:teuthology.orchestra.run.smithi160.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T18:03:03.787 INFO:teuthology.orchestra.run.smithi029.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T18:03:03.789 INFO:teuthology.orchestra.run.smithi029.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T18:03:03.790 INFO:teuthology.orchestra.run.smithi029.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T18:03:03.790 INFO:teuthology.orchestra.run.smithi029.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T18:03:03.793 INFO:teuthology.orchestra.run.smithi160.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T18:03:03.795 INFO:teuthology.orchestra.run.smithi160.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T18:03:03.802 INFO:teuthology.orchestra.run.smithi029.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T18:03:03.802 INFO:teuthology.orchestra.run.smithi029.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T18:03:03.803 INFO:teuthology.orchestra.run.smithi029.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T18:03:03.803 INFO:teuthology.orchestra.run.smithi029.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T18:03:03.804 INFO:teuthology.orchestra.run.smithi029.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T18:03:03.805 INFO:teuthology.orchestra.run.smithi160.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T18:03:03.807 INFO:teuthology.orchestra.run.smithi160.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T18:03:03.808 INFO:teuthology.orchestra.run.smithi160.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T18:03:03.809 INFO:teuthology.orchestra.run.smithi160.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T18:03:03.810 INFO:teuthology.orchestra.run.smithi070.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T18:03:03.811 INFO:teuthology.orchestra.run.smithi160.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T18:03:03.818 INFO:teuthology.orchestra.run.smithi040.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5073-gc69a6b39-1jammy [3,335 kB] 2024-09-21T18:03:03.819 INFO:teuthology.orchestra.run.smithi029.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T18:03:03.820 INFO:teuthology.orchestra.run.smithi029.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T18:03:03.822 INFO:teuthology.orchestra.run.smithi070.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T18:03:03.823 INFO:teuthology.orchestra.run.smithi029.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T18:03:03.823 INFO:teuthology.orchestra.run.smithi029.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T18:03:03.823 INFO:teuthology.orchestra.run.smithi029.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T18:03:03.836 INFO:teuthology.orchestra.run.smithi029.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T18:03:03.837 INFO:teuthology.orchestra.run.smithi029.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T18:03:03.838 INFO:teuthology.orchestra.run.smithi029.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T18:03:03.838 INFO:teuthology.orchestra.run.smithi029.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T18:03:03.839 INFO:teuthology.orchestra.run.smithi029.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T18:03:03.855 INFO:teuthology.orchestra.run.smithi029.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T18:03:03.855 INFO:teuthology.orchestra.run.smithi029.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T18:03:03.861 INFO:teuthology.orchestra.run.smithi029.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T18:03:03.861 INFO:teuthology.orchestra.run.smithi029.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T18:03:03.862 INFO:teuthology.orchestra.run.smithi029.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T18:03:03.865 INFO:teuthology.orchestra.run.smithi160.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T18:03:03.866 INFO:teuthology.orchestra.run.smithi160.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T18:03:03.867 INFO:teuthology.orchestra.run.smithi160.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T18:03:03.868 INFO:teuthology.orchestra.run.smithi160.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T18:03:03.869 INFO:teuthology.orchestra.run.smithi160.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T18:03:03.873 INFO:teuthology.orchestra.run.smithi029.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T18:03:03.875 INFO:teuthology.orchestra.run.smithi029.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T18:03:03.877 INFO:teuthology.orchestra.run.smithi029.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T18:03:03.877 INFO:teuthology.orchestra.run.smithi029.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T18:03:03.879 INFO:teuthology.orchestra.run.smithi029.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T18:03:03.886 INFO:teuthology.orchestra.run.smithi160.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T18:03:03.888 INFO:teuthology.orchestra.run.smithi160.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T18:03:03.888 INFO:teuthology.orchestra.run.smithi160.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T18:03:03.891 INFO:teuthology.orchestra.run.smithi029.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T18:03:03.893 INFO:teuthology.orchestra.run.smithi029.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T18:03:03.893 INFO:teuthology.orchestra.run.smithi029.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T18:03:03.894 INFO:teuthology.orchestra.run.smithi029.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T18:03:03.897 INFO:teuthology.orchestra.run.smithi029.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T18:03:03.899 INFO:teuthology.orchestra.run.smithi070.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5073-gc69a6b39-1jammy [112 kB] 2024-09-21T18:03:03.902 INFO:teuthology.orchestra.run.smithi070.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5073-gc69a6b39-1jammy [473 kB] 2024-09-21T18:03:03.903 INFO:teuthology.orchestra.run.smithi160.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T18:03:03.903 INFO:teuthology.orchestra.run.smithi160.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T18:03:03.904 INFO:teuthology.orchestra.run.smithi160.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T18:03:03.904 INFO:teuthology.orchestra.run.smithi070.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T18:03:03.905 INFO:teuthology.orchestra.run.smithi160.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T18:03:03.905 INFO:teuthology.orchestra.run.smithi160.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T18:03:03.905 INFO:teuthology.orchestra.run.smithi160.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T18:03:03.911 INFO:teuthology.orchestra.run.smithi070.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5073-gc69a6b39-1jammy [28.3 MB] 2024-09-21T18:03:03.918 INFO:teuthology.orchestra.run.smithi160.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T18:03:03.919 INFO:teuthology.orchestra.run.smithi160.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T18:03:03.922 INFO:teuthology.orchestra.run.smithi160.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T18:03:03.935 INFO:teuthology.orchestra.run.smithi160.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T18:03:03.936 INFO:teuthology.orchestra.run.smithi160.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T18:03:03.937 INFO:teuthology.orchestra.run.smithi160.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T18:03:03.940 INFO:teuthology.orchestra.run.smithi160.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T18:03:03.942 INFO:teuthology.orchestra.run.smithi160.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T18:03:03.943 INFO:teuthology.orchestra.run.smithi160.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T18:03:03.943 INFO:teuthology.orchestra.run.smithi160.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T18:03:03.949 INFO:teuthology.orchestra.run.smithi029.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T18:03:03.950 INFO:teuthology.orchestra.run.smithi029.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T18:03:03.953 INFO:teuthology.orchestra.run.smithi029.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T18:03:03.954 INFO:teuthology.orchestra.run.smithi029.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T18:03:03.954 INFO:teuthology.orchestra.run.smithi029.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T18:03:03.954 INFO:teuthology.orchestra.run.smithi160.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T18:03:03.955 INFO:teuthology.orchestra.run.smithi160.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T18:03:03.958 INFO:teuthology.orchestra.run.smithi160.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T18:03:03.971 INFO:teuthology.orchestra.run.smithi029.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T18:03:03.971 INFO:teuthology.orchestra.run.smithi160.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T18:03:03.972 INFO:teuthology.orchestra.run.smithi040.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5073-gc69a6b39-1jammy [722 kB] 2024-09-21T18:03:03.972 INFO:teuthology.orchestra.run.smithi160.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T18:03:03.972 INFO:teuthology.orchestra.run.smithi160.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T18:03:03.973 INFO:teuthology.orchestra.run.smithi029.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T18:03:03.973 INFO:teuthology.orchestra.run.smithi029.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T18:03:03.974 INFO:teuthology.orchestra.run.smithi160.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T18:03:03.975 INFO:teuthology.orchestra.run.smithi160.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T18:03:03.981 INFO:teuthology.orchestra.run.smithi160.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T18:03:03.981 INFO:teuthology.orchestra.run.smithi160.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T18:03:03.987 INFO:teuthology.orchestra.run.smithi029.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T18:03:03.987 INFO:teuthology.orchestra.run.smithi029.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T18:03:03.988 INFO:teuthology.orchestra.run.smithi029.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T18:03:03.988 INFO:teuthology.orchestra.run.smithi029.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T18:03:03.989 INFO:teuthology.orchestra.run.smithi160.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T18:03:03.989 INFO:teuthology.orchestra.run.smithi029.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T18:03:03.989 INFO:teuthology.orchestra.run.smithi029.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T18:03:03.990 INFO:teuthology.orchestra.run.smithi160.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T18:03:03.991 INFO:teuthology.orchestra.run.smithi160.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T18:03:04.003 INFO:teuthology.orchestra.run.smithi029.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T18:03:04.003 INFO:teuthology.orchestra.run.smithi029.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T18:03:04.006 INFO:teuthology.orchestra.run.smithi029.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T18:03:04.012 INFO:teuthology.orchestra.run.smithi040.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T18:03:04.018 INFO:teuthology.orchestra.run.smithi040.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5073-gc69a6b39-1jammy [356 kB] 2024-09-21T18:03:04.020 INFO:teuthology.orchestra.run.smithi029.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T18:03:04.021 INFO:teuthology.orchestra.run.smithi029.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T18:03:04.022 INFO:teuthology.orchestra.run.smithi029.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T18:03:04.025 INFO:teuthology.orchestra.run.smithi029.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T18:03:04.027 INFO:teuthology.orchestra.run.smithi040.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5073-gc69a6b39-1jammy [32.8 kB] 2024-09-21T18:03:04.027 INFO:teuthology.orchestra.run.smithi040.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T18:03:04.027 INFO:teuthology.orchestra.run.smithi029.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T18:03:04.028 INFO:teuthology.orchestra.run.smithi029.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T18:03:04.028 INFO:teuthology.orchestra.run.smithi029.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T18:03:04.029 INFO:teuthology.orchestra.run.smithi040.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5073-gc69a6b39-1jammy [180 kB] 2024-09-21T18:03:04.035 INFO:teuthology.orchestra.run.smithi040.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5073-gc69a6b39-1jammy [72.9 kB] 2024-09-21T18:03:04.037 INFO:teuthology.orchestra.run.smithi029.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T18:03:04.037 INFO:teuthology.orchestra.run.smithi040.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5073-gc69a6b39-1jammy [339 kB] 2024-09-21T18:03:04.038 INFO:teuthology.orchestra.run.smithi029.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T18:03:04.040 INFO:teuthology.orchestra.run.smithi029.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T18:03:04.054 INFO:teuthology.orchestra.run.smithi029.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T18:03:04.055 INFO:teuthology.orchestra.run.smithi029.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T18:03:04.055 INFO:teuthology.orchestra.run.smithi029.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T18:03:04.056 INFO:teuthology.orchestra.run.smithi029.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T18:03:04.057 INFO:teuthology.orchestra.run.smithi029.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T18:03:04.063 INFO:teuthology.orchestra.run.smithi029.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T18:03:04.063 INFO:teuthology.orchestra.run.smithi029.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T18:03:04.071 INFO:teuthology.orchestra.run.smithi029.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T18:03:04.073 INFO:teuthology.orchestra.run.smithi029.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T18:03:04.074 INFO:teuthology.orchestra.run.smithi029.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T18:03:04.074 INFO:teuthology.orchestra.run.smithi040.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5073-gc69a6b39-1jammy [7,966 kB] 2024-09-21T18:03:04.106 INFO:teuthology.orchestra.run.smithi040.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T18:03:04.114 INFO:teuthology.orchestra.run.smithi160.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5073-gc69a6b39-1jammy [3,335 kB] 2024-09-21T18:03:04.133 INFO:teuthology.orchestra.run.smithi070.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T18:03:04.144 INFO:teuthology.orchestra.run.smithi070.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T18:03:04.174 INFO:teuthology.orchestra.run.smithi070.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T18:03:04.182 INFO:teuthology.orchestra.run.smithi070.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-21T18:03:04.185 INFO:teuthology.orchestra.run.smithi070.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T18:03:04.185 INFO:teuthology.orchestra.run.smithi070.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T18:03:04.187 INFO:teuthology.orchestra.run.smithi070.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T18:03:04.204 INFO:teuthology.orchestra.run.smithi070.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T18:03:04.209 INFO:teuthology.orchestra.run.smithi070.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T18:03:04.215 INFO:teuthology.orchestra.run.smithi070.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T18:03:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T18:03:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T18:03:04.300 INFO:teuthology.orchestra.run.smithi070.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T18:03:04.304 INFO:teuthology.orchestra.run.smithi070.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T18:03:04.308 INFO:teuthology.orchestra.run.smithi070.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T18:03:04.309 INFO:teuthology.orchestra.run.smithi070.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T18:03:04.310 INFO:teuthology.orchestra.run.smithi070.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T18:03:04.312 INFO:teuthology.orchestra.run.smithi070.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T18:03:04.312 INFO:teuthology.orchestra.run.smithi070.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T18:03:04.334 INFO:teuthology.orchestra.run.smithi040.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T18:03:04.345 INFO:teuthology.orchestra.run.smithi040.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T18:03:04.375 INFO:teuthology.orchestra.run.smithi040.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T18:03:04.378 INFO:teuthology.orchestra.run.smithi070.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T18:03:04.383 INFO:teuthology.orchestra.run.smithi040.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-21T18:03:04.386 INFO:teuthology.orchestra.run.smithi040.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T18:03:04.387 INFO:teuthology.orchestra.run.smithi040.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T18:03:04.388 INFO:teuthology.orchestra.run.smithi040.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T18:03:04.395 INFO:teuthology.orchestra.run.smithi070.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T18:03:04.396 INFO:teuthology.orchestra.run.smithi070.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T18:03:04.397 INFO:teuthology.orchestra.run.smithi070.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T18:03:04.398 INFO:teuthology.orchestra.run.smithi070.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T18:03:04.403 INFO:teuthology.orchestra.run.smithi070.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T18:03:04.404 INFO:teuthology.orchestra.run.smithi070.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T18:03:04.405 INFO:teuthology.orchestra.run.smithi040.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T18:03:04.410 INFO:teuthology.orchestra.run.smithi040.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T18:03:04.417 INFO:teuthology.orchestra.run.smithi040.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T18:03:04.460 INFO:teuthology.orchestra.run.smithi070.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T18:03:04.461 INFO:teuthology.orchestra.run.smithi070.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T18:03:04.463 INFO:teuthology.orchestra.run.smithi070.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T18:03:04.463 INFO:teuthology.orchestra.run.smithi040.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5073-gc69a6b39-1jammy [112 kB] 2024-09-21T18:03:04.466 INFO:teuthology.orchestra.run.smithi040.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5073-gc69a6b39-1jammy [473 kB] 2024-09-21T18:03:04.476 INFO:teuthology.orchestra.run.smithi040.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5073-gc69a6b39-1jammy [28.3 MB] 2024-09-21T18:03:04.496 INFO:teuthology.orchestra.run.smithi040.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T18:03:04.496 INFO:teuthology.orchestra.run.smithi040.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T18:03:04.501 INFO:teuthology.orchestra.run.smithi040.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T18:03:04.505 INFO:teuthology.orchestra.run.smithi040.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T18:03:04.509 INFO:teuthology.orchestra.run.smithi040.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T18:03:04.510 INFO:teuthology.orchestra.run.smithi040.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T18:03:04.511 INFO:teuthology.orchestra.run.smithi040.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T18:03:04.513 INFO:teuthology.orchestra.run.smithi040.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T18:03:04.513 INFO:teuthology.orchestra.run.smithi040.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T18:03:04.526 INFO:teuthology.orchestra.run.smithi160.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5073-gc69a6b39-1jammy [722 kB] 2024-09-21T18:03:04.541 INFO:teuthology.orchestra.run.smithi070.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T18:03:04.542 INFO:teuthology.orchestra.run.smithi070.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T18:03:04.544 INFO:teuthology.orchestra.run.smithi070.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T18:03:04.545 INFO:teuthology.orchestra.run.smithi070.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T18:03:04.546 INFO:teuthology.orchestra.run.smithi070.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T18:03:04.557 INFO:teuthology.orchestra.run.smithi070.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T18:03:04.580 INFO:teuthology.orchestra.run.smithi040.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T18:03:04.596 INFO:teuthology.orchestra.run.smithi040.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T18:03:04.597 INFO:teuthology.orchestra.run.smithi040.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T18:03:04.597 INFO:teuthology.orchestra.run.smithi040.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T18:03:04.598 INFO:teuthology.orchestra.run.smithi040.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T18:03:04.603 INFO:teuthology.orchestra.run.smithi040.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T18:03:04.604 INFO:teuthology.orchestra.run.smithi040.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T18:03:04.608 INFO:teuthology.orchestra.run.smithi160.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5073-gc69a6b39-1jammy [356 kB] 2024-09-21T18:03:04.623 INFO:teuthology.orchestra.run.smithi070.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T18:03:04.624 INFO:teuthology.orchestra.run.smithi070.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T18:03:04.625 INFO:teuthology.orchestra.run.smithi070.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T18:03:04.627 INFO:teuthology.orchestra.run.smithi070.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T18:03:04.662 INFO:teuthology.orchestra.run.smithi040.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T18:03:04.662 INFO:teuthology.orchestra.run.smithi040.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T18:03:04.664 INFO:teuthology.orchestra.run.smithi040.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T18:03:04.704 INFO:teuthology.orchestra.run.smithi070.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T18:03:04.705 INFO:teuthology.orchestra.run.smithi070.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T18:03:04.708 INFO:teuthology.orchestra.run.smithi070.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T18:03:04.710 INFO:teuthology.orchestra.run.smithi070.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T18:03:04.714 INFO:teuthology.orchestra.run.smithi070.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T18:03:04.715 INFO:teuthology.orchestra.run.smithi070.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T18:03:04.733 INFO:teuthology.orchestra.run.smithi160.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5073-gc69a6b39-1jammy [32.8 kB] 2024-09-21T18:03:04.734 INFO:teuthology.orchestra.run.smithi160.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5073-gc69a6b39-1jammy [180 kB] 2024-09-21T18:03:04.738 INFO:teuthology.orchestra.run.smithi160.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5073-gc69a6b39-1jammy [72.9 kB] 2024-09-21T18:03:04.740 INFO:teuthology.orchestra.run.smithi160.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5073-gc69a6b39-1jammy [339 kB] 2024-09-21T18:03:04.742 INFO:teuthology.orchestra.run.smithi040.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T18:03:04.743 INFO:teuthology.orchestra.run.smithi040.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T18:03:04.745 INFO:teuthology.orchestra.run.smithi040.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T18:03:04.747 INFO:teuthology.orchestra.run.smithi160.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5073-gc69a6b39-1jammy [7,966 kB] 2024-09-21T18:03:04.747 INFO:teuthology.orchestra.run.smithi040.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T18:03:04.748 INFO:teuthology.orchestra.run.smithi040.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T18:03:04.758 INFO:teuthology.orchestra.run.smithi040.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T18:03:04.786 INFO:teuthology.orchestra.run.smithi070.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T18:03:04.790 INFO:teuthology.orchestra.run.smithi070.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T18:03:04.824 INFO:teuthology.orchestra.run.smithi040.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T18:03:04.825 INFO:teuthology.orchestra.run.smithi040.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T18:03:04.826 INFO:teuthology.orchestra.run.smithi040.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T18:03:04.828 INFO:teuthology.orchestra.run.smithi040.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T18:03:04.905 INFO:teuthology.orchestra.run.smithi040.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T18:03:04.907 INFO:teuthology.orchestra.run.smithi040.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T18:03:04.909 INFO:teuthology.orchestra.run.smithi040.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T18:03:04.911 INFO:teuthology.orchestra.run.smithi040.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T18:03:04.915 INFO:teuthology.orchestra.run.smithi040.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T18:03:04.916 INFO:teuthology.orchestra.run.smithi040.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T18:03:04.986 INFO:teuthology.orchestra.run.smithi040.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T18:03:04.991 INFO:teuthology.orchestra.run.smithi040.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T18:03:04.993 INFO:teuthology.orchestra.run.smithi070.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5073-gc69a6b39-1jammy [5,211 kB] 2024-09-21T18:03:04.995 INFO:teuthology.orchestra.run.smithi070.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T18:03:04.997 INFO:teuthology.orchestra.run.smithi070.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T18:03:05.001 INFO:teuthology.orchestra.run.smithi070.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T18:03:05.001 INFO:teuthology.orchestra.run.smithi070.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T18:03:05.004 INFO:teuthology.orchestra.run.smithi070.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T18:03:05.042 INFO:teuthology.orchestra.run.smithi029.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-5073-gc69a6b39-1jammy [3,335 kB] 2024-09-21T18:03:05.047 INFO:teuthology.orchestra.run.smithi070.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T18:03:05.050 INFO:teuthology.orchestra.run.smithi070.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T18:03:05.051 INFO:teuthology.orchestra.run.smithi070.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T18:03:05.120 INFO:teuthology.orchestra.run.smithi070.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T18:03:05.120 INFO:teuthology.orchestra.run.smithi070.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T18:03:05.125 INFO:teuthology.orchestra.run.smithi070.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5073-gc69a6b39-1jammy [252 kB] 2024-09-21T18:03:05.131 INFO:teuthology.orchestra.run.smithi070.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5073-gc69a6b39-1jammy [124 kB] 2024-09-21T18:03:05.132 INFO:teuthology.orchestra.run.smithi070.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5073-gc69a6b39-1jammy [1,561 kB] 2024-09-21T18:03:05.153 INFO:teuthology.orchestra.run.smithi040.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T18:03:05.155 INFO:teuthology.orchestra.run.smithi040.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T18:03:05.157 INFO:teuthology.orchestra.run.smithi040.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T18:03:05.158 INFO:teuthology.orchestra.run.smithi040.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T18:03:05.160 INFO:teuthology.orchestra.run.smithi040.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T18:03:05.163 INFO:teuthology.orchestra.run.smithi070.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T18:03:05.164 INFO:teuthology.orchestra.run.smithi070.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T18:03:05.165 INFO:teuthology.orchestra.run.smithi070.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T18:03:05.166 INFO:teuthology.orchestra.run.smithi070.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T18:03:05.167 INFO:teuthology.orchestra.run.smithi070.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T18:03:05.168 INFO:teuthology.orchestra.run.smithi070.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T18:03:05.175 INFO:teuthology.orchestra.run.smithi070.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T18:03:05.192 INFO:teuthology.orchestra.run.smithi070.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5073-gc69a6b39-1jammy [6,312 kB] 2024-09-21T18:03:05.195 INFO:teuthology.orchestra.run.smithi040.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T18:03:05.198 INFO:teuthology.orchestra.run.smithi040.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T18:03:05.198 INFO:teuthology.orchestra.run.smithi040.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T18:03:05.247 INFO:teuthology.orchestra.run.smithi070.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T18:03:05.248 INFO:teuthology.orchestra.run.smithi070.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T18:03:05.249 INFO:teuthology.orchestra.run.smithi070.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T18:03:05.258 INFO:teuthology.orchestra.run.smithi040.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T18:03:05.259 INFO:teuthology.orchestra.run.smithi040.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T18:03:05.319 INFO:teuthology.orchestra.run.smithi040.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T18:03:05.320 INFO:teuthology.orchestra.run.smithi040.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T18:03:05.321 INFO:teuthology.orchestra.run.smithi040.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T18:03:05.322 INFO:teuthology.orchestra.run.smithi040.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T18:03:05.323 INFO:teuthology.orchestra.run.smithi040.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T18:03:05.324 INFO:teuthology.orchestra.run.smithi040.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T18:03:05.326 INFO:teuthology.orchestra.run.smithi070.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T18:03:05.331 INFO:teuthology.orchestra.run.smithi040.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T18:03:05.332 INFO:teuthology.orchestra.run.smithi070.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T18:03:05.335 INFO:teuthology.orchestra.run.smithi070.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T18:03:05.336 INFO:teuthology.orchestra.run.smithi070.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T18:03:05.347 INFO:teuthology.orchestra.run.smithi070.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T18:03:05.348 INFO:teuthology.orchestra.run.smithi070.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T18:03:05.356 INFO:teuthology.orchestra.run.smithi070.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T18:03:05.401 INFO:teuthology.orchestra.run.smithi040.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T18:03:05.404 INFO:teuthology.orchestra.run.smithi040.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T18:03:05.406 INFO:teuthology.orchestra.run.smithi040.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T18:03:05.408 INFO:teuthology.orchestra.run.smithi070.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T18:03:05.409 INFO:teuthology.orchestra.run.smithi070.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T18:03:05.410 INFO:teuthology.orchestra.run.smithi070.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T18:03:05.465 INFO:teuthology.orchestra.run.smithi070.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5073-gc69a6b39-1jammy [23.1 MB] 2024-09-21T18:03:05.482 INFO:teuthology.orchestra.run.smithi040.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T18:03:05.489 INFO:teuthology.orchestra.run.smithi040.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T18:03:05.489 INFO:teuthology.orchestra.run.smithi070.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T18:03:05.491 INFO:teuthology.orchestra.run.smithi040.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T18:03:05.491 INFO:teuthology.orchestra.run.smithi070.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T18:03:05.493 INFO:teuthology.orchestra.run.smithi040.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T18:03:05.504 INFO:teuthology.orchestra.run.smithi040.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T18:03:05.505 INFO:teuthology.orchestra.run.smithi040.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T18:03:05.508 INFO:teuthology.orchestra.run.smithi070.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T18:03:05.509 INFO:teuthology.orchestra.run.smithi070.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T18:03:05.510 INFO:teuthology.orchestra.run.smithi070.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T18:03:05.513 INFO:teuthology.orchestra.run.smithi040.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T18:03:05.515 INFO:teuthology.orchestra.run.smithi070.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T18:03:05.516 INFO:teuthology.orchestra.run.smithi070.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T18:03:05.546 INFO:teuthology.orchestra.run.smithi160.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5073-gc69a6b39-1jammy [112 kB] 2024-09-21T18:03:05.548 INFO:teuthology.orchestra.run.smithi160.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5073-gc69a6b39-1jammy [473 kB] 2024-09-21T18:03:05.563 INFO:teuthology.orchestra.run.smithi040.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T18:03:05.565 INFO:teuthology.orchestra.run.smithi040.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T18:03:05.566 INFO:teuthology.orchestra.run.smithi040.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T18:03:05.593 INFO:teuthology.orchestra.run.smithi029.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-5073-gc69a6b39-1jammy [722 kB] 2024-09-21T18:03:05.605 INFO:teuthology.orchestra.run.smithi160.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5073-gc69a6b39-1jammy [28.3 MB] 2024-09-21T18:03:05.645 INFO:teuthology.orchestra.run.smithi040.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T18:03:05.647 INFO:teuthology.orchestra.run.smithi040.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T18:03:05.657 INFO:teuthology.orchestra.run.smithi029.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-5073-gc69a6b39-1jammy [356 kB] 2024-09-21T18:03:05.664 INFO:teuthology.orchestra.run.smithi040.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T18:03:05.665 INFO:teuthology.orchestra.run.smithi040.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T18:03:05.665 INFO:teuthology.orchestra.run.smithi040.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T18:03:05.670 INFO:teuthology.orchestra.run.smithi040.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T18:03:05.672 INFO:teuthology.orchestra.run.smithi040.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T18:03:05.715 INFO:teuthology.orchestra.run.smithi029.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-5073-gc69a6b39-1jammy [32.8 kB] 2024-09-21T18:03:05.716 INFO:teuthology.orchestra.run.smithi029.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-5073-gc69a6b39-1jammy [180 kB] 2024-09-21T18:03:05.770 INFO:teuthology.orchestra.run.smithi029.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-5073-gc69a6b39-1jammy [72.9 kB] 2024-09-21T18:03:05.773 INFO:teuthology.orchestra.run.smithi029.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-5073-gc69a6b39-1jammy [339 kB] 2024-09-21T18:03:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-5073-gc69a6b39-1jammy [7,966 kB] 2024-09-21T18:03:05.894 INFO:teuthology.orchestra.run.smithi040.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5073-gc69a6b39-1jammy [5,211 kB] 2024-09-21T18:03:06.029 INFO:teuthology.orchestra.run.smithi040.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5073-gc69a6b39-1jammy [252 kB] 2024-09-21T18:03:06.066 INFO:teuthology.orchestra.run.smithi040.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5073-gc69a6b39-1jammy [124 kB] 2024-09-21T18:03:06.068 INFO:teuthology.orchestra.run.smithi040.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5073-gc69a6b39-1jammy [1,561 kB] 2024-09-21T18:03:06.144 INFO:teuthology.orchestra.run.smithi040.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5073-gc69a6b39-1jammy [6,312 kB] 2024-09-21T18:03:06.477 INFO:teuthology.orchestra.run.smithi040.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5073-gc69a6b39-1jammy [23.1 MB] 2024-09-21T18:03:06.552 INFO:teuthology.orchestra.run.smithi070.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5073-gc69a6b39-1jammy [14.0 kB] 2024-09-21T18:03:06.552 INFO:teuthology.orchestra.run.smithi070.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [880 kB] 2024-09-21T18:03:06.569 INFO:teuthology.orchestra.run.smithi070.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5073-gc69a6b39-1jammy [2,456 kB] 2024-09-21T18:03:06.578 INFO:teuthology.orchestra.run.smithi029.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-5073-gc69a6b39-1jammy [112 kB] 2024-09-21T18:03:06.618 INFO:teuthology.orchestra.run.smithi070.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5073-gc69a6b39-1jammy [814 kB] 2024-09-21T18:03:06.619 INFO:teuthology.orchestra.run.smithi029.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-5073-gc69a6b39-1jammy [473 kB] 2024-09-21T18:03:06.628 INFO:teuthology.orchestra.run.smithi029.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-5073-gc69a6b39-1jammy [28.3 MB] 2024-09-21T18:03:06.634 INFO:teuthology.orchestra.run.smithi070.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5073-gc69a6b39-1jammy [160 kB] 2024-09-21T18:03:06.637 INFO:teuthology.orchestra.run.smithi070.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5073-gc69a6b39-1jammy [61.5 MB] 2024-09-21T18:03:07.711 INFO:teuthology.orchestra.run.smithi040.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5073-gc69a6b39-1jammy [14.0 kB] 2024-09-21T18:03:07.712 INFO:teuthology.orchestra.run.smithi040.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [880 kB] 2024-09-21T18:03:07.729 INFO:teuthology.orchestra.run.smithi040.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5073-gc69a6b39-1jammy [2,456 kB] 2024-09-21T18:03:07.800 INFO:teuthology.orchestra.run.smithi040.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5073-gc69a6b39-1jammy [814 kB] 2024-09-21T18:03:07.814 INFO:teuthology.orchestra.run.smithi040.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5073-gc69a6b39-1jammy [160 kB] 2024-09-21T18:03:07.816 INFO:teuthology.orchestra.run.smithi040.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5073-gc69a6b39-1jammy [61.5 MB] 2024-09-21T18:03:09.023 INFO:teuthology.orchestra.run.smithi160.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5073-gc69a6b39-1jammy [5,211 kB] 2024-09-21T18:03:09.036 INFO:teuthology.orchestra.run.smithi029.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-5073-gc69a6b39-1jammy [5,211 kB] 2024-09-21T18:03:09.239 INFO:teuthology.orchestra.run.smithi029.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5073-gc69a6b39-1jammy [252 kB] 2024-09-21T18:03:09.244 INFO:teuthology.orchestra.run.smithi029.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5073-gc69a6b39-1jammy [124 kB] 2024-09-21T18:03:09.247 INFO:teuthology.orchestra.run.smithi029.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5073-gc69a6b39-1jammy [1,561 kB] 2024-09-21T18:03:09.359 INFO:teuthology.orchestra.run.smithi029.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5073-gc69a6b39-1jammy [6,312 kB] 2024-09-21T18:03:09.521 INFO:teuthology.orchestra.run.smithi160.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-5073-gc69a6b39-1jammy [252 kB] 2024-09-21T18:03:09.557 INFO:teuthology.orchestra.run.smithi160.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-5073-gc69a6b39-1jammy [124 kB] 2024-09-21T18:03:09.591 INFO:teuthology.orchestra.run.smithi160.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-5073-gc69a6b39-1jammy [1,561 kB] 2024-09-21T18:03:09.753 INFO:teuthology.orchestra.run.smithi160.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-5073-gc69a6b39-1jammy [6,312 kB] 2024-09-21T18:03:09.825 INFO:teuthology.orchestra.run.smithi029.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5073-gc69a6b39-1jammy [23.1 MB] 2024-09-21T18:03:10.251 INFO:teuthology.orchestra.run.smithi070.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5073-gc69a6b39-1jammy [8,625 kB] 2024-09-21T18:03:10.450 INFO:teuthology.orchestra.run.smithi160.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-5073-gc69a6b39-1jammy [23.1 MB] 2024-09-21T18:03:10.559 INFO:teuthology.orchestra.run.smithi070.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5073-gc69a6b39-1jammy [14.2 kB] 2024-09-21T18:03:10.559 INFO:teuthology.orchestra.run.smithi070.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5073-gc69a6b39-1jammy [48.5 MB] 2024-09-21T18:03:10.952 INFO:teuthology.orchestra.run.smithi040.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5073-gc69a6b39-1jammy [8,625 kB] 2024-09-21T18:03:11.213 INFO:teuthology.orchestra.run.smithi040.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5073-gc69a6b39-1jammy [14.2 kB] 2024-09-21T18:03:11.213 INFO:teuthology.orchestra.run.smithi040.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5073-gc69a6b39-1jammy [48.5 MB] 2024-09-21T18:03:11.663 INFO:teuthology.orchestra.run.smithi029.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5073-gc69a6b39-1jammy [14.0 kB] 2024-09-21T18:03:11.664 INFO:teuthology.orchestra.run.smithi029.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [880 kB] 2024-09-21T18:03:11.681 INFO:teuthology.orchestra.run.smithi029.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5073-gc69a6b39-1jammy [2,456 kB] 2024-09-21T18:03:11.879 INFO:teuthology.orchestra.run.smithi029.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5073-gc69a6b39-1jammy [814 kB] 2024-09-21T18:03:11.930 INFO:teuthology.orchestra.run.smithi029.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5073-gc69a6b39-1jammy [160 kB] 2024-09-21T18:03:11.957 INFO:teuthology.orchestra.run.smithi029.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5073-gc69a6b39-1jammy [61.5 MB] 2024-09-21T18:03:13.157 INFO:teuthology.orchestra.run.smithi070.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5073-gc69a6b39-1jammy [130 kB] 2024-09-21T18:03:13.159 INFO:teuthology.orchestra.run.smithi070.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5073-gc69a6b39-1jammy [39.0 kB] 2024-09-21T18:03:13.160 INFO:teuthology.orchestra.run.smithi070.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5073-gc69a6b39-1jammy [15.8 MB] 2024-09-21T18:03:13.566 INFO:teuthology.orchestra.run.smithi160.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-5073-gc69a6b39-1jammy [14.0 kB] 2024-09-21T18:03:13.566 INFO:teuthology.orchestra.run.smithi160.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [880 kB] 2024-09-21T18:03:13.583 INFO:teuthology.orchestra.run.smithi040.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5073-gc69a6b39-1jammy [130 kB] 2024-09-21T18:03:13.583 INFO:teuthology.orchestra.run.smithi160.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-5073-gc69a6b39-1jammy [2,456 kB] 2024-09-21T18:03:13.586 INFO:teuthology.orchestra.run.smithi040.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5073-gc69a6b39-1jammy [39.0 kB] 2024-09-21T18:03:13.587 INFO:teuthology.orchestra.run.smithi040.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5073-gc69a6b39-1jammy [15.8 MB] 2024-09-21T18:03:13.808 INFO:teuthology.orchestra.run.smithi070.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [91.2 kB] 2024-09-21T18:03:13.821 INFO:teuthology.orchestra.run.smithi160.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-5073-gc69a6b39-1jammy [814 kB] 2024-09-21T18:03:13.882 INFO:teuthology.orchestra.run.smithi160.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-5073-gc69a6b39-1jammy [160 kB] 2024-09-21T18:03:13.886 INFO:teuthology.orchestra.run.smithi160.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-5073-gc69a6b39-1jammy [61.5 MB] 2024-09-21T18:03:14.131 INFO:teuthology.orchestra.run.smithi040.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [91.2 kB] 2024-09-21T18:03:14.385 INFO:teuthology.orchestra.run.smithi070.stdout:Fetched 244 MB in 11s (23.0 MB/s) 2024-09-21T18:03:14.544 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T18:03:14.644 INFO:teuthology.orchestra.run.smithi070.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T18:03:14.650 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T18:03:14.726 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:03:14.733 INFO:teuthology.orchestra.run.smithi040.stdout:Fetched 244 MB in 11s (22.7 MB/s) 2024-09-21T18:03:14.904 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T18:03:15.007 INFO:teuthology.orchestra.run.smithi040.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T18:03:15.013 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T18:03:15.094 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:03:15.146 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T18:03:15.172 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T18:03:15.204 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:03:15.406 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T18:03:15.432 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T18:03:15.472 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:03:15.541 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T18:03:15.567 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T18:03:15.607 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:03:15.792 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T18:03:15.819 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T18:03:15.858 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:03:15.907 INFO:teuthology.orchestra.run.smithi029.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5073-gc69a6b39-1jammy [8,625 kB] 2024-09-21T18:03:16.036 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T18:03:16.062 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:16.070 INFO:teuthology.orchestra.run.smithi029.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5073-gc69a6b39-1jammy [14.2 kB] 2024-09-21T18:03:16.071 INFO:teuthology.orchestra.run.smithi029.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5073-gc69a6b39-1jammy [48.5 MB] 2024-09-21T18:03:16.111 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:16.304 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T18:03:16.331 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:16.379 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:16.489 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T18:03:16.507 INFO:teuthology.orchestra.run.smithi160.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-5073-gc69a6b39-1jammy [8,625 kB] 2024-09-21T18:03:16.515 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:16.547 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:16.717 INFO:teuthology.orchestra.run.smithi160.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-5073-gc69a6b39-1jammy [14.2 kB] 2024-09-21T18:03:16.724 INFO:teuthology.orchestra.run.smithi160.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-5073-gc69a6b39-1jammy [48.5 MB] 2024-09-21T18:03:16.782 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T18:03:16.809 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:16.848 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:16.867 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T18:03:16.893 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:16.933 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:17.194 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T18:03:17.221 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:17.260 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:17.303 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T18:03:17.329 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T18:03:17.370 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:03:17.613 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T18:03:17.640 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T18:03:17.656 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librados2. 2024-09-21T18:03:17.682 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../007-librados2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:17.690 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:03:17.722 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:17.780 INFO:teuthology.orchestra.run.smithi029.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5073-gc69a6b39-1jammy [130 kB] 2024-09-21T18:03:17.783 INFO:teuthology.orchestra.run.smithi029.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5073-gc69a6b39-1jammy [39.0 kB] 2024-09-21T18:03:17.784 INFO:teuthology.orchestra.run.smithi029.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5073-gc69a6b39-1jammy [15.8 MB] 2024-09-21T18:03:17.999 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librados2. 2024-09-21T18:03:18.023 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../007-librados2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:18.057 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:18.184 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libnbd0. 2024-09-21T18:03:18.210 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-21T18:03:18.223 INFO:teuthology.orchestra.run.smithi029.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [91.2 kB] 2024-09-21T18:03:18.250 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-21T18:03:18.494 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libnbd0. 2024-09-21T18:03:18.521 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-21T18:03:18.553 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librbd1. 2024-09-21T18:03:18.561 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-21T18:03:18.579 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../009-librbd1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:18.619 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:18.710 INFO:teuthology.orchestra.run.smithi160.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-5073-gc69a6b39-1jammy [130 kB] 2024-09-21T18:03:18.712 INFO:teuthology.orchestra.run.smithi160.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-5073-gc69a6b39-1jammy [39.0 kB] 2024-09-21T18:03:18.713 INFO:teuthology.orchestra.run.smithi160.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-5073-gc69a6b39-1jammy [15.8 MB] 2024-09-21T18:03:18.837 INFO:teuthology.orchestra.run.smithi029.stdout:Fetched 244 MB in 15s (16.6 MB/s) 2024-09-21T18:03:18.880 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librbd1. 2024-09-21T18:03:18.907 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../009-librbd1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:18.947 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.008 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T18:03:19.064 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T18:03:19.090 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:19.112 INFO:teuthology.orchestra.run.smithi029.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T18:03:19.118 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T18:03:19.130 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.199 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:03:19.205 INFO:teuthology.orchestra.run.smithi160.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-rf-qprio-w-qa/c69a6b39fef6dd04706886608793361ef510a986/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-5073-gc69a6b39-1jammy [91.2 kB] 2024-09-21T18:03:19.366 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T18:03:19.374 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rados. 2024-09-21T18:03:19.382 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:19.390 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:19.424 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.433 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.548 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T18:03:19.575 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T18:03:19.615 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:03:19.686 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rados. 2024-09-21T18:03:19.713 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:19.744 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T18:03:19.752 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.770 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:19.780 INFO:teuthology.orchestra.run.smithi160.stdout:Fetched 244 MB in 16s (15.5 MB/s) 2024-09-21T18:03:19.810 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:19.944 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T18:03:19.980 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T18:03:20.007 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T18:03:20.045 INFO:teuthology.orchestra.run.smithi160.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T18:03:20.047 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:03:20.051 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T18:03:20.072 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T18:03:20.099 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:20.134 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:03:20.138 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:20.391 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T18:03:20.417 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:20.421 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T18:03:20.447 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:20.457 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:20.479 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:20.513 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T18:03:20.540 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:20.588 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:20.752 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T18:03:20.779 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:20.790 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T18:03:20.810 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:20.817 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:20.857 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:21.063 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T18:03:21.086 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T18:03:21.121 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T18:03:21.130 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:03:21.148 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T18:03:21.185 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T18:03:21.188 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:03:21.211 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T18:03:21.252 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:03:21.286 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T18:03:21.313 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:21.353 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:21.441 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T18:03:21.468 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T18:03:21.507 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T18:03:21.508 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T18:03:21.534 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T18:03:21.571 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T18:03:21.574 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:03:21.597 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T18:03:21.638 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T18:03:21.668 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T18:03:21.695 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:21.735 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:21.777 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T18:03:21.804 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:21.844 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:21.890 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T18:03:21.916 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:21.944 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T18:03:21.957 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:21.971 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:22.019 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:22.100 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T18:03:22.127 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T18:03:22.175 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:03:22.180 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T18:03:22.207 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T18:03:22.238 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:03:22.276 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T18:03:22.302 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T18:03:22.343 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:03:22.414 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T18:03:22.439 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:22.466 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T18:03:22.480 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:22.481 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T18:03:22.514 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librados2. 2024-09-21T18:03:22.524 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:03:22.540 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../007-librados2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:22.580 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:22.612 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T18:03:22.638 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T18:03:22.679 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:03:22.836 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T18:03:22.858 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T18:03:22.863 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T18:03:22.884 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T18:03:22.902 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:03:22.925 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:03:22.981 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T18:03:22.997 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T18:03:23.039 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:03:23.137 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libnbd0. 2024-09-21T18:03:23.164 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-21T18:03:23.212 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-21T18:03:23.239 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua5.1. 2024-09-21T18:03:23.266 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T18:03:23.286 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T18:03:23.306 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:03:23.312 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T18:03:23.352 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:03:23.359 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua5.1. 2024-09-21T18:03:23.386 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T18:03:23.417 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:03:23.527 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librbd1. 2024-09-21T18:03:23.554 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../009-librbd1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:23.583 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-any. 2024-09-21T18:03:23.599 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-21T18:03:23.602 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:23.633 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T18:03:23.689 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librados2. 2024-09-21T18:03:23.715 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../007-librados2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:23.731 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-any. 2024-09-21T18:03:23.746 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-21T18:03:23.755 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:23.789 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T18:03:23.886 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package zip. 2024-09-21T18:03:23.903 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-21T18:03:23.945 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T18:03:24.058 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package zip. 2024-09-21T18:03:24.084 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-21T18:03:24.125 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T18:03:24.132 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T18:03:24.159 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:24.199 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:24.241 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libnbd0. 2024-09-21T18:03:24.265 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package unzip. 2024-09-21T18:03:24.267 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-21T18:03:24.292 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T18:03:24.307 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-21T18:03:24.348 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:03:24.411 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package unzip. 2024-09-21T18:03:24.437 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T18:03:24.469 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:03:24.497 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rados. 2024-09-21T18:03:24.512 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:24.555 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:24.560 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librbd1. 2024-09-21T18:03:24.575 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../009-librbd1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:24.610 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:24.693 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package luarocks. 2024-09-21T18:03:24.710 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T18:03:24.743 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:03:24.873 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package luarocks. 2024-09-21T18:03:24.885 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T18:03:24.900 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T18:03:24.912 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:24.952 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:24.956 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:03:25.038 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T18:03:25.064 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.084 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package librgw2. 2024-09-21T18:03:25.104 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.112 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../026-librgw2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.151 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.216 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T18:03:25.232 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.275 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.348 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package librgw2. 2024-09-21T18:03:25.374 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../026-librgw2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.390 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rados. 2024-09-21T18:03:25.414 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.416 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../011-python3-rados_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.456 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.598 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T18:03:25.624 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:25.664 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.762 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T18:03:25.778 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:25.820 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:25.968 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T18:03:25.988 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T18:03:25.994 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:26.013 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T18:03:26.016 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T18:03:26.034 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.043 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:26.053 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:03:26.083 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.173 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T18:03:26.200 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T18:03:26.240 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:03:26.295 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T18:03:26.322 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:26.362 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.378 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T18:03:26.404 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T18:03:26.444 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:03:26.493 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T18:03:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T18:03:26.519 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:26.545 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T18:03:26.551 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T18:03:26.681 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T18:03:26.707 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:26.747 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.755 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T18:03:26.782 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:26.822 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.857 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T18:03:26.883 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:26.932 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:26.980 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-common. 2024-09-21T18:03:27.006 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:27.042 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T18:03:27.046 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:27.068 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T18:03:27.100 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:03:27.250 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-common. 2024-09-21T18:03:27.277 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:27.289 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T18:03:27.315 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T18:03:27.316 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:27.355 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:03:27.411 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T18:03:27.438 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T18:03:27.478 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T18:03:27.645 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T18:03:27.660 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T18:03:27.703 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:03:27.721 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T18:03:27.737 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:27.780 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:28.016 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T18:03:28.032 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T18:03:28.075 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:03:28.124 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T18:03:28.150 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T18:03:28.191 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:03:28.424 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua5.1. 2024-09-21T18:03:28.451 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T18:03:28.460 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T18:03:28.475 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T18:03:28.491 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:03:28.510 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:03:28.568 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-base. 2024-09-21T18:03:28.594 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:28.644 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:28.804 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T18:03:28.805 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-base. 2024-09-21T18:03:28.814 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-any. 2024-09-21T18:03:28.820 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T18:03:28.832 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:28.841 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-21T18:03:28.863 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:03:28.880 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:28.890 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T18:03:29.136 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package zip. 2024-09-21T18:03:29.163 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-21T18:03:29.172 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T18:03:29.199 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package lua5.1. 2024-09-21T18:03:29.199 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T18:03:29.202 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T18:03:29.225 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T18:03:29.230 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:03:29.265 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:03:29.401 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T18:03:29.427 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T18:03:29.459 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:03:29.533 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T18:03:29.552 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package unzip. 2024-09-21T18:03:29.560 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T18:03:29.576 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package lua-any. 2024-09-21T18:03:29.578 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T18:03:29.592 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-21T18:03:29.600 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:03:29.618 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:03:29.635 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T18:03:29.753 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T18:03:29.780 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T18:03:29.828 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:03:29.845 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package zip. 2024-09-21T18:03:29.861 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-21T18:03:29.896 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T18:03:30.035 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package luarocks. 2024-09-21T18:03:30.062 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T18:03:30.102 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:03:30.140 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T18:03:30.166 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T18:03:30.197 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T18:03:30.206 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:03:30.224 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T18:03:30.240 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package unzip. 2024-09-21T18:03:30.263 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:03:30.266 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T18:03:30.307 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:03:30.484 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package librgw2. 2024-09-21T18:03:30.511 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../026-librgw2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:30.550 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T18:03:30.559 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:30.576 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T18:03:30.608 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:03:30.684 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T18:03:30.710 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T18:03:30.750 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:03:30.796 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package luarocks. 2024-09-21T18:03:30.823 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T18:03:30.902 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:03:30.910 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T18:03:30.937 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T18:03:30.977 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:03:31.003 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T18:03:31.029 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T18:03:31.061 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:03:31.226 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package librgw2. 2024-09-21T18:03:31.241 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../026-librgw2_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:31.275 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:31.288 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T18:03:31.315 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T18:03:31.348 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T18:03:31.355 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T18:03:31.374 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T18:03:31.381 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T18:03:31.397 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:31.414 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T18:03:31.440 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:31.658 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-portend. 2024-09-21T18:03:31.685 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-21T18:03:31.717 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-portend. 2024-09-21T18:03:31.724 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T18:03:31.743 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-21T18:03:31.783 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T18:03:31.820 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T18:03:31.822 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T18:03:31.835 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:31.849 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T18:03:31.878 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:31.889 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:03:31.994 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T18:03:32.021 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T18:03:32.060 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T18:03:32.086 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T18:03:32.113 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T18:03:32.153 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T18:03:32.194 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T18:03:32.221 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:32.222 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T18:03:32.248 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T18:03:32.269 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:32.289 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:03:32.364 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T18:03:32.390 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T18:03:32.422 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T18:03:32.430 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:03:32.449 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T18:03:32.489 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:03:32.608 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T18:03:32.635 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:32.683 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:32.708 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T18:03:32.710 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-common. 2024-09-21T18:03:32.724 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T18:03:32.736 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:32.767 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T18:03:32.776 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:32.842 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T18:03:32.869 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T18:03:32.909 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T18:03:33.028 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-common. 2024-09-21T18:03:33.055 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../030-ceph-common_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:33.061 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T18:03:33.086 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:33.088 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T18:03:33.128 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T18:03:33.187 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T18:03:33.213 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T18:03:33.245 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T18:03:33.439 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T18:03:33.466 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T18:03:33.506 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:03:33.548 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T18:03:33.575 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T18:03:33.615 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:03:33.792 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-mako. 2024-09-21T18:03:33.819 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T18:03:33.858 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:03:33.918 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-mako. 2024-09-21T18:03:33.944 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T18:03:33.984 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:03:34.136 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T18:03:34.152 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T18:03:34.186 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:03:34.312 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T18:03:34.339 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T18:03:34.379 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:03:34.481 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T18:03:34.508 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T18:03:34.516 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-base. 2024-09-21T18:03:34.534 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:34.547 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:03:34.583 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:34.657 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T18:03:34.684 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T18:03:34.687 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-base. 2024-09-21T18:03:34.714 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../031-ceph-base_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:34.723 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:03:34.764 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:34.875 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webob. 2024-09-21T18:03:34.902 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T18:03:34.942 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:03:34.984 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-webob. 2024-09-21T18:03:35.011 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T18:03:35.042 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:03:35.162 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T18:03:35.189 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T18:03:35.229 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:03:35.312 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T18:03:35.328 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T18:03:35.339 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T18:03:35.344 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T18:03:35.382 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:03:35.390 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:03:35.464 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T18:03:35.490 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T18:03:35.530 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:03:35.531 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T18:03:35.559 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T18:03:35.846 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T18:03:35.872 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T18:03:35.912 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:03:36.040 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:03:36.217 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T18:03:36.244 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T18:03:36.292 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:03:36.432 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T18:03:36.458 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T18:03:36.459 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T18:03:36.484 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T18:03:36.490 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:03:36.523 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:03:36.710 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T18:03:36.736 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T18:03:36.768 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-paste. 2024-09-21T18:03:36.776 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:03:36.784 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-paste. 2024-09-21T18:03:36.795 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T18:03:36.811 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T18:03:36.835 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:03:36.843 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:03:37.029 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T18:03:37.055 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T18:03:37.095 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:03:37.146 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T18:03:37.171 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T18:03:37.173 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T18:03:37.197 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T18:03:37.212 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:03:37.237 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:03:37.339 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T18:03:37.366 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T18:03:37.406 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:03:37.499 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T18:03:37.526 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T18:03:37.540 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T18:03:37.565 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:03:37.566 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T18:03:37.607 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:03:37.717 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T18:03:37.743 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T18:03:37.784 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T18:03:37.868 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T18:03:37.886 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T18:03:37.895 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T18:03:37.911 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T18:03:37.943 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T18:03:37.951 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T18:03:38.087 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-portend. 2024-09-21T18:03:38.113 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-21T18:03:38.153 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T18:03:38.204 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T18:03:38.231 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T18:03:38.254 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T18:03:38.262 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:03:38.281 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T18:03:38.317 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T18:03:38.321 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:03:38.343 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T18:03:38.383 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:03:38.406 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T18:03:38.432 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T18:03:38.464 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T18:03:38.592 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T18:03:38.618 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T18:03:38.658 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T18:03:38.684 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T18:03:38.690 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T18:03:38.711 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T18:03:38.716 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T18:03:38.733 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T18:03:38.751 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T18:03:38.756 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:03:38.760 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T18:03:38.800 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:03:38.978 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-idna. 2024-09-21T18:03:39.004 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T18:03:39.045 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:03:39.045 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T18:03:39.070 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-idna. 2024-09-21T18:03:39.072 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T18:03:39.098 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T18:03:39.120 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T18:03:39.128 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T18:03:39.137 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:03:39.155 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T18:03:39.195 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T18:03:39.339 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T18:03:39.366 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T18:03:39.405 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:03:39.415 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T18:03:39.435 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-portend. 2024-09-21T18:03:39.442 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T18:03:39.462 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-21T18:03:39.481 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:03:39.502 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T18:03:39.506 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T18:03:39.532 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T18:03:39.572 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T18:03:39.734 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-requests. 2024-09-21T18:03:39.751 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests. 2024-09-21T18:03:39.760 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T18:03:39.768 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T18:03:39.800 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:03:39.809 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:03:39.817 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T18:03:39.844 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T18:03:39.875 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T18:03:39.883 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T18:03:39.902 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T18:03:39.942 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:03:40.103 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T18:03:40.112 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T18:03:40.118 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T18:03:40.140 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T18:03:40.153 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:03:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T18:03:40.207 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T18:03:40.211 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-mako. 2024-09-21T18:03:40.227 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T18:03:40.247 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:03:40.261 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:03:40.320 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:03:40.431 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T18:03:40.458 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:40.497 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:40.556 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T18:03:40.580 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T18:03:40.582 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T18:03:40.607 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T18:03:40.614 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:03:40.623 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T18:03:40.640 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:40.655 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T18:03:40.682 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:40.877 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T18:03:40.900 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T18:03:40.904 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:40.923 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T18:03:40.944 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:40.987 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T18:03:41.014 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T18:03:41.020 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T18:03:41.047 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:41.054 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T18:03:41.086 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:41.166 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:03:41.263 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T18:03:41.290 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T18:03:41.330 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:03:41.367 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T18:03:41.394 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T18:03:41.406 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T18:03:41.433 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T18:03:41.434 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:03:41.473 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:03:41.480 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webob. 2024-09-21T18:03:41.504 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T18:03:41.544 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:03:41.608 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T18:03:41.637 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T18:03:41.683 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T18:03:41.758 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-mako. 2024-09-21T18:03:41.775 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T18:03:41.784 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T18:03:41.803 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T18:03:41.824 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:03:41.842 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T18:03:41.889 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T18:03:41.915 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T18:03:41.958 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:03:42.011 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T18:03:42.037 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:42.077 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:42.145 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T18:03:42.148 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T18:03:42.172 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:42.175 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T18:03:42.203 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:42.223 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:03:42.258 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T18:03:42.284 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T18:03:42.324 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:03:42.480 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T18:03:42.506 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:42.528 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T18:03:42.547 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:42.555 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T18:03:42.590 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T18:03:42.594 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-paste. 2024-09-21T18:03:42.595 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:03:42.617 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:42.620 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T18:03:42.656 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:42.660 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:03:42.943 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-webob. 2024-09-21T18:03:42.970 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T18:03:42.980 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T18:03:43.006 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T18:03:43.010 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:03:43.046 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:03:43.125 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T18:03:43.152 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:43.191 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:43.291 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T18:03:43.293 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T18:03:43.317 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T18:03:43.320 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:43.349 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:03:43.359 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:43.367 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T18:03:43.394 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T18:03:43.445 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:03:43.635 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T18:03:43.662 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T18:03:43.702 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T18:03:43.764 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T18:03:43.791 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T18:03:43.831 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:03:44.013 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T18:03:44.039 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T18:03:44.079 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:03:44.129 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-paste. 2024-09-21T18:03:44.156 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T18:03:44.204 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:03:44.311 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph. 2024-09-21T18:03:44.326 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../067-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:44.361 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:44.429 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph. 2024-09-21T18:03:44.434 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T18:03:44.445 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../067-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:44.460 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T18:03:44.487 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:44.500 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T18:03:44.537 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T18:03:44.563 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T18:03:44.603 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:03:44.664 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T18:03:44.691 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:44.730 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:44.782 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T18:03:44.809 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:44.848 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:44.853 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-idna. 2024-09-21T18:03:44.880 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T18:03:44.908 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T18:03:44.920 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:03:44.935 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T18:03:44.983 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:03:45.091 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T18:03:45.119 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:45.158 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:45.181 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T18:03:45.207 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T18:03:45.210 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T18:03:45.237 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:45.239 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:03:45.273 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T18:03:45.293 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:45.301 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T18:03:45.340 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T18:03:45.509 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-requests. 2024-09-21T18:03:45.521 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T18:03:45.527 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package cephadm. 2024-09-21T18:03:45.550 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:03:45.554 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../070-cephadm_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:45.594 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:45.638 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T18:03:45.664 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T18:03:45.696 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package cephadm. 2024-09-21T18:03:45.712 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:03:45.723 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../070-cephadm_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:45.762 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:45.845 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T18:03:45.871 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T18:03:45.911 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:03:46.015 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T18:03:46.043 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T18:03:46.072 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T18:03:46.073 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T18:03:46.089 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T18:03:46.100 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T18:03:46.115 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T18:03:46.140 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T18:03:46.155 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T18:03:46.181 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T18:03:46.207 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:46.250 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:46.376 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T18:03:46.401 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:46.443 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:46.449 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T18:03:46.462 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-idna. 2024-09-21T18:03:46.476 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:46.478 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T18:03:46.516 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:46.520 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:03:46.661 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T18:03:46.687 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:46.727 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:46.836 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T18:03:46.865 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T18:03:47.047 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T18:03:47.073 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T18:03:47.077 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:03:47.113 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:03:47.383 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-requests. 2024-09-21T18:03:47.411 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T18:03:47.415 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T18:03:47.442 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T18:03:47.458 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:03:47.470 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-babel. 2024-09-21T18:03:47.472 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-babel. 2024-09-21T18:03:47.482 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T18:03:47.486 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:47.500 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:47.520 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:47.539 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:47.765 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T18:03:47.785 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T18:03:47.792 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T18:03:47.806 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T18:03:47.812 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:47.822 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T18:03:47.831 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:03:47.851 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:47.856 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:03:47.892 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T18:03:47.920 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T18:03:47.958 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:03:48.122 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T18:03:48.138 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:48.151 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T18:03:48.166 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:48.201 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:48.263 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T18:03:48.289 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:48.320 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T18:03:48.329 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:48.348 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:48.386 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:48.521 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T18:03:48.536 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T18:03:48.571 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T18:03:48.574 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T18:03:48.601 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:48.641 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:48.723 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T18:03:48.751 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T18:03:48.790 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T18:03:48.824 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-routes. 2024-09-21T18:03:48.850 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T18:03:48.890 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:03:48.933 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T18:03:48.959 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:48.962 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T18:03:48.990 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T18:03:48.999 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:49.029 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:03:49.084 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-routes. 2024-09-21T18:03:49.112 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T18:03:49.151 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:03:49.177 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T18:03:49.203 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:49.235 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:49.335 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T18:03:49.363 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T18:03:49.402 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T18:03:49.487 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T18:03:49.515 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:49.554 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:49.751 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T18:03:49.778 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:49.826 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:50.087 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph. 2024-09-21T18:03:50.103 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../067-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:50.146 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:50.231 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T18:03:50.259 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:50.298 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:50.448 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T18:03:50.475 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:50.514 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:50.851 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T18:03:50.878 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:50.909 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:51.048 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T18:03:51.075 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:51.114 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:51.295 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package cephadm. 2024-09-21T18:03:51.322 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../070-cephadm_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:51.362 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:51.906 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T18:03:51.906 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T18:03:51.923 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T18:03:51.933 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T18:03:51.964 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:03:51.972 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T18:03:52.275 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T18:03:52.302 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:52.333 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:52.367 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T18:03:52.385 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T18:03:52.417 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:03:52.451 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T18:03:52.479 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T18:03:52.518 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:03:52.538 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph. 2024-09-21T18:03:52.554 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../067-ceph_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:52.597 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:52.871 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T18:03:52.898 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T18:03:52.920 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T18:03:52.926 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T18:03:52.929 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:03:52.948 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T18:03:52.953 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:52.987 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:03:53.002 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:53.215 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T18:03:53.243 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T18:03:53.297 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-babel. 2024-09-21T18:03:53.324 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:53.332 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T18:03:53.359 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:03:53.360 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T18:03:53.363 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:53.399 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:03:53.442 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T18:03:53.469 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:53.508 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:53.643 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T18:03:53.671 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T18:03:53.674 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T18:03:53.701 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T18:03:53.741 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:03:53.795 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:03:53.948 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package cephadm. 2024-09-21T18:03:53.975 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../070-cephadm_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:03:54.015 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:54.085 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T18:03:54.094 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T18:03:54.113 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:54.120 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:54.151 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:54.160 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:54.370 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T18:03:54.397 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T18:03:54.445 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T18:03:54.446 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T18:03:54.463 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:54.497 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T18:03:54.504 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:54.524 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T18:03:54.563 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T18:03:54.760 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T18:03:54.778 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:54.818 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:54.866 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-routes. 2024-09-21T18:03:54.893 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T18:03:54.932 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:03:55.104 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T18:03:55.132 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T18:03:55.170 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T18:03:55.294 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T18:03:55.321 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:55.360 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:55.448 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T18:03:55.476 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T18:03:55.540 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:03:55.624 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T18:03:55.654 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T18:03:55.749 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T18:03:55.870 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T18:03:55.887 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-babel. 2024-09-21T18:03:55.896 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T18:03:55.915 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T18:03:55.935 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:03:55.962 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:03:56.094 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T18:03:56.122 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T18:03:56.160 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:03:56.263 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T18:03:56.285 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T18:03:56.290 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-21T18:03:56.313 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T18:03:56.329 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T18:03:56.352 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:03:56.489 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T18:03:56.517 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T18:03:56.555 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:03:56.608 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T18:03:56.635 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T18:03:56.674 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T18:03:56.799 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T18:03:56.956 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:56.956 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:56.967 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T18:03:56.995 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-21T18:03:57.025 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T18:03:57.053 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T18:03:57.080 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T18:03:57.111 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:03:57.312 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T18:03:57.340 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T18:03:57.378 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T18:03:57.389 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T18:03:57.399 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T18:03:57.416 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T18:03:57.426 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T18:03:57.455 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T18:03:57.474 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T18:03:57.698 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T18:03:57.726 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T18:03:57.764 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:03:57.784 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T18:03:57.789 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-routes. 2024-09-21T18:03:57.811 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T18:03:57.816 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T18:03:57.850 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:03:57.855 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:03:58.076 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T18:03:58.104 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T18:03:58.142 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T18:03:58.203 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T18:03:58.230 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:58.269 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:58.282 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T18:03:58.299 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T18:03:58.340 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:03:58.454 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T18:03:58.482 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T18:03:58.520 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:03:58.522 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T18:03:58.539 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:58.580 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:58.827 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T18:03:58.854 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T18:03:58.875 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T18:03:58.902 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T18:03:58.918 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:03:58.933 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:03:59.126 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T18:03:59.143 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:03:59.176 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:03:59.246 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T18:03:59.262 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T18:03:59.269 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T18:03:59.297 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:03:59.304 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:03:59.352 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:03:59.495 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T18:03:59.524 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T18:03:59.562 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:03:59.599 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T18:03:59.614 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package jq. 2024-09-21T18:03:59.626 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T18:03:59.641 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:03:59.680 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T18:03:59.759 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:03:59.898 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T18:03:59.924 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package socat. 2024-09-21T18:03:59.927 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:03:59.941 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T18:03:59.965 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:03:59.983 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:00.201 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package jq. 2024-09-21T18:04:00.230 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:04:00.259 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:00.294 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T18:04:00.323 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T18:04:00.360 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:00.410 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T18:04:00.438 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:00.476 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:00.537 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package socat. 2024-09-21T18:04:00.566 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T18:04:00.603 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:00.707 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-test. 2024-09-21T18:04:00.723 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:00.765 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:00.898 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T18:04:00.915 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T18:04:00.948 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:01.307 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T18:04:01.327 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-test. 2024-09-21T18:04:01.337 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T18:04:01.356 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:01.379 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T18:04:01.382 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:04:01.394 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:01.407 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T18:04:01.446 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:01.724 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T18:04:01.751 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T18:04:01.790 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:01.881 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T18:04:01.909 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T18:04:01.948 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:04:02.127 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T18:04:02.154 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T18:04:02.193 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:02.297 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T18:04:02.325 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T18:04:02.372 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:04:02.530 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T18:04:02.557 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-21T18:04:02.596 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T18:04:02.702 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T18:04:02.730 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T18:04:02.858 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:04:02.908 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T18:04:02.935 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T18:04:02.974 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:03.211 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T18:04:03.239 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:03.277 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:03.286 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T18:04:03.313 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T18:04:03.352 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:03.503 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T18:04:03.532 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:03.578 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:03.590 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T18:04:03.606 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:03.640 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T18:04:03.647 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T18:04:03.648 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:03.668 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:03.674 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T18:04:03.706 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:03.713 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T18:04:03.967 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T18:04:03.995 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T18:04:04.025 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T18:04:04.033 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:04.036 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T18:04:04.053 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T18:04:04.066 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:04.091 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:04.102 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:04.315 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T18:04:04.343 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T18:04:04.355 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T18:04:04.372 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T18:04:04.381 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:04.413 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:04.688 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T18:04:04.705 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:04.736 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T18:04:04.747 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:04.764 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T18:04:04.768 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T18:04:04.796 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T18:04:04.802 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:04.834 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:04.969 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T18:04:04.997 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T18:04:05.035 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:05.066 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T18:04:05.094 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T18:04:05.132 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:05.196 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T18:04:05.225 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T18:04:05.241 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package pkg-config. 2024-09-21T18:04:05.263 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:05.269 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T18:04:05.307 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:05.382 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T18:04:05.410 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T18:04:05.449 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:05.460 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T18:04:05.488 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:04:05.527 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:05.568 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T18:04:05.596 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T18:04:05.634 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:05.635 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package pkg-config. 2024-09-21T18:04:05.652 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T18:04:05.684 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:05.755 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package jq. 2024-09-21T18:04:05.782 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T18:04:05.782 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:04:05.810 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T18:04:05.813 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:05.848 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:05.937 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T18:04:05.954 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T18:04:05.996 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:06.071 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T18:04:06.082 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package socat. 2024-09-21T18:04:06.099 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T18:04:06.110 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T18:04:06.113 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T18:04:06.137 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:06.142 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-21T18:04:06.149 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:06.180 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T18:04:06.365 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T18:04:06.382 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T18:04:06.406 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T18:04:06.423 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T18:04:06.432 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:06.456 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:06.460 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T18:04:06.486 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T18:04:06.488 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T18:04:06.515 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T18:04:06.526 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:06.553 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:06.710 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T18:04:06.738 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T18:04:06.760 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T18:04:06.768 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:06.788 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T18:04:06.826 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:06.835 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T18:04:06.864 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T18:04:06.889 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-test. 2024-09-21T18:04:06.902 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:06.905 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:06.939 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:07.088 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T18:04:07.117 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T18:04:07.154 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T18:04:07.154 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:07.182 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T18:04:07.184 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T18:04:07.212 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T18:04:07.220 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:07.250 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T18:04:07.450 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T18:04:07.478 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T18:04:07.516 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:07.532 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-py. 2024-09-21T18:04:07.559 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-21T18:04:07.598 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T18:04:07.815 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T18:04:07.843 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T18:04:07.844 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-py. 2024-09-21T18:04:07.872 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-21T18:04:07.881 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:07.910 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T18:04:07.926 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T18:04:07.954 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T18:04:07.993 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:08.222 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T18:04:08.250 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T18:04:08.288 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:08.480 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T18:04:08.508 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T18:04:08.555 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:08.653 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T18:04:08.682 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:08.720 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:08.776 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T18:04:08.805 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T18:04:08.842 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:08.866 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-toml. 2024-09-21T18:04:08.894 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-21T18:04:08.932 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T18:04:09.041 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T18:04:09.069 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T18:04:09.116 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:09.162 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-toml. 2024-09-21T18:04:09.168 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T18:04:09.190 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-21T18:04:09.196 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:09.210 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T18:04:09.226 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:09.228 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T18:04:09.238 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T18:04:09.268 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:09.465 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T18:04:09.489 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T18:04:09.494 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:04:09.497 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T18:04:09.513 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:09.518 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T18:04:09.532 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:09.547 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:09.547 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:09.588 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T18:04:09.617 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T18:04:09.663 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:09.780 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package jq. 2024-09-21T18:04:09.808 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T18:04:09.853 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T18:04:09.854 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:09.882 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T18:04:09.883 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T18:04:09.899 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T18:04:09.911 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:09.941 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:09.957 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T18:04:09.986 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T18:04:10.024 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:10.110 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package socat. 2024-09-21T18:04:10.138 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T18:04:10.147 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T18:04:10.164 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T18:04:10.197 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:10.279 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T18:04:10.307 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T18:04:10.345 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:10.674 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T18:04:10.702 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T18:04:10.741 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:10.752 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package radosgw. 2024-09-21T18:04:10.781 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../117-radosgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:10.819 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:10.893 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package radosgw. 2024-09-21T18:04:10.921 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../117-radosgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:10.967 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:11.129 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package pkg-config. 2024-09-21T18:04:11.157 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T18:04:11.195 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:11.326 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:11.431 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T18:04:11.448 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T18:04:11.490 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:11.705 INFO:teuthology.orchestra.run.smithi070.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T18:04:11.733 INFO:teuthology.orchestra.run.smithi070.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:11.772 INFO:teuthology.orchestra.run.smithi070.stdout:Unpacking rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:11.876 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T18:04:11.904 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T18:04:11.937 INFO:teuthology.orchestra.run.smithi040.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T18:04:11.942 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:11.954 INFO:teuthology.orchestra.run.smithi040.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:11.995 INFO:teuthology.orchestra.run.smithi040.stdout:Unpacking rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:12.067 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:12.245 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T18:04:12.273 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T18:04:12.309 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:12.311 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:12.326 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:04:12.428 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:12.550 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:04:12.640 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T18:04:12.668 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T18:04:12.678 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T18:04:12.684 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:12.706 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:12.958 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T18:04:13.018 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T18:04:13.045 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T18:04:13.046 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T18:04:13.085 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:13.285 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T18:04:13.462 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-py. 2024-09-21T18:04:13.490 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-21T18:04:13.494 INFO:teuthology.orchestra.run.smithi070.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T18:04:13.528 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T18:04:13.563 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:13.712 INFO:teuthology.orchestra.run.smithi040.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T18:04:13.778 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:13.865 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T18:04:13.893 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T18:04:13.932 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:14.079 INFO:teuthology.orchestra.run.smithi070.stdout:Adding system user cephadm....done 2024-09-21T18:04:14.235 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:04:14.244 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user cephadm....done 2024-09-21T18:04:14.345 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T18:04:14.372 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T18:04:14.401 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:04:14.402 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:14.434 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T18:04:14.462 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T18:04:14.508 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:14.511 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:04:14.677 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:04:14.713 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-toml. 2024-09-21T18:04:14.741 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-21T18:04:14.771 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:14.779 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T18:04:14.851 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-test. 2024-09-21T18:04:14.868 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../098-ceph-test_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:14.897 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:04:14.909 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:14.937 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:15.063 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:04:15.082 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T18:04:15.110 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T18:04:15.149 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:15.156 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T18:04:15.313 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T18:04:15.408 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:04:15.494 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T18:04:15.522 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T18:04:15.534 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T18:04:15.560 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:15.573 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:04:15.674 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T18:04:15.830 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T18:04:15.835 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:04:15.858 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T18:04:15.896 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:15.968 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:04:16.195 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:16.345 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:16.455 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T18:04:16.604 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T18:04:16.650 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package radosgw. 2024-09-21T18:04:16.679 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../117-radosgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:16.716 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:16.739 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:04:16.880 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:04:16.957 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:04:17.157 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:04:17.208 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:17.408 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:17.451 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:04:17.636 INFO:teuthology.orchestra.run.smithi160.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T18:04:17.659 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:04:17.665 INFO:teuthology.orchestra.run.smithi160.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:17.702 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:17.702 INFO:teuthology.orchestra.run.smithi160.stdout:Unpacking rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:17.927 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:17.970 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:04:17.999 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:18.104 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-21T18:04:18.220 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:04:18.230 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:18.249 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:04:18.346 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-21T18:04:18.347 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:04:18.375 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:18.448 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:04:18.472 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:18.544 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T18:04:18.573 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-5073-gc69a6b39-1jammy_all.deb ... 2024-09-21T18:04:18.581 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:04:18.619 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:18.628 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T18:04:18.682 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:04:18.682 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:04:18.919 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T18:04:18.925 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:04:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:18.965 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T18:04:18.985 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:19.072 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T18:04:19.106 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T18:04:19.169 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:04:19.295 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:19.332 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T18:04:19.332 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T18:04:19.361 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T18:04:19.364 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T18:04:19.407 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:19.426 INFO:teuthology.orchestra.run.smithi160.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T18:04:19.428 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:04:19.485 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:19.554 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:19.688 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:19.759 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T18:04:19.787 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T18:04:19.814 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T18:04:19.833 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:19.922 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:19.951 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user cephadm....done 2024-09-21T18:04:20.023 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T18:04:20.073 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:04:20.091 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:04:20.209 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T18:04:20.238 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T18:04:20.274 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:04:20.284 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:20.316 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:20.351 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:04:20.525 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:20.609 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T18:04:20.609 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:20.703 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package pkg-config. 2024-09-21T18:04:20.727 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:04:20.732 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T18:04:20.735 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:20.778 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:20.835 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T18:04:20.961 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:20.979 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T18:04:21.060 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T18:04:21.088 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T18:04:21.126 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:21.238 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:04:21.364 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T18:04:21.378 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:04:21.565 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T18:04:21.594 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T18:04:21.640 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:21.680 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:04:21.682 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:04:21.819 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:04:21.947 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T18:04:21.947 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:04:21.976 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T18:04:22.021 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:22.034 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:22.057 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:22.130 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:22.158 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:04:22.239 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:04:22.269 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T18:04:22.396 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T18:04:22.425 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T18:04:22.450 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T18:04:22.462 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:22.515 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T18:04:22.537 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:04:22.776 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T18:04:22.780 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:04:22.805 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T18:04:22.835 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T18:04:22.843 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:22.900 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T18:04:23.031 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:23.086 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:04:23.142 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:04:23.183 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-py. 2024-09-21T18:04:23.212 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-21T18:04:23.249 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T18:04:23.274 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:04:23.479 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:04:23.508 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:23.519 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:04:23.582 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T18:04:23.611 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T18:04:23.657 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:23.781 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:04:23.804 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:04:23.835 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:04:23.968 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-21T18:04:24.069 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:24.107 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:04:24.116 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T18:04:24.138 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:04:24.145 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T18:04:24.191 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:24.204 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:04:24.330 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:04:24.342 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:24.381 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:24.476 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:24.482 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:24.529 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-toml. 2024-09-21T18:04:24.558 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-21T18:04:24.589 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:04:24.595 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T18:04:24.758 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:04:24.794 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:04:24.902 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T18:04:24.931 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T18:04:24.970 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T18:04:24.977 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:25.003 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T18:04:25.067 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:04:25.084 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:04:25.120 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:04:25.193 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:25.327 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T18:04:25.356 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T18:04:25.393 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:25.544 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:25.653 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T18:04:25.673 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T18:04:25.702 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T18:04:25.740 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:25.921 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:04:26.189 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:26.240 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:26.351 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package radosgw. 2024-09-21T18:04:26.460 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../117-radosgw_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:26.491 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:26.499 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T18:04:26.624 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:26.625 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T18:04:26.893 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:04:26.953 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:27.019 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:27.087 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T18:04:27.154 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:04:27.276 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:04:27.355 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:04:27.456 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:27.463 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:27.519 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:04:27.557 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:04:27.655 INFO:teuthology.orchestra.run.smithi029.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T18:04:27.661 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:27.684 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-5073-gc69a6b39-1jammy_amd64.deb ... 2024-09-21T18:04:27.697 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T18:04:27.730 INFO:teuthology.orchestra.run.smithi029.stdout:Unpacking rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:27.796 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:04:27.850 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:27.932 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:28.041 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:28.073 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T18:04:28.088 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T18:04:28.135 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T18:04:28.324 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T18:04:28.342 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:04:28.403 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:28.459 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T18:04:28.529 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:28.593 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T18:04:28.656 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T18:04:28.734 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T18:04:28.853 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T18:04:28.863 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:04:28.899 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:04:29.075 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T18:04:29.112 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:04:29.114 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T18:04:29.284 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:04:29.347 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T18:04:29.374 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T18:04:29.535 INFO:teuthology.orchestra.run.smithi029.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T18:04:29.576 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:04:29.581 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:04:29.605 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:29.625 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:04:29.698 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T18:04:29.902 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T18:04:29.903 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:04:29.991 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:30.146 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:30.151 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:04:30.169 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:04:30.176 INFO:teuthology.orchestra.run.smithi029.stdout:Adding system user cephadm....done 2024-09-21T18:04:30.272 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:30.295 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T18:04:30.344 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T18:04:30.443 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:04:30.573 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:04:30.604 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:30.621 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T18:04:30.730 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:04:30.778 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:04:30.882 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T18:04:30.899 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:04:31.011 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T18:04:31.031 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:04:31.037 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:04:31.163 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:31.264 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:31.271 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T18:04:31.333 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:04:31.336 INFO:teuthology.orchestra.run.smithi070.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T18:04:31.540 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T18:04:31.566 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:31.575 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:04:31.659 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T18:04:31.691 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:31.701 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:31.828 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:31.959 INFO:teuthology.orchestra.run.smithi040.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T18:04:31.963 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T18:04:32.046 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:32.093 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:04:32.213 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:32.265 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:32.339 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:32.342 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T18:04:32.399 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T18:04:32.427 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T18:04:32.562 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T18:04:32.613 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T18:04:32.658 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:04:32.723 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:04:32.784 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:32.821 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T18:04:32.910 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:04:32.930 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:04:32.966 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T18:04:33.084 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T18:04:33.160 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T18:04:33.203 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:33.248 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:33.352 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:33.412 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T18:04:33.464 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T18:04:33.478 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:04:33.558 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:04:33.674 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T18:04:33.707 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:33.821 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:33.833 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:33.834 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:33.934 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-ceph-argparse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:33.984 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T18:04:34.135 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:04:34.138 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:04:34.260 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T18:04:34.294 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T18:04:34.369 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T18:04:34.414 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:34.421 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-21T18:04:34.515 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T18:04:34.549 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T18:04:34.611 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T18:04:34.630 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:34.676 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T18:04:34.803 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T18:04:34.808 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T18:04:34.881 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:04:34.896 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:04:35.063 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T18:04:35.147 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T18:04:35.187 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:35.258 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:35.431 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:04:35.455 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:04:35.484 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T18:04:35.523 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T18:04:35.540 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T18:04:35.542 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:04:35.588 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T18:04:35.715 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T18:04:35.817 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:35.831 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:35.870 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:04:35.934 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:04:35.996 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.102 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:36.116 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:04:36.122 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.227 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:04:36.228 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T18:04:36.265 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:04:36.477 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:04:36.497 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T18:04:36.519 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:04:36.534 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.603 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.652 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.729 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:36.771 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:04:36.774 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T18:04:36.786 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:36.855 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:04:36.897 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:37.023 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:37.054 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.079 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T18:04:37.103 INFO:teuthology.orchestra.run.smithi160.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T18:04:37.155 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.191 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.205 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T18:04:37.281 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.301 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.358 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:37.410 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:37.483 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:37.492 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:37.492 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:37.679 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.805 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:37.868 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:04:37.884 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T18:04:37.939 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.001 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.125 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.128 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T18:04:38.144 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T18:04:38.188 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:38.189 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:38.251 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.254 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T18:04:38.273 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T18:04:38.377 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:38.408 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T18:04:38.538 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:38.657 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.674 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:04:38.751 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T18:04:38.783 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.900 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:38.982 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:39.010 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:39.137 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T18:04:39.275 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:04:39.391 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T18:04:39.534 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:39.547 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:39.660 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T18:04:39.673 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:39.777 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T18:04:39.799 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:39.925 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:39.927 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T18:04:40.051 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.071 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T18:04:40.255 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.297 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:40.381 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.425 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T18:04:40.507 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.557 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:04:40.594 INFO:teuthology.orchestra.run.smithi070.stdout:Adding group ceph....done 2024-09-21T18:04:40.633 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.678 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T18:04:40.759 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.804 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:40.933 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:40.961 INFO:teuthology.orchestra.run.smithi070.stdout:Adding system user ceph....done 2024-09-21T18:04:41.115 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T18:04:41.131 INFO:teuthology.orchestra.run.smithi070.stdout:Setting system user ceph properties....done 2024-09-21T18:04:41.145 INFO:teuthology.orchestra.run.smithi070.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T18:04:41.218 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:04:41.279 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T18:04:41.294 INFO:teuthology.orchestra.run.smithi040.stdout:Adding group ceph....done 2024-09-21T18:04:41.477 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T18:04:41.579 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:04:41.612 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T18:04:41.629 INFO:teuthology.orchestra.run.smithi040.stdout:Adding system user ceph....done 2024-09-21T18:04:41.688 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:41.748 INFO:teuthology.orchestra.run.smithi040.stdout:Setting system user ceph properties....done 2024-09-21T18:04:41.767 INFO:teuthology.orchestra.run.smithi040.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T18:04:41.790 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:41.911 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T18:04:41.924 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:04:42.085 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:42.211 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:42.258 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T18:04:42.277 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:42.403 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:42.529 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:42.671 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:42.671 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:42.732 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T18:04:42.805 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:42.906 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.010 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:43.015 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.062 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.129 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.146 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T18:04:43.188 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.228 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:43.228 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:43.415 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T18:04:43.473 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T18:04:43.542 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T18:04:43.666 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:43.666 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:43.669 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T18:04:43.700 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.826 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.952 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.956 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:43.973 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T18:04:44.086 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:44.131 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:44.169 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:44.169 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:44.233 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T18:04:44.494 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:44.495 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T18:04:44.620 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T18:04:44.649 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:44.866 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:44.866 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:44.933 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T18:04:44.958 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.193 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:45.193 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:45.193 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T18:04:45.290 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.351 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.391 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.454 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T18:04:45.534 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.623 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:45.624 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:45.660 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.668 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.714 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T18:04:45.786 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:45.896 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:46.001 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T18:04:46.094 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:46.220 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:46.278 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T18:04:46.336 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:46.337 INFO:teuthology.orchestra.run.smithi160.stdout:Adding group ceph....done 2024-09-21T18:04:46.371 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:46.405 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T18:04:46.589 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:46.589 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:46.649 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:46.649 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:46.690 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T18:04:46.819 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T18:04:46.821 INFO:teuthology.orchestra.run.smithi160.stdout:Adding system user ceph....done 2024-09-21T18:04:46.973 INFO:teuthology.orchestra.run.smithi160.stdout:Setting system user ceph properties....done 2024-09-21T18:04:46.991 INFO:teuthology.orchestra.run.smithi160.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T18:04:47.063 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.104 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T18:04:47.121 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.131 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T18:04:47.189 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.290 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.406 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T18:04:47.425 INFO:teuthology.orchestra.run.smithi070.stdout:Setting up ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.443 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.454 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T18:04:47.595 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:47.667 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T18:04:47.803 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T18:04:47.815 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:47.815 INFO:teuthology.orchestra.run.smithi040.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:47.930 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T18:04:47.936 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.020 INFO:teuthology.orchestra.run.smithi029.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T18:04:48.021 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T18:04:48.062 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.121 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T18:04:48.265 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T18:04:48.279 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.294 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T18:04:48.419 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T18:04:48.430 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.539 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:48.539 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:04:48.556 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.707 INFO:teuthology.orchestra.run.smithi040.stdout:Setting up ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:48.943 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T18:04:49.022 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:49.060 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T18:04:49.169 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T18:04:49.316 INFO:teuthology.orchestra.run.smithi070.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T18:04:49.321 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T18:04:49.373 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T18:04:49.600 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T18:04:49.727 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T18:04:49.841 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:49.982 INFO:teuthology.orchestra.run.smithi070.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T18:04:49.987 DEBUG:teuthology.parallel:result is None 2024-09-21T18:04:49.988 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T18:04:50.062 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:50.062 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:04:50.115 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T18:04:50.447 INFO:teuthology.orchestra.run.smithi040.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T18:04:50.468 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T18:04:50.551 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:50.767 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:50.767 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:04:50.778 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T18:04:51.073 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T18:04:51.164 INFO:teuthology.orchestra.run.smithi040.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T18:04:51.169 DEBUG:teuthology.parallel:result is None 2024-09-21T18:04:51.200 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T18:04:51.227 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:51.477 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T18:04:51.492 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:51.492 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:04:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T18:04:51.970 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:52.096 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:52.137 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T18:04:52.239 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:52.432 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:52.432 INFO:teuthology.orchestra.run.smithi160.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:04:52.557 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T18:04:52.862 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T18:04:52.898 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.033 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.158 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.224 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T18:04:53.309 INFO:teuthology.orchestra.run.smithi160.stdout:Setting up ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.353 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librados2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.480 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libsqlite3-mod-ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.607 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T18:04:53.662 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T18:04:53.771 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T18:04:53.947 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libcephfs2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:53.958 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T18:04:54.074 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libradosstriper1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:54.201 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T18:04:54.489 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librbd1 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:54.616 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-modules-core (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:54.743 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:54.965 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:54.965 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T18:04:55.150 INFO:teuthology.orchestra.run.smithi160.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T18:04:55.520 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up libcephfs-dev (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:55.640 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rados (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:55.767 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up librgw2 (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:55.771 INFO:teuthology.orchestra.run.smithi160.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T18:04:55.776 DEBUG:teuthology.parallel:result is None 2024-09-21T18:04:55.894 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T18:04:57.179 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rbd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:57.306 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up rbd-fuse (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:57.433 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-rgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:57.570 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up python3-cephfs (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:57.696 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-common (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:04:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:Adding group ceph....done 2024-09-21T18:04:58.684 INFO:teuthology.orchestra.run.smithi029.stdout:Adding system user ceph....done 2024-09-21T18:04:58.854 INFO:teuthology.orchestra.run.smithi029.stdout:Setting system user ceph properties....done 2024-09-21T18:04:58.873 INFO:teuthology.orchestra.run.smithi029.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T18:04:59.019 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T18:04:59.395 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T18:04:59.876 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-test (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:00.003 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up radosgw (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:00.584 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:05:00.584 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T18:05:01.091 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-base (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:01.447 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T18:05:01.937 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mds (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:02.261 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:05:02.261 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T18:05:03.118 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:03.349 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:05:03.349 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T18:05:03.854 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-osd (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:04.137 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:05:04.137 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T18:05:04.626 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-k8sevents (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:04.947 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:05.099 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mon (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:05.321 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:05:05.321 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T18:05:05.811 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-cephadm (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:05.980 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:06.107 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-mgr-dashboard (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:06.259 INFO:teuthology.orchestra.run.smithi029.stdout:Setting up ceph-volume (19.3.0-5073-gc69a6b39-1jammy) ... 2024-09-21T18:05:06.623 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T18:05:06.733 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T18:05:06.887 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T18:05:08.057 INFO:teuthology.orchestra.run.smithi029.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T18:05:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T18:05:08.786 DEBUG:teuthology.parallel:result is None 2024-09-21T18:05:08.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:05:08.928 DEBUG:teuthology.orchestra.run.smithi029:> dpkg-query -W -f '${Version}' ceph 2024-09-21T18:05:08.948 INFO:teuthology.orchestra.run.smithi029.stdout:19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:08.948 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:08.948 INFO:teuthology.task.install:The correct ceph version 19.3.0-5073-gc69a6b39-1jammy is installed. 2024-09-21T18:05:08.950 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:05:09.090 DEBUG:teuthology.orchestra.run.smithi040:> dpkg-query -W -f '${Version}' ceph 2024-09-21T18:05:09.109 INFO:teuthology.orchestra.run.smithi040.stdout:19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.110 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.110 INFO:teuthology.task.install:The correct ceph version 19.3.0-5073-gc69a6b39-1jammy is installed. 2024-09-21T18:05:09.112 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:05:09.249 DEBUG:teuthology.orchestra.run.smithi070:> dpkg-query -W -f '${Version}' ceph 2024-09-21T18:05:09.268 INFO:teuthology.orchestra.run.smithi070.stdout:19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.268 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.268 INFO:teuthology.task.install:The correct ceph version 19.3.0-5073-gc69a6b39-1jammy is installed. 2024-09-21T18:05:09.270 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c69a6b39fef6dd04706886608793361ef510a986 2024-09-21T18:05:09.414 DEBUG:teuthology.orchestra.run.smithi160:> dpkg-query -W -f '${Version}' ceph 2024-09-21T18:05:09.433 INFO:teuthology.orchestra.run.smithi160.stdout:19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.433 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-5073-gc69a6b39-1jammy 2024-09-21T18:05:09.433 INFO:teuthology.task.install:The correct ceph version 19.3.0-5073-gc69a6b39-1jammy is installed. 2024-09-21T18:05:09.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T18:05:09.435 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:09.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:05:09.450 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:09.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:05:09.464 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:09.464 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:05:09.478 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:09.478 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T18:05:09.491 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T18:05:09.491 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:09.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:05:09.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:05:09.562 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:09.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:05:09.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:05:09.630 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:09.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:05:09.644 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:05:09.696 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:09.696 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T18:05:09.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T18:05:09.764 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T18:05:09.764 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:09.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:05:09.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:05:09.834 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:09.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:05:09.848 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:05:09.907 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:09.907 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:05:09.920 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:05:09.977 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:09.977 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T18:05:09.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T18:05:10.044 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T18:05:10.044 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:10.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:05:10.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:05:10.114 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:10.115 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:05:10.128 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:05:10.186 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:10.186 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:05:10.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:05:10.252 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:10.252 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T18:05:10.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T18:05:10.319 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T18:05:10.393 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T18:05:10.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-09-21T18:05:10.395 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 777 /var/log/ceph 2024-09-21T18:05:10.397 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-09-21T18:05:10.399 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 777 /var/log/ceph 2024-09-21T18:05:10.412 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T18:05:10.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T18:05:10.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T18:05:10.457 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T18:05:10.459 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T18:05:10.471 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T18:05:10.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T18:05:10.515 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T18:05:10.517 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T18:05:10.520 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T18:05:10.551 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T18:05:10.551 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-21T18:05:10.552 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219', 'branch': 'wip-rf-qprio-w-qa', 'description': 'rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/cache}', 'email': None, 'first_in_suite': False, 'job_id': '7914219', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-rf-qprio-w-qa'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-qprio-w-qa', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}}, 'owner': 'scheduled_rfriedma@teuthology', 'priority': 75, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'osd.0', 'osd.4', 'osd.8', 'osd.12'], ['mon.b', 'osd.1', 'osd.5', 'osd.9', 'osd.13'], ['mon.c', 'osd.2', 'osd.6', 'osd.10', 'osd.14'], ['mgr.x', 'osd.3', 'osd.7', 'osd.11', 'osd.15', 'client.0']], 'seed': 9109, 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986', 'sleep_before_teardown': 0, 'subset': '3/444', 'suite': 'rados', 'suite_branch': 'wip-rf-qprio-w-qa', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c69a6b39fef6dd04706886608793361ef510a986/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c69a6b39fef6dd04706886608793361ef510a986', 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDfGfx3GUkiesxylWtdDqhksjAWAN584U+jEQ+xyMJMNmqLTC5GVWGmS2rzo1qEUkvlZJbsQS6s5o+OZY9nrdL4=', 'smithi040.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDeKb+ps/cSLHfthzbjB9j+EEoGZXGJunEMA8nMsHCDjWV2k4tvyLn8r28ZASRK87Y2dyAoXMyQWLhgJFP7i2Ao=', 'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOqXSJyMZSCAiEgvt5zfxAeLr3tvqXmfCKl35I1JIElFTvkBDMI86gFiTxvrl3NkF8k6z+9bkyjH3tUd2jiXxkg=', 'smithi160.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLwo1yNojRWJw3dt7SfwCf53/m0v1TwVITZWQ8uldbKo3Bei0al/m6+12EfZ8jJEkNqOYqnMrjMKps/FTKPnaGU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986', 'cluster': 'ceph'}}, {'exec': {'mon.a': ['ceph config set mgr mgr/balancer/log_level debug', 'ceph balancer status', 'ceph osd set-require-min-compat-client reef', 'ceph balancer mode upmap-read', 'ceph balancer on', 'ceph balancer status', 'ceph balancer status detail']}}, {'thrashosds': {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1800}}, {'exec': {'client.0': ['sudo ceph osd pool create base 4', 'sudo ceph osd pool application enable base rados', 'sudo ceph osd pool create cache 4', 'sudo ceph osd tier add base cache', 'sudo ceph osd tier cache-mode cache writeback', 'sudo ceph osd tier set-overlay base cache', 'sudo ceph osd pool set cache hit_set_type bloom', 'sudo ceph osd pool set cache hit_set_count 8', 'sudo ceph osd pool set cache hit_set_period 3600', 'sudo ceph osd pool set cache min_read_recency_for_promote 0', 'sudo ceph osd pool set cache min_write_recency_for_promote 0']}}, {'rados': {'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_17:35:20', 'tube': 'smithi', 'user': 'rfriedma', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T18:05:10.552 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T18:05:10.571 DEBUG:teuthology.orchestra.run.smithi040:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T18:05:10.573 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T18:05:10.576 DEBUG:teuthology.orchestra.run.smithi160:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T18:05:10.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T18:05:10.619 DEBUG:teuthology.orchestra.run.smithi040:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T18:05:10.623 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T18:05:10.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T18:05:10.640 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:10.640 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:05:10.678 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:05:10.678 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 17:59:14.740835777 +0000 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 17:59:14.456838313 +0000 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 17:59:14.456838313 +0000 2024-09-21T18:05:10.727 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-09-21T18:05:10.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:05:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T18:05:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T18:05:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000306684 s, 1.7 MB/s 2024-09-21T18:05:10.783 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:05:10.832 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 17:59:15.216831524 +0000 2024-09-21T18:05:10.878 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 17:59:14.960833812 +0000 2024-09-21T18:05:10.879 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 17:59:14.960833812 +0000 2024-09-21T18:05:10.879 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-09-21T18:05:10.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:05:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T18:05:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T18:05:10.933 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000300557 s, 1.7 MB/s 2024-09-21T18:05:10.934 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:05:10.984 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 17:59:15.692827273 +0000 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 17:59:15.436829559 +0000 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 17:59:15.436829559 +0000 2024-09-21T18:05:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-09-21T18:05:11.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:05:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T18:05:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T18:05:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000297501 s, 1.7 MB/s 2024-09-21T18:05:11.086 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:05:11.136 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-09-21T18:05:11.182 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:05:11.182 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 939 Links: 1 2024-09-21T18:05:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.183 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-09-21 17:59:15.916825272 +0000 2024-09-21T18:05:11.183 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-09-21 17:59:15.912825308 +0000 2024-09-21T18:05:11.183 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-09-21 17:59:15.912825308 +0000 2024-09-21T18:05:11.183 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: - 2024-09-21T18:05:11.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:05:11.236 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-09-21T18:05:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-09-21T18:05:11.237 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000295445 s, 1.7 MB/s 2024-09-21T18:05:11.238 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:05:11.287 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:11.288 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:11.288 DEBUG:teuthology.orchestra.run.smithi040:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:05:11.293 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:05:11.293 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_1 2024-09-21T18:05:11.339 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:05:11.339 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 881 Links: 1 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-21 17:59:06.481226503 +0000 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-21 17:59:06.233215822 +0000 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-21 17:59:06.233215822 +0000 2024-09-21T18:05:11.340 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-09-21T18:05:11.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:05:11.393 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-21T18:05:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-21T18:05:11.394 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000390597 s, 1.3 MB/s 2024-09-21T18:05:11.395 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:05:11.444 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_2 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 900 Links: 1 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-21 17:59:06.949246659 +0000 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-21 17:59:06.701235978 +0000 2024-09-21T18:05:11.495 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-21 17:59:06.701235978 +0000 2024-09-21T18:05:11.496 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-09-21T18:05:11.496 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:05:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-21T18:05:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-21T18:05:11.549 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000407987 s, 1.3 MB/s 2024-09-21T18:05:11.550 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:05:11.600 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_3 2024-09-21T18:05:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-21 17:59:07.445267135 +0000 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-21 17:59:07.201257114 +0000 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-21 17:59:07.201257114 +0000 2024-09-21T18:05:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-09-21T18:05:11.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:05:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-21T18:05:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-21T18:05:11.706 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000429517 s, 1.2 MB/s 2024-09-21T18:05:11.707 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:05:11.760 DEBUG:teuthology.orchestra.run.smithi040:> stat /dev/vg_nvme/lv_4 2024-09-21T18:05:11.807 INFO:teuthology.orchestra.run.smithi040.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:05:11.807 INFO:teuthology.orchestra.run.smithi040.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.807 INFO:teuthology.orchestra.run.smithi040.stdout:Device: 5h/5d Inode: 932 Links: 1 2024-09-21T18:05:11.807 INFO:teuthology.orchestra.run.smithi040.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.808 INFO:teuthology.orchestra.run.smithi040.stdout:Access: 2024-09-21 17:59:07.913286354 +0000 2024-09-21T18:05:11.808 INFO:teuthology.orchestra.run.smithi040.stdout:Modify: 2024-09-21 17:59:07.685276991 +0000 2024-09-21T18:05:11.808 INFO:teuthology.orchestra.run.smithi040.stdout:Change: 2024-09-21 17:59:07.685276991 +0000 2024-09-21T18:05:11.808 INFO:teuthology.orchestra.run.smithi040.stdout: Birth: - 2024-09-21T18:05:11.808 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:05:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records in 2024-09-21T18:05:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:1+0 records out 2024-09-21T18:05:11.862 INFO:teuthology.orchestra.run.smithi040.stderr:512 bytes copied, 0.000391679 s, 1.3 MB/s 2024-09-21T18:05:11.863 DEBUG:teuthology.orchestra.run.smithi040:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:05:11.912 INFO:tasks.ceph:osd dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:11.913 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:11.913 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:05:11.919 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:05:11.919 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-09-21T18:05:11.965 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:05:11.965 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 17:59:00.750824779 +0000 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 17:59:00.514826887 +0000 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 17:59:00.514826887 +0000 2024-09-21T18:05:11.966 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-09-21T18:05:11.966 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:05:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T18:05:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T18:05:12.020 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000320463 s, 1.6 MB/s 2024-09-21T18:05:12.021 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:05:12.070 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 17:59:01.214820635 +0000 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 17:59:00.974822779 +0000 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 17:59:00.974822779 +0000 2024-09-21T18:05:12.117 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-09-21T18:05:12.118 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:05:12.171 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T18:05:12.172 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T18:05:12.172 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000297495 s, 1.7 MB/s 2024-09-21T18:05:12.173 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:05:12.222 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-09-21T18:05:12.268 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 922 Links: 1 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 17:59:01.682816455 +0000 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 17:59:01.442818599 +0000 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 17:59:01.442818599 +0000 2024-09-21T18:05:12.269 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-09-21T18:05:12.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:05:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T18:05:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T18:05:12.323 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000320264 s, 1.6 MB/s 2024-09-21T18:05:12.324 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:05:12.374 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-09-21T18:05:12.421 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:05:12.421 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.421 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 940 Links: 1 2024-09-21T18:05:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-09-21 17:59:01.918814347 +0000 2024-09-21T18:05:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-09-21 17:59:01.914814383 +0000 2024-09-21T18:05:12.422 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-09-21 17:59:01.914814383 +0000 2024-09-21T18:05:12.422 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: - 2024-09-21T18:05:12.422 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:05:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-09-21T18:05:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-09-21T18:05:12.475 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000298581 s, 1.7 MB/s 2024-09-21T18:05:12.476 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:05:12.527 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:12.527 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:12.527 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T18:05:12.533 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T18:05:12.533 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_1 2024-09-21T18:05:12.581 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 17:59:01.950852809 +0000 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 17:59:01.706854990 +0000 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 17:59:01.706854990 +0000 2024-09-21T18:05:12.582 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2024-09-21T18:05:12.582 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T18:05:12.636 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T18:05:12.636 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T18:05:12.636 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00045152 s, 1.1 MB/s 2024-09-21T18:05:12.637 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T18:05:12.686 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_2 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 910 Links: 1 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 17:59:02.194850630 +0000 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 17:59:02.190850666 +0000 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 17:59:02.190850666 +0000 2024-09-21T18:05:12.733 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2024-09-21T18:05:12.733 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T18:05:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T18:05:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T18:05:12.788 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000374597 s, 1.4 MB/s 2024-09-21T18:05:12.789 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T18:05:12.838 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_3 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 17:59:02.982843591 +0000 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 17:59:02.694846164 +0000 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 17:59:02.694846164 +0000 2024-09-21T18:05:12.885 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2024-09-21T18:05:12.886 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T18:05:12.939 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T18:05:12.939 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T18:05:12.939 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.00037715 s, 1.4 MB/s 2024-09-21T18:05:12.940 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T18:05:12.990 DEBUG:teuthology.orchestra.run.smithi160:> stat /dev/vg_nvme/lv_4 2024-09-21T18:05:13.036 INFO:teuthology.orchestra.run.smithi160.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout:Device: 5h/5d Inode: 938 Links: 1 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout:Access: 2024-09-21 17:59:03.454839375 +0000 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout:Modify: 2024-09-21 17:59:03.206841592 +0000 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout:Change: 2024-09-21 17:59:03.206841592 +0000 2024-09-21T18:05:13.037 INFO:teuthology.orchestra.run.smithi160.stdout: Birth: - 2024-09-21T18:05:13.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T18:05:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records in 2024-09-21T18:05:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:1+0 records out 2024-09-21T18:05:13.091 INFO:teuthology.orchestra.run.smithi160.stderr:512 bytes copied, 0.000333782 s, 1.5 MB/s 2024-09-21T18:05:13.092 DEBUG:teuthology.orchestra.run.smithi160:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T18:05:13.142 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:13.142 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-21T18:05:13.142 INFO:tasks.ceph:Generating config... 2024-09-21T18:05:13.143 INFO:tasks.ceph:[osd] debug monc = 20 2024-09-21T18:05:13.143 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T18:05:13.143 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-21T18:05:13.143 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] osd beacon report interval = 30 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-21T18:05:13.144 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd inject bad map crc probability = 0.1 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd map cache size = 1 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd max backfills = 6 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd max scrubs = 5 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd scrub during recovery = False 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd_max_pg_log_entries = 3000 2024-09-21T18:05:13.145 INFO:tasks.ceph:[osd] osd_min_pg_log_entries = 3000 2024-09-21T18:05:13.145 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] ms type = async 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd debug inject dispatch delay duration = 0.1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd debug inject dispatch delay probability = 0.1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd_async_recovery_min_cost = 1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd_object_clean_region_max_num_intervals = 1000 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2024-09-21T18:05:13.146 INFO:tasks.ceph:[global] osd_pool_default_size = 2 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T18:05:13.146 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-09-21T18:05:13.147 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-21T18:05:13.147 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-21T18:05:13.147 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-21T18:05:13.147 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-21T18:05:13.147 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-09-21T18:05:13.147 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T18:05:13.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T18:05:13.183 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T18:05:13.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T18:05:13.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T18:05:13.338 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.29:6789]'), ('mon.b', '[v1:172.21.15.40:6789]'), ('mon.c', '[v1:172.21.15.70:6789]')] 2024-09-21T18:05:13.338 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]', 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-21T18:05:13.339 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:13.339 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T18:05:13.386 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.29:6789]' --addv b '[v1:172.21.15.40:6789]' --addv c '[v1:172.21.15.70:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:fsid ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2024-09-21T18:05:13.475413+0000 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-09-21T18:05:13.475413+0000 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 17 (quincy) 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:0: v1:172.21.15.29:6789/0 mon.a 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:1: v1:172.21.15.40:6789/0 mon.b 2024-09-21T18:05:13.475 INFO:teuthology.orchestra.run.smithi029.stdout:2: v1:172.21.15.70:6789/0 mon.c 2024-09-21T18:05:13.476 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T18:05:13.479 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T18:05:13.484 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ea312541-f204-4ac1-b2ae-f5daa6b7f078... 2024-09-21T18:05:13.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T18:05:13.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T18:05:13.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T18:05:13.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2024-09-21T18:05:13.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2024-09-21T18:05:13.561 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2024-09-21T18:05:13.562 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]" 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: mon election default strategy = 1 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: ms bind msgr2 = False 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: ms type = async 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd_async_recovery_min_cost = 1 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_min_size = 2 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_size = 2 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2024-09-21T18:05:13.563 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 120 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug trim objects = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2024-09-21T18:05:13.564 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: debug monc = 20 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression algorithm = snappy 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore compression mode = aggressive 2024-09-21T18:05:13.565 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore write v2 random = True 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore zero block detection = True 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 20 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 20 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 10 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd beacon report interval = 30 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd blocked scrub grace period = 3600 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify cached snaps = True 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify missing on start = True 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2024-09-21T18:05:13.566 INFO:teuthology.orchestra.run.smithi029.stdout: osd inject bad map crc probability = 0.1 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd map cache size = 1 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd max backfills = 6 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd max markdown count = 1000 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd max scrubs = 5 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock override recovery settings = True 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue = debug_random 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd op queue cut off = debug_random 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub during recovery = False 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub min interval = 60 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd shutdown pgref assert = True 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd_max_pg_log_entries = 3000 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: osd_min_pg_log_entries = 3000 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.567 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2024-09-21T18:05:13.568 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.569 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: mon min osdmap epochs = 50 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune interval = 2 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune min = 15 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: mon osdmap full prune txsize = 2 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: mon scrub interval = 300 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: paxos service trim min = 10 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2024-09-21T18:05:13.570 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2024-09-21T18:05:13.571 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2024-09-21T18:05:13.571 INFO:teuthology.orchestra.run.smithi160.stdout:[global] 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: chdir = "" 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: auth supported = cephx 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: filestore xattr use omap = true 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: mon clock drift allowed = 1.000 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: osd crush chooseleaf type = 0 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: auth debug = true 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on old message = true 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: ms die on bug = true 2024-09-21T18:05:13.572 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon pg warn max object skew = 0 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default size = 2 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow primary affinity = true 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd allow pg remap = true 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on legacy crush tunables = false 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on crush straw calc version zero = false 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on no sortbitwise = false 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on osd down out interval zero = false 2024-09-21T18:05:13.573 INFO:teuthology.orchestra.run.smithi160.stdout: mon warn on too few osds = false 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon_allow_pool_size_one = true 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: osd default data pool replay window = 5 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon allow pool delete = true 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon cluster log file level = debug 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: debug asserts on shutdown = true 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon health detail to clog = false 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]" 2024-09-21T18:05:13.574 INFO:teuthology.orchestra.run.smithi160.stdout: mon election default strategy = 1 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: ms bind msgr2 = False 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: ms type = async 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd_async_recovery_min_cost = 1 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_min_size = 2 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: osd_pool_default_size = 2 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: fsid = ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.575 INFO:teuthology.orchestra.run.smithi160.stdout:[osd] 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd journal size = 100 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub load threshold = 5.0 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub max interval = 120 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock profile = high_recovery_ops 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd recover clone overlap = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd recovery max chunk = 1048576 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug shutdown = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug op order = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify stray on activate = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug trim objects = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd open classes on start = true 2024-09-21T18:05:13.576 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug pg log writeout = true 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: osd deep scrub update digest min age = 30 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: osd map max advance = 10 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: journal zero on create = true 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: filestore ondisk finisher threads = 3 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: filestore apply finisher threads = 3 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: bdev debug aio = true 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug misdirected ops = true 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: debug monc = 20 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore block size = 96636764160 2024-09-21T18:05:13.577 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore compression algorithm = snappy 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore compression mode = aggressive 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore fsck on mount = True 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore write v2 random = True 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: bluestore zero block detection = True 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluefs = 20 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: debug bluestore = 20 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: debug osd = 20 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: debug rocksdb = 10 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd full ratio = 0.9 2024-09-21T18:05:13.578 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd nearfull ratio = 0.8 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd beacon report interval = 30 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd blocked scrub grace period = 3600 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify cached snaps = True 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd debug verify missing on start = True 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd failsafe full ratio = 0.95 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd inject bad map crc probability = 0.1 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd map cache size = 1 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd max backfills = 6 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd max markdown count = 1000 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd max scrubs = 5 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd mclock override recovery settings = True 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd objectstore = bluestore 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd op queue = debug_random 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd op queue cut off = debug_random 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub during recovery = False 2024-09-21T18:05:13.579 INFO:teuthology.orchestra.run.smithi160.stdout: osd scrub min interval = 60 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: osd shutdown pgref assert = True 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: osd_max_pg_log_entries = 3000 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: osd_min_pg_log_entries = 3000 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout:[mgr] 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug mgr = 20 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: mgr/telemetry/nag = false 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout:[mon] 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug ms = 1 2024-09-21T18:05:13.580 INFO:teuthology.orchestra.run.smithi160.stdout: debug mon = 20 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: debug paxos = 20 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: debug auth = 20 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon data avail warn = 5 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon mgr mkfs grace = 240 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd reporter subtree level = osd 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon osd prime pg temp = true 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T18:05:13.581 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: # 1m isn't quite enough 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon_down_mkfs_grace = 2m 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon_warn_on_filestore_osds = false 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon min osdmap epochs = 50 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon osdmap full prune interval = 2 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon osdmap full prune min = 15 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon osdmap full prune txsize = 2 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: mon scrub interval = 300 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: paxos service trim min = 10 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout:[client] 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: rgw cache enabled = true 2024-09-21T18:05:13.582 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable ops log = true 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout: rgw enable usage log = true 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.a] 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.b] 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi160.stdout:[mon.c] 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout:[global] 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: chdir = "" 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: auth supported = cephx 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: filestore xattr use omap = true 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: mon clock drift allowed = 1.000 2024-09-21T18:05:13.583 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd crush chooseleaf type = 0 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: auth debug = true 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on old message = true 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: ms die on bug = true 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: mon pg warn max object skew = 0 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default size = 2 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow primary affinity = true 2024-09-21T18:05:13.584 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd allow pg remap = true 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on legacy crush tunables = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on crush straw calc version zero = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on no sortbitwise = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on osd down out interval zero = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon warn on too few osds = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon_allow_pool_size_one = true 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: osd default data pool replay window = 5 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon allow pool delete = true 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.585 INFO:teuthology.orchestra.run.smithi040.stdout: mon cluster log file level = debug 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: debug asserts on shutdown = true 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: mon health detail to clog = false 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]" 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: mon election default strategy = 1 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: ms bind msgr2 = False 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: ms type = async 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd_async_recovery_min_cost = 1 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_min_size = 2 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: osd_pool_default_size = 2 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: fsid = ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.586 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout:[osd] 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd journal size = 100 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub load threshold = 5.0 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub max interval = 120 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock profile = high_recovery_ops 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd recover clone overlap = true 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd recovery max chunk = 1048576 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug shutdown = true 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug op order = true 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify stray on activate = true 2024-09-21T18:05:13.587 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug trim objects = true 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd open classes on start = true 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug pg log writeout = true 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd deep scrub update digest min age = 30 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd map max advance = 10 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: journal zero on create = true 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: filestore ondisk finisher threads = 3 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: filestore apply finisher threads = 3 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: bdev debug aio = true 2024-09-21T18:05:13.588 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug misdirected ops = true 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug monc = 20 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore block size = 96636764160 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression algorithm = snappy 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore compression mode = aggressive 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore fsck on mount = True 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore write v2 random = True 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: bluestore zero block detection = True 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluefs = 20 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug bluestore = 20 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug osd = 20 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: debug rocksdb = 10 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd full ratio = 0.9 2024-09-21T18:05:13.589 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd nearfull ratio = 0.8 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd beacon report interval = 30 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd blocked scrub grace period = 3600 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify cached snaps = True 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd debug verify missing on start = True 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd failsafe full ratio = 0.95 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd inject bad map crc probability = 0.1 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd map cache size = 1 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd max backfills = 6 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd max markdown count = 1000 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd max scrubs = 5 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd mclock override recovery settings = True 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd objectstore = bluestore 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue = debug_random 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd op queue cut off = debug_random 2024-09-21T18:05:13.590 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub during recovery = False 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd scrub min interval = 60 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd shutdown pgref assert = True 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd_max_pg_log_entries = 3000 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: osd_min_pg_log_entries = 3000 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr] 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: debug mgr = 20 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: mgr/telemetry/nag = false 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout:[mon] 2024-09-21T18:05:13.591 INFO:teuthology.orchestra.run.smithi040.stdout: debug ms = 1 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug mon = 20 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug paxos = 20 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: debug auth = 20 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon data avail warn = 5 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon mgr mkfs grace = 240 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd reporter subtree level = osd 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon osd prime pg temp = true 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T18:05:13.592 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: # 1m isn't quite enough 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon_down_mkfs_grace = 2m 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon_warn_on_filestore_osds = false 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon min osdmap epochs = 50 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune interval = 2 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune min = 15 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon osdmap full prune txsize = 2 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: mon scrub interval = 300 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: paxos service trim min = 10 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout: 2024-09-21T18:05:13.593 INFO:teuthology.orchestra.run.smithi040.stdout:[client] 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: rgw cache enabled = true 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable ops log = true 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: rgw enable usage log = true 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.a] 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.b] 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi040.stdout:[mon.c] 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-09-21T18:05:13.594 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.595 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.596 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = "[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]" 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: mon election default strategy = 1 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: ms bind msgr2 = False 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: ms type = async 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug inject dispatch delay duration = 0.1 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug inject dispatch delay probability = 0.1 2024-09-21T18:05:13.597 INFO:teuthology.orchestra.run.smithi070.stdout: osd_async_recovery_min_cost = 1 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd_object_clean_region_max_num_intervals = 1000 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_min_size = 2 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_size = 2 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 120 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-09-21T18:05:13.598 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.599 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression algorithm = snappy 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression mode = aggressive 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore write v2 random = True 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore zero block detection = True 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-21T18:05:13.600 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd beacon report interval = 30 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd blocked scrub grace period = 3600 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify cached snaps = True 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify missing on start = True 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd inject bad map crc probability = 0.1 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd map cache size = 1 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd max backfills = 6 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd max markdown count = 1000 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd max scrubs = 5 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock override recovery settings = True 2024-09-21T18:05:13.601 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue = debug_random 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue cut off = debug_random 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub during recovery = False 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub min interval = 60 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd_max_pg_log_entries = 3000 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: osd_min_pg_log_entries = 3000 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.602 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T18:05:13.603 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon min osdmap epochs = 50 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune interval = 2 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune min = 15 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune txsize = 2 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: mon scrub interval = 300 2024-09-21T18:05:13.604 INFO:teuthology.orchestra.run.smithi070.stdout: paxos service trim min = 10 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.b] 2024-09-21T18:05:13.605 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.c] 2024-09-21T18:05:13.607 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T18:05:13.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T18:05:13.700 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T18:05:13.700 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:13.700 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T18:05:13.706 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:13.706 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T18:05:13.753 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:05:13.753 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:13.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T18:05:13.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T18:05:13.823 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:05:13.823 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:13.870 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:05:13.870 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:13.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T18:05:13.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T18:05:13.894 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:05:13.894 DEBUG:teuthology.orchestra.run.smithi040:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:13.943 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:05:13.943 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:13.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T18:05:13.943 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T18:05:13.966 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:05:13.966 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:14.013 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:05:14.013 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:14.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T18:05:14.013 DEBUG:teuthology.orchestra.run.smithi160:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T18:05:14.037 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:05:14.037 DEBUG:teuthology.orchestra.run.smithi160:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:14.085 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T18:05:14.085 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T18:05:14.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T18:05:14.133 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T18:05:14.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T18:05:14.178 INFO:teuthology.orchestra.run.smithi160.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T18:05:14.187 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T18:05:14.187 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T18:05:14.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T18:05:14.261 INFO:teuthology.orchestra.run.smithi160.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T18:05:14.271 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T18:05:14.271 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi040.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi160.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'}} 2024-09-21T18:05:14.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T18:05:14.287 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:14.287 INFO:tasks.ceph:role: osd.0 2024-09-21T18:05:14.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:05:14.287 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:14.342 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:14.366 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-09-21T18:05:14.368 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-09-21T18:05:14.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T18:05:14.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T18:05:14.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:14.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-21T18:05:14.537 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:14.537 INFO:tasks.ceph:role: osd.4 2024-09-21T18:05:14.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:05:14.538 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T18:05:14.594 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:14.595 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:14.618 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-09-21T18:05:14.620 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-09-21T18:05:14.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-21T18:05:14.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-21T18:05:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:14.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:14.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-8 2024-09-21T18:05:14.786 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:14.786 INFO:tasks.ceph:role: osd.8 2024-09-21T18:05:14.786 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:05:14.786 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T18:05:14.841 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:14.842 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:14.866 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-09-21T18:05:14.867 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-09-21T18:05:14.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-8 2024-09-21T18:05:14.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-8 2024-09-21T18:05:14.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:14.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:14.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-12 2024-09-21T18:05:15.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.8': '/dev/vg_nvme/lv_3', 'osd.12': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:15.043 INFO:tasks.ceph:role: osd.12 2024-09-21T18:05:15.043 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:05:15.043 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T18:05:15.097 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:15.098 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:15.121 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2024-09-21T18:05:15.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2024-09-21T18:05:15.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-12 2024-09-21T18:05:15.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-12 2024-09-21T18:05:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:15.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:15.347+0000 7fdefc1ed7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T18:05:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:15.347+0000 7fdefc1ed7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T18:05:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:15.347+0000 7fdefc1ed7c0 -1 bdev(0x56495aa5d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:15.351+0000 7fdefc1ed7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T18:05:18.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T18:05:18.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:18.579+0000 7f223254c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-21T18:05:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:18.583+0000 7f223254c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-21T18:05:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:18.583+0000 7f223254c7c0 -1 bdev(0x55d7ba367800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:18.583+0000 7f223254c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-21T18:05:21.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T18:05:21.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 8 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:21.895+0000 7f33119697c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-8/keyring: can't open /var/lib/ceph/osd/ceph-8/keyring: (2) No such file or directory 2024-09-21T18:05:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:21.895+0000 7f33119697c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-8/keyring 2024-09-21T18:05:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:21.895+0000 7f33119697c0 -1 bdev(0x560bdf2f7800 /var/lib/ceph/osd/ceph-8/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:21.895+0000 7f33119697c0 -1 bluestore(/var/lib/ceph/osd/ceph-8) _read_fsid unparsable uuid 2024-09-21T18:05:25.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2024-09-21T18:05:25.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 12 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:25.187+0000 7f4271bdd7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-12/keyring: can't open /var/lib/ceph/osd/ceph-12/keyring: (2) No such file or directory 2024-09-21T18:05:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:25.187+0000 7f4271bdd7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-12/keyring 2024-09-21T18:05:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:25.187+0000 7f4271bdd7c0 -1 bdev(0x55ff23987800 /var/lib/ceph/osd/ceph-12/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:25.193 INFO:teuthology.orchestra.run.smithi029.stderr:2024-09-21T18:05:25.187+0000 7f4271bdd7c0 -1 bluestore(/var/lib/ceph/osd/ceph-12) _read_fsid unparsable uuid 2024-09-21T18:05:28.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-12 2024-09-21T18:05:28.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T18:05:28.386 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:28.386 INFO:tasks.ceph:role: osd.1 2024-09-21T18:05:28.386 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:05:28.386 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:28.451 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:28.452 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:28.925 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-21T18:05:28.927 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-21T18:05:28.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2024-09-21T18:05:29.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T18:05:29.091 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:29.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-21T18:05:29.147 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:29.147 INFO:tasks.ceph:role: osd.5 2024-09-21T18:05:29.147 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:05:29.147 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:29.211 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:29.660 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-21T18:05:29.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-21T18:05:29.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-21T18:05:29.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-21T18:05:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:29.806 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:29.806 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-9 2024-09-21T18:05:29.864 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:29.864 INFO:tasks.ceph:role: osd.9 2024-09-21T18:05:29.864 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:05:29.864 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:29.931 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:29.932 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:29.932 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:29.932 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:30.392 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-21T18:05:30.394 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-21T18:05:30.394 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-9 2024-09-21T18:05:30.529 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-9 2024-09-21T18:05:30.541 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:30.542 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/osd/ceph-13 2024-09-21T18:05:30.600 INFO:tasks.ceph:roles_to_devs: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.9': '/dev/vg_nvme/lv_3', 'osd.13': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:30.600 INFO:tasks.ceph:role: osd.13 2024-09-21T18:05:30.600 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:05:30.600 DEBUG:teuthology.orchestra.run.smithi040:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T18:05:30.663 INFO:teuthology.orchestra.run.smithi040.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:30.663 INFO:teuthology.orchestra.run.smithi040.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:30.663 INFO:teuthology.orchestra.run.smithi040.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:30.664 INFO:teuthology.orchestra.run.smithi040.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:30.664 INFO:teuthology.orchestra.run.smithi040.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:30.664 INFO:teuthology.orchestra.run.smithi040.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:30.664 INFO:teuthology.orchestra.run.smithi040.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:31.149 INFO:teuthology.orchestra.run.smithi040.stdout:Discarding blocks...Done. 2024-09-21T18:05:31.151 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi040.front.sepia.ceph.com -o noatime 2024-09-21T18:05:31.151 DEBUG:teuthology.orchestra.run.smithi040:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-13 2024-09-21T18:05:31.307 DEBUG:teuthology.orchestra.run.smithi040:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-13 2024-09-21T18:05:31.319 INFO:teuthology.orchestra.run.smithi040.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:31.320 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:31.439 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:31.435+0000 7f1f1123e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T18:05:31.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:31.435+0000 7f1f1123e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T18:05:31.440 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:31.435+0000 7f1f1123e7c0 -1 bdev(0x55b7f4009800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:31.441 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:31.435+0000 7f1f1123e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T18:05:34.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T18:05:34.597 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:34.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:34.675+0000 7f84484da7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-21T18:05:34.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:34.675+0000 7f84484da7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-21T18:05:34.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:34.675+0000 7f84484da7c0 -1 bdev(0x55ad76993800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:34.679 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:34.675+0000 7f84484da7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-21T18:05:37.835 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T18:05:37.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 9 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:37.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:37.971+0000 7f8e5e76e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-9/keyring: can't open /var/lib/ceph/osd/ceph-9/keyring: (2) No such file or directory 2024-09-21T18:05:37.975 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:37.971+0000 7f8e5e76e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-9/keyring 2024-09-21T18:05:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:37.971+0000 7f8e5e76e7c0 -1 bdev(0x562ba3603800 /var/lib/ceph/osd/ceph-9/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:37.976 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:37.971+0000 7f8e5e76e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-9) _read_fsid unparsable uuid 2024-09-21T18:05:41.109 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2024-09-21T18:05:41.164 DEBUG:teuthology.orchestra.run.smithi040:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 13 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:41.246 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:41.243+0000 7f0a19a737c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-13/keyring: can't open /var/lib/ceph/osd/ceph-13/keyring: (2) No such file or directory 2024-09-21T18:05:41.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:41.243+0000 7f0a19a737c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-13/keyring 2024-09-21T18:05:41.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:41.243+0000 7f0a19a737c0 -1 bdev(0x55efab8c1800 /var/lib/ceph/osd/ceph-13/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:41.247 INFO:teuthology.orchestra.run.smithi040.stderr:2024-09-21T18:05:41.243+0000 7f0a19a737c0 -1 bluestore(/var/lib/ceph/osd/ceph-13) _read_fsid unparsable uuid 2024-09-21T18:05:44.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-13 2024-09-21T18:05:44.455 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T18:05:44.469 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:44.469 INFO:tasks.ceph:role: osd.2 2024-09-21T18:05:44.469 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:05:44.469 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T18:05:44.523 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:44.524 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:44.548 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-21T18:05:44.629 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-21T18:05:44.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2024-09-21T18:05:44.689 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T18:05:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:44.742 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:44.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-21T18:05:44.795 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:44.796 INFO:tasks.ceph:role: osd.6 2024-09-21T18:05:44.796 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:05:44.796 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:44.853 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:44.876 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-21T18:05:44.878 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-21T18:05:44.878 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2024-09-21T18:05:44.934 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-21T18:05:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:44.985 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:44.985 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-10 2024-09-21T18:05:45.038 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:45.038 INFO:tasks.ceph:role: osd.10 2024-09-21T18:05:45.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:05:45.038 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:45.090 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:45.091 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:45.113 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-21T18:05:45.114 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-21T18:05:45.115 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-10 2024-09-21T18:05:45.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-10 2024-09-21T18:05:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:45.221 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:45.221 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-14 2024-09-21T18:05:45.274 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.6': '/dev/vg_nvme/lv_2', 'osd.10': '/dev/vg_nvme/lv_3', 'osd.14': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:45.274 INFO:tasks.ceph:role: osd.14 2024-09-21T18:05:45.274 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:05:45.274 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T18:05:45.325 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:45.325 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:45.325 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:45.325 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:45.326 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:45.349 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-09-21T18:05:45.350 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-09-21T18:05:45.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-14 2024-09-21T18:05:45.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-14 2024-09-21T18:05:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:45.458 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:45.458 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:45.571+0000 7f19eb6c97c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T18:05:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:45.571+0000 7f19eb6c97c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T18:05:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:45.571+0000 7f19eb6c97c0 -1 bdev(0x55626a661800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:45.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:45.571+0000 7f19eb6c97c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T18:05:48.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T18:05:48.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:48.893 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:48.891+0000 7f31905687c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-21T18:05:48.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:48.891+0000 7f31905687c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-21T18:05:48.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:48.891+0000 7f31905687c0 -1 bdev(0x55c75f531800 /var/lib/ceph/osd/ceph-6/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:48.894 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:48.891+0000 7f31905687c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-21T18:05:52.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T18:05:52.092 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 10 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:52.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:52.171+0000 7fe226d0d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-10/keyring: can't open /var/lib/ceph/osd/ceph-10/keyring: (2) No such file or directory 2024-09-21T18:05:52.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:52.171+0000 7fe226d0d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-10/keyring 2024-09-21T18:05:52.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:52.171+0000 7fe226d0d7c0 -1 bdev(0x557c38497800 /var/lib/ceph/osd/ceph-10/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:52.175 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:52.171+0000 7fe226d0d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-10) _read_fsid unparsable uuid 2024-09-21T18:05:55.318 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-10 2024-09-21T18:05:55.414 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 14 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:05:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:55.491+0000 7fbdceb297c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-14/keyring: can't open /var/lib/ceph/osd/ceph-14/keyring: (2) No such file or directory 2024-09-21T18:05:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:55.495+0000 7fbdceb297c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-14/keyring 2024-09-21T18:05:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:55.495+0000 7fbdceb297c0 -1 bdev(0x55a53869b800 /var/lib/ceph/osd/ceph-14/block) open stat got: (1) Operation not permitted 2024-09-21T18:05:55.496 INFO:teuthology.orchestra.run.smithi070.stderr:2024-09-21T18:05:55.495+0000 7fbdceb297c0 -1 bluestore(/var/lib/ceph/osd/ceph-14) _read_fsid unparsable uuid 2024-09-21T18:05:58.654 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-14 2024-09-21T18:05:58.708 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T18:05:58.722 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:58.722 INFO:tasks.ceph:role: osd.3 2024-09-21T18:05:58.722 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:05:58.722 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:58.785 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:05:59.268 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2024-09-21T18:05:59.269 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2024-09-21T18:05:59.269 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-21T18:05:59.433 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T18:05:59.444 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:05:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:05:59.445 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-21T18:05:59.500 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-21T18:05:59.500 INFO:tasks.ceph:role: osd.7 2024-09-21T18:05:59.500 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:05:59.500 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T18:05:59.564 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:05:59.564 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:05:59.564 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:05:59.565 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:06:00.018 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2024-09-21T18:06:00.019 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2024-09-21T18:06:00.019 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-7 2024-09-21T18:06:00.114 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-21T18:06:00.123 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:06:00.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:06:00.124 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/ceph-11 2024-09-21T18:06:00.180 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-21T18:06:00.180 INFO:tasks.ceph:role: osd.11 2024-09-21T18:06:00.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:06:00.180 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:06:00.245 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:06:00.703 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2024-09-21T18:06:00.705 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2024-09-21T18:06:00.706 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11 2024-09-21T18:06:00.838 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-11 2024-09-21T18:06:00.849 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:06:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:06:00.850 DEBUG:teuthology.orchestra.run.smithi160:> sudo mkdir -p /var/lib/ceph/osd/ceph-15 2024-09-21T18:06:00.904 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.7': '/dev/vg_nvme/lv_2', 'osd.11': '/dev/vg_nvme/lv_3', 'osd.15': '/dev/vg_nvme/lv_4'} 2024-09-21T18:06:00.904 INFO:tasks.ceph:role: osd.15 2024-09-21T18:06:00.904 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:06:00.904 DEBUG:teuthology.orchestra.run.smithi160:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: = sunit=0 swidth=0 blks 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T18:06:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T18:06:00.969 INFO:teuthology.orchestra.run.smithi160.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T18:06:01.461 INFO:teuthology.orchestra.run.smithi160.stdout:Discarding blocks...Done. 2024-09-21T18:06:01.463 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi160.front.sepia.ceph.com -o noatime 2024-09-21T18:06:01.463 DEBUG:teuthology.orchestra.run.smithi160:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-15 2024-09-21T18:06:01.599 DEBUG:teuthology.orchestra.run.smithi160:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-15 2024-09-21T18:06:01.609 INFO:teuthology.orchestra.run.smithi160.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T18:06:01.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:06:01.610 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:01.718 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:01.713+0000 7f71d4b5e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T18:06:01.720 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:01.717+0000 7f71d4b5e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T18:06:01.720 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:01.717+0000 7f71d4b5e7c0 -1 bdev(0x5617bf397800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-21T18:06:01.721 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:01.717+0000 7f71d4b5e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T18:06:04.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T18:06:04.909 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:04.990 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:04.985+0000 7fb8caedf7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-21T18:06:04.991 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:04.985+0000 7fb8caedf7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-21T18:06:04.991 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:04.985+0000 7fb8caedf7c0 -1 bdev(0x55d2c8c13800 /var/lib/ceph/osd/ceph-7/block) open stat got: (1) Operation not permitted 2024-09-21T18:06:04.991 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:04.985+0000 7fb8caedf7c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-21T18:06:08.112 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T18:06:08.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 11 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:08.255 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:08.249+0000 7fdc952c17c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-11/keyring: can't open /var/lib/ceph/osd/ceph-11/keyring: (2) No such file or directory 2024-09-21T18:06:08.256 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:08.249+0000 7fdc952c17c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-11/keyring 2024-09-21T18:06:08.256 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:08.249+0000 7fdc952c17c0 -1 bdev(0x55d988ccf800 /var/lib/ceph/osd/ceph-11/block) open stat got: (1) Operation not permitted 2024-09-21T18:06:08.256 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:08.253+0000 7fdc952c17c0 -1 bluestore(/var/lib/ceph/osd/ceph-11) _read_fsid unparsable uuid 2024-09-21T18:06:11.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-11 2024-09-21T18:06:11.476 DEBUG:teuthology.orchestra.run.smithi160:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 15 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:11.558 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:11.553+0000 7f95cdf167c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-15/keyring: can't open /var/lib/ceph/osd/ceph-15/keyring: (2) No such file or directory 2024-09-21T18:06:11.559 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:11.553+0000 7f95cdf167c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-15/keyring 2024-09-21T18:06:11.559 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:11.553+0000 7f95cdf167c0 -1 bdev(0x55e9577b9800 /var/lib/ceph/osd/ceph-15/block) open stat got: (1) Operation not permitted 2024-09-21T18:06:11.559 INFO:teuthology.orchestra.run.smithi160.stderr:2024-09-21T18:06:11.553+0000 7f95cdf167c0 -1 bluestore(/var/lib/ceph/osd/ceph-15) _read_fsid unparsable uuid 2024-09-21T18:06:14.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-15 2024-09-21T18:06:14.777 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T18:06:14.777 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:14.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-21T18:06:14.791 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:14.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T18:06:14.846 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:14.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-21T18:06:14.902 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:14.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-8/keyring of=/dev/stdout 2024-09-21T18:06:14.958 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:14.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-12/keyring of=/dev/stdout 2024-09-21T18:06:15.013 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:15.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T18:06:15.028 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:15.028 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-21T18:06:15.082 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:15.082 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-9/keyring of=/dev/stdout 2024-09-21T18:06:15.138 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:15.139 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-13/keyring of=/dev/stdout 2024-09-21T18:06:15.194 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:15.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T18:06:15.208 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:15.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-21T18:06:15.263 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:15.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-10/keyring of=/dev/stdout 2024-09-21T18:06:15.318 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:15.319 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-14/keyring of=/dev/stdout 2024-09-21T18:06:15.371 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.371 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T18:06:15.385 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.385 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T18:06:15.439 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-21T18:06:15.491 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.491 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-11/keyring of=/dev/stdout 2024-09-21T18:06:15.544 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.545 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-15/keyring of=/dev/stdout 2024-09-21T18:06:15.599 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:15.599 DEBUG:teuthology.orchestra.run.smithi160:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T18:06:15.644 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T18:06:15.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T18:06:15.648 DEBUG:teuthology.orchestra.run.smithi040:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T18:06:15.650 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T18:06:15.657 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDaCu9mcn3yBxAAbRVSr4xvuBSXhTXN7cjB9g== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDbCu9mdmUVFRAAkpQ7dBSND6W+FvFBzvJJ4Q== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDeCu9mjrvtIhAA7ZnQDqe2vwGQHLzzgRymkQ== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.8] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDhCu9mlAmcNRAAp8T38xCvxo5P/ZTzVyjNJA== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.12] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDlCu9m8dJ+CxAAYpYAmuUv5f9wbBBBAPdViQ== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDrCu9mKrstGhAAYCg9ZsZGzke+KgKz2+Qrpw== 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2024-09-21T18:06:15.658 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDuCu9m2tp3KBAAvWTP1BTaUOteopPUAucSfg== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.9] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDxCu9mLMIiOhAAh0iTjSiF2zNnrN4oxMdutw== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.13] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQD1Cu9mo121DhAAPi05AFYg1lomoppdXpM4nA== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQD5Cu9moh8zIhAABGbdJaltUBR84fB2/j2FlQ== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQD8Cu9mbfRaNRAAG9bZSOFRw/BrXXNQla1iWA== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.10] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAAC+9mwgCHChAARcFBtu8REmXPz4KSc0HTxA== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.14] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQADC+9m9OyoHRAACioDj8JDSJqEnHdAjyNxmQ== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDaCu9mk0CsChAAtxXNxz2RP+iqlm4ebwZaPQ== 2024-09-21T18:06:15.659 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAJC+9mb+LeKhAAjhYhqxUFA4HApl0NdFDiJQ== 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAMC+9miY8TOxAAdNaqCy87O3k/taswNYEdvA== 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.11] 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAQC+9mrZxCDxAAM74Sox/dxkIUY0WiB31W5A== 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.15] 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQATC+9mQvtPIRAAe2ojDiRVRzilJJZR+UTwgQ== 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQDaCu9ml5SUDxAA15HUxetmDeoslOkc1nV91A== 2024-09-21T18:06:15.660 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.y] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDaCu9mcn3yBxAAbRVSr4xvuBSXhTXN7cjB9g== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.0] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDbCu9mdmUVFRAAkpQ7dBSND6W+FvFBzvJJ4Q== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.4] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDeCu9mjrvtIhAA7ZnQDqe2vwGQHLzzgRymkQ== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.8] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDhCu9mlAmcNRAAp8T38xCvxo5P/ZTzVyjNJA== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.12] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDlCu9m8dJ+CxAAYpYAmuUv5f9wbBBBAPdViQ== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.1] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDrCu9mKrstGhAAYCg9ZsZGzke+KgKz2+Qrpw== 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.5] 2024-09-21T18:06:15.661 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDuCu9m2tp3KBAAvWTP1BTaUOteopPUAucSfg== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.9] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDxCu9mLMIiOhAAh0iTjSiF2zNnrN4oxMdutw== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.13] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD1Cu9mo121DhAAPi05AFYg1lomoppdXpM4nA== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.2] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD5Cu9moh8zIhAABGbdJaltUBR84fB2/j2FlQ== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.6] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQD8Cu9mbfRaNRAAG9bZSOFRw/BrXXNQla1iWA== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.10] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAAC+9mwgCHChAARcFBtu8REmXPz4KSc0HTxA== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.14] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQADC+9m9OyoHRAACioDj8JDSJqEnHdAjyNxmQ== 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout:[mgr.x] 2024-09-21T18:06:15.662 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDaCu9mk0CsChAAtxXNxz2RP+iqlm4ebwZaPQ== 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.3] 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAJC+9mb+LeKhAAjhYhqxUFA4HApl0NdFDiJQ== 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.7] 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAMC+9miY8TOxAAdNaqCy87O3k/taswNYEdvA== 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.11] 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQAQC+9mrZxCDxAAM74Sox/dxkIUY0WiB31W5A== 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout:[osd.15] 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQATC+9mQvtPIRAAe2ojDiRVRzilJJZR+UTwgQ== 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout:[client.0] 2024-09-21T18:06:15.663 INFO:teuthology.orchestra.run.smithi040.stdout: key = AQDaCu9ml5SUDxAA15HUxetmDeoslOkc1nV91A== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.y] 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDaCu9mcn3yBxAAbRVSr4xvuBSXhTXN7cjB9g== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDbCu9mdmUVFRAAkpQ7dBSND6W+FvFBzvJJ4Q== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.4] 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDeCu9mjrvtIhAA7ZnQDqe2vwGQHLzzgRymkQ== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.8] 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDhCu9mlAmcNRAAp8T38xCvxo5P/ZTzVyjNJA== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.12] 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDlCu9m8dJ+CxAAYpYAmuUv5f9wbBBBAPdViQ== 2024-09-21T18:06:15.665 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDrCu9mKrstGhAAYCg9ZsZGzke+KgKz2+Qrpw== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.5] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDuCu9m2tp3KBAAvWTP1BTaUOteopPUAucSfg== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.9] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDxCu9mLMIiOhAAh0iTjSiF2zNnrN4oxMdutw== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.13] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQD1Cu9mo121DhAAPi05AFYg1lomoppdXpM4nA== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQD5Cu9moh8zIhAABGbdJaltUBR84fB2/j2FlQ== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.6] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQD8Cu9mbfRaNRAAG9bZSOFRw/BrXXNQla1iWA== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.10] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAAC+9mwgCHChAARcFBtu8REmXPz4KSc0HTxA== 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.14] 2024-09-21T18:06:15.666 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQADC+9m9OyoHRAACioDj8JDSJqEnHdAjyNxmQ== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDaCu9mk0CsChAAtxXNxz2RP+iqlm4ebwZaPQ== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAJC+9mb+LeKhAAjhYhqxUFA4HApl0NdFDiJQ== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.7] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAMC+9miY8TOxAAdNaqCy87O3k/taswNYEdvA== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.11] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAQC+9mrZxCDxAAM74Sox/dxkIUY0WiB31W5A== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.15] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQATC+9mQvtPIRAAe2ojDiRVRzilJJZR+UTwgQ== 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-09-21T18:06:15.667 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQDaCu9ml5SUDxAA15HUxetmDeoslOkc1nV91A== 2024-09-21T18:06:15.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:15.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:15.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:15.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.891 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.893 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:15.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.8 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.021 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.12 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.196 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.274 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.9 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.352 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.13 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.438 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.440 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.523 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.615 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.10 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.660 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.14 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:16.748 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:16.788 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T18:06:16.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.838 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.872 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.931 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:16.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.019 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.11 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.106 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.109 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.15 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T18:06:17.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T18:06:17.195 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T18:06:17.197 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T18:06:17.244 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T18:06:17.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T18:06:17.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T18:06:17.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T18:06:17.404 DEBUG:teuthology.orchestra.run.smithi040:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T18:06:17.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T18:06:17.519 DEBUG:teuthology.orchestra.run.smithi040:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T18:06:17.531 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T18:06:17.545 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T18:06:17.640 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T18:06:17.651 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:17.654 DEBUG:teuthology.orchestra.run.smithi040:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:17.656 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T18:06:17.699 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T18:06:17.699 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T18:06:17.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T18:06:17.706 INFO:tasks.ceph.mon.a:Started 2024-09-21T18:06:17.707 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T18:06:17.707 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T18:06:17.709 INFO:tasks.ceph.mon.b:Started 2024-09-21T18:06:17.709 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T18:06:17.709 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T18:06:17.744 INFO:tasks.ceph.mon.c:Started 2024-09-21T18:06:17.745 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T18:06:17.745 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-21T18:06:17.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-21T18:06:17.747 INFO:tasks.ceph.mgr.y:Started 2024-09-21T18:06:17.748 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T18:06:17.748 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T18:06:17.750 INFO:tasks.ceph.mgr.x:Started 2024-09-21T18:06:17.750 DEBUG:tasks.ceph:set 0 configs 2024-09-21T18:06:17.750 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T18:06:17.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T18:06:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-09-21T18:06:28.269 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T18:06:28.269 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T18:06:28.269 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:28.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T18:06:28.281 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:28.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-21T18:06:28.341 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:28.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-8/fsid of=/dev/stdout 2024-09-21T18:06:28.398 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-09-21T18:06:28.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-12/fsid of=/dev/stdout 2024-09-21T18:06:28.458 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:28.458 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T18:06:28.474 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:28.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-21T18:06:28.531 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:28.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-9/fsid of=/dev/stdout 2024-09-21T18:06:28.589 DEBUG:teuthology.orchestra.run.smithi040:> set -ex 2024-09-21T18:06:28.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo dd if=/var/lib/ceph/osd/ceph-13/fsid of=/dev/stdout 2024-09-21T18:06:28.643 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:28.643 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T18:06:28.658 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:28.658 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-21T18:06:28.712 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:28.712 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-10/fsid of=/dev/stdout 2024-09-21T18:06:28.763 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-09-21T18:06:28.763 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-14/fsid of=/dev/stdout 2024-09-21T18:06:28.816 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:28.816 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T18:06:28.829 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:28.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-21T18:06:28.885 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:28.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-11/fsid of=/dev/stdout 2024-09-21T18:06:28.939 DEBUG:teuthology.orchestra.run.smithi160:> set -ex 2024-09-21T18:06:28.939 DEBUG:teuthology.orchestra.run.smithi160:> sudo dd if=/var/lib/ceph/osd/ceph-15/fsid of=/dev/stdout 2024-09-21T18:06:28.992 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7 0 2024-09-21T18:06:29.236 INFO:teuthology.orchestra.run.smithi160.stdout:0 2024-09-21T18:06:29.251 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 020df4b2-13cb-4e14-88a1-a8219b5b34b0 1 2024-09-21T18:06:29.447 INFO:teuthology.orchestra.run.smithi160.stdout:1 2024-09-21T18:06:29.464 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 57a83527-6610-4eb4-a01e-e22c3ade98a5 2 2024-09-21T18:06:29.652 INFO:teuthology.orchestra.run.smithi160.stdout:2 2024-09-21T18:06:29.668 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 42758eff-818c-4e2b-98c8-5f9c01c883c9 3 2024-09-21T18:06:29.862 INFO:teuthology.orchestra.run.smithi160.stdout:3 2024-09-21T18:06:29.878 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new c358d178-864a-4a0a-a1a8-0c42d1b10d86 4 2024-09-21T18:06:30.073 INFO:teuthology.orchestra.run.smithi160.stdout:4 2024-09-21T18:06:30.089 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 73790513-6074-4d9b-80b2-0403d3c1f0c6 5 2024-09-21T18:06:30.284 INFO:teuthology.orchestra.run.smithi160.stdout:5 2024-09-21T18:06:30.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new e588347d-7853-4ac0-a871-03a7311247e4 6 2024-09-21T18:06:30.483 INFO:teuthology.orchestra.run.smithi160.stdout:6 2024-09-21T18:06:30.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new f47cab83-6e0b-478f-88e0-0cf069bb3f3c 7 2024-09-21T18:06:30.792 INFO:teuthology.orchestra.run.smithi160.stdout:7 2024-09-21T18:06:30.808 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 427789c1-98e4-43d1-a12c-2606a8a6cd4b 8 2024-09-21T18:06:31.010 INFO:teuthology.orchestra.run.smithi160.stdout:8 2024-09-21T18:06:31.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 268cb87f-9f31-4543-8fc5-4e05b6d9ed0d 9 2024-09-21T18:06:31.069 INFO:tasks.ceph.mgr.x.smithi160.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-21T18:06:31.069 INFO:tasks.ceph.mgr.x.smithi160.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-21T18:06:31.069 INFO:tasks.ceph.mgr.x.smithi160.stderr: from numpy import show_config as show_numpy_config 2024-09-21T18:06:31.227 INFO:teuthology.orchestra.run.smithi160.stdout:9 2024-09-21T18:06:31.243 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 85e66756-8ef8-4ba9-911f-08d74f04e0a1 10 2024-09-21T18:06:31.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-21T18:06:31.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-21T18:06:31.373 INFO:tasks.ceph.mgr.y.smithi029.stderr: from numpy import show_config as show_numpy_config 2024-09-21T18:06:31.440 INFO:teuthology.orchestra.run.smithi160.stdout:10 2024-09-21T18:06:31.455 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 952e9412-fdb5-4b68-b109-def00c946b31 11 2024-09-21T18:06:31.654 INFO:teuthology.orchestra.run.smithi160.stdout:11 2024-09-21T18:06:31.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 1b4bd0f6-ed9f-426e-bb82-4e4328199d79 12 2024-09-21T18:06:31.867 INFO:teuthology.orchestra.run.smithi160.stdout:12 2024-09-21T18:06:31.883 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new a535d39d-7bee-4185-a28d-4155f9b17207 13 2024-09-21T18:06:32.075 INFO:teuthology.orchestra.run.smithi160.stdout:13 2024-09-21T18:06:32.090 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 95b8504b-67c0-4ed9-be7e-81eb3dacacf6 14 2024-09-21T18:06:32.282 INFO:teuthology.orchestra.run.smithi160.stdout:14 2024-09-21T18:06:32.298 DEBUG:teuthology.orchestra.run.smithi160:> sudo ceph --cluster ceph osd new 84b8cc82-bc18-4cce-901e-ed48e85ba7e3 15 2024-09-21T18:06:32.505 INFO:teuthology.orchestra.run.smithi160.stdout:15 2024-09-21T18:06:32.521 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T18:06:32.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T18:06:32.523 INFO:tasks.ceph.osd.0:Started 2024-09-21T18:06:32.523 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-21T18:06:32.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-21T18:06:32.526 INFO:tasks.ceph.osd.4:Started 2024-09-21T18:06:32.526 INFO:tasks.ceph.osd.8:Restarting daemon 2024-09-21T18:06:32.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 8 2024-09-21T18:06:32.528 INFO:tasks.ceph.osd.8:Started 2024-09-21T18:06:32.528 INFO:tasks.ceph.osd.12:Restarting daemon 2024-09-21T18:06:32.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 12 2024-09-21T18:06:32.530 INFO:tasks.ceph.osd.12:Started 2024-09-21T18:06:32.531 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T18:06:32.531 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T18:06:32.533 INFO:tasks.ceph.osd.1:Started 2024-09-21T18:06:32.533 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T18:06:32.534 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T18:06:32.536 INFO:tasks.ceph.osd.5:Started 2024-09-21T18:06:32.536 INFO:tasks.ceph.osd.9:Restarting daemon 2024-09-21T18:06:32.536 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 9 2024-09-21T18:06:32.538 INFO:tasks.ceph.osd.9:Started 2024-09-21T18:06:32.538 INFO:tasks.ceph.osd.13:Restarting daemon 2024-09-21T18:06:32.539 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 13 2024-09-21T18:06:32.541 INFO:tasks.ceph.osd.13:Started 2024-09-21T18:06:32.541 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T18:06:32.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T18:06:32.544 INFO:tasks.ceph.osd.2:Started 2024-09-21T18:06:32.544 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-21T18:06:32.544 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-21T18:06:32.546 INFO:tasks.ceph.osd.6:Started 2024-09-21T18:06:32.546 INFO:tasks.ceph.osd.10:Restarting daemon 2024-09-21T18:06:32.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 10 2024-09-21T18:06:32.549 INFO:tasks.ceph.osd.10:Started 2024-09-21T18:06:32.549 INFO:tasks.ceph.osd.14:Restarting daemon 2024-09-21T18:06:32.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 14 2024-09-21T18:06:32.551 INFO:tasks.ceph.osd.14:Started 2024-09-21T18:06:32.552 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T18:06:32.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T18:06:32.553 INFO:tasks.ceph.osd.3:Started 2024-09-21T18:06:32.554 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-21T18:06:32.554 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-21T18:06:32.556 INFO:tasks.ceph.osd.7:Started 2024-09-21T18:06:32.556 INFO:tasks.ceph.osd.11:Restarting daemon 2024-09-21T18:06:32.556 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 11 2024-09-21T18:06:32.558 INFO:tasks.ceph.osd.11:Started 2024-09-21T18:06:32.558 INFO:tasks.ceph.osd.15:Restarting daemon 2024-09-21T18:06:32.558 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 15 2024-09-21T18:06:32.560 INFO:tasks.ceph.osd.15:Started 2024-09-21T18:06:32.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:06:32.775 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:32.775 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:06:32.496673+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:06:32.791 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T18:06:32.791 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T18:06:33.671 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:06:33.666+0000 7ff41bfba7c0 -1 Falling back to public interface 2024-09-21T18:06:33.711 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:33.709+0000 7f3cf4ca17c0 -1 Falling back to public interface 2024-09-21T18:06:33.871 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:06:33.868+0000 7f66fea397c0 -1 Falling back to public interface 2024-09-21T18:06:33.903 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:06:33.898+0000 7fb9335d37c0 -1 Falling back to public interface 2024-09-21T18:06:33.904 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:06:33.898+0000 7f49846407c0 -1 Falling back to public interface 2024-09-21T18:06:33.922 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:33.921+0000 7fca60f637c0 -1 Falling back to public interface 2024-09-21T18:06:33.934 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:06:33.930+0000 7f095b7fb7c0 -1 Falling back to public interface 2024-09-21T18:06:33.948 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:33.942+0000 7f4419e307c0 -1 Falling back to public interface 2024-09-21T18:06:33.949 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:33.942+0000 7fd7acb227c0 -1 Falling back to public interface 2024-09-21T18:06:33.954 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:33.952+0000 7f94ebb677c0 -1 Falling back to public interface 2024-09-21T18:06:33.958 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:33.956+0000 7fb27122f7c0 -1 Falling back to public interface 2024-09-21T18:06:33.981 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:33.980+0000 7f6a4b2c67c0 -1 Falling back to public interface 2024-09-21T18:06:34.052 INFO:tasks.ceph.mgr.x.smithi160.stderr:2024-09-21T18:06:34.048+0000 7ff897908640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T18:06:34.053 INFO:tasks.ceph.mgr.x.smithi160.stderr:2024-09-21T18:06:34.052+0000 7ff897908640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T18:06:34.114 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:34.113+0000 7f9bc7eb47c0 -1 Falling back to public interface 2024-09-21T18:06:34.145 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:34.138+0000 7f0b7882f7c0 -1 Falling back to public interface 2024-09-21T18:06:34.155 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:06:34.150+0000 7f8c486e77c0 -1 Falling back to public interface 2024-09-21T18:06:34.158 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:06:34.157+0000 7f7291b247c0 -1 Falling back to public interface 2024-09-21T18:06:34.404 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T18:06:34.717 INFO:teuthology.misc.health.smithi029.stdout: 2024-09-21T18:06:34.717 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":18,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:06:32.496673+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:06:34.734 DEBUG:teuthology.misc:0 of 16 OSDs are up 2024-09-21T18:06:37.731 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:37.728+0000 7f3cf4ca17c0 -1 osd.6 0 log_to_monitors true 2024-09-21T18:06:37.739 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:06:37.738+0000 7ff41bfba7c0 -1 osd.4 0 log_to_monitors true 2024-09-21T18:06:37.837 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:06:37.836+0000 7f66fea397c0 -1 osd.7 0 log_to_monitors true 2024-09-21T18:06:37.918 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:06:37.918+0000 7f095b7fb7c0 -1 osd.0 0 log_to_monitors true 2024-09-21T18:06:37.923 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:06:37.922+0000 7fb9335d37c0 -1 osd.8 0 log_to_monitors true 2024-09-21T18:06:37.939 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:06:37.934+0000 7f49846407c0 -1 osd.5 0 log_to_monitors true 2024-09-21T18:06:37.974 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:37.972+0000 7fb27122f7c0 -1 osd.11 0 log_to_monitors true 2024-09-21T18:06:37.988 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:37.984+0000 7fca60f637c0 -1 osd.2 0 log_to_monitors true 2024-09-21T18:06:38.192 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:06:38.188+0000 7f7291b247c0 -1 osd.14 0 log_to_monitors true 2024-09-21T18:06:38.271 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:38.266+0000 7fd7acb227c0 -1 osd.1 0 log_to_monitors true 2024-09-21T18:06:38.295 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:38.292+0000 7f94ebb677c0 -1 osd.15 0 log_to_monitors true 2024-09-21T18:06:38.402 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:38.394+0000 7f0b7882f7c0 -1 osd.9 0 log_to_monitors true 2024-09-21T18:06:38.413 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:06:38.410+0000 7f8c486e77c0 -1 osd.12 0 log_to_monitors true 2024-09-21T18:06:38.530 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:38.522+0000 7f4419e307c0 -1 osd.13 0 log_to_monitors true 2024-09-21T18:06:38.573 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:38.568+0000 7f6a4b2c67c0 -1 osd.3 0 log_to_monitors true 2024-09-21T18:06:38.577 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:38.576+0000 7f9bc7eb47c0 -1 osd.10 0 log_to_monitors true 2024-09-21T18:06:39.470 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:39.468+0000 7f3cf0c4a640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T18:06:39.470 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:39.468+0000 7f3ce4680640 -1 osd.6 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:39.474 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:39.472+0000 7f3ce8a7c640 -1 osd.6 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.493 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:06:39.490+0000 7ff417f63640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T18:06:39.495 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:06:39.494+0000 7ff40b999640 -1 osd.4 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:39.500 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:06:39.498+0000 7ff40fd95640 -1 osd.4 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.816 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:39.810+0000 7f0b747d8640 -1 osd.9 0 waiting for initial osdmap 2024-09-21T18:06:39.826 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:06:39.822+0000 7f09577a4640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T18:06:39.826 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:06:39.824+0000 7f66fa9e2640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T18:06:39.826 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:06:39.826+0000 7fb92f57c640 -1 osd.8 0 waiting for initial osdmap 2024-09-21T18:06:39.826 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:39.824+0000 7f6a4726f640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T18:06:39.827 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:06:39.826+0000 7f8c44690640 -1 osd.12 0 waiting for initial osdmap 2024-09-21T18:06:39.827 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:06:39.824+0000 7f728dacd640 -1 osd.14 0 waiting for initial osdmap 2024-09-21T18:06:39.827 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:06:39.818+0000 7f49805e9640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T18:06:39.828 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:39.824+0000 7f6a3ecad640 -1 osd.3 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:39.828 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:39.818+0000 7f0b6c216640 -1 osd.9 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:39.835 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:39.832+0000 7f6a430a9640 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.835 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:06:39.832+0000 7f66f681c640 -1 osd.7 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.837 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:06:39.834+0000 7f09535de640 -1 osd.0 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.837 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:06:39.836+0000 7f7289907640 -1 osd.14 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.839 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:06:39.838+0000 7f8c404ca640 -1 osd.12 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.840 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:39.834+0000 7f0b70612640 -1 osd.9 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.841 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:06:39.842+0000 7fb92b3b6640 -1 osd.8 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:39.847 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:06:39.842+0000 7f497c423640 -1 osd.5 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.553 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:40.552+0000 7f9bc3e5d640 -1 osd.10 0 waiting for initial osdmap 2024-09-21T18:06:40.554 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:40.552+0000 7f9bb7893640 -1 osd.10 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.558 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:40.556+0000 7fca5cf0c640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T18:06:40.560 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:40.556+0000 7fca50942640 -1 osd.2 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.645 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:40.644+0000 7fb26d9ea640 -1 osd.11 0 waiting for initial osdmap 2024-09-21T18:06:40.647 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:40.644+0000 7fb260c0e640 -1 osd.11 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.659 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:40.656+0000 7f9bbbc8f640 -1 osd.10 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.665 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:40.660+0000 7fca54d3e640 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.699 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:40.694+0000 7f44165eb640 -1 osd.13 0 waiting for initial osdmap 2024-09-21T18:06:40.700 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:40.694+0000 7f440980f640 -1 osd.13 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.711 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:40.708+0000 7f94e7b10640 -1 osd.15 0 waiting for initial osdmap 2024-09-21T18:06:40.712 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:40.708+0000 7f94db546640 -1 osd.15 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.714 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:40.710+0000 7fd7a92dd640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T18:06:40.717 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:40.710+0000 7fd79c501640 -1 osd.1 0 handle_osd_map injecting map crc failure 2024-09-21T18:06:40.753 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:40.748+0000 7fb26500a640 -1 osd.11 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.811 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:40.806+0000 7f440dc0b640 -1 osd.13 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.823 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:40.820+0000 7f94df942640 -1 osd.15 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:40.827 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:40.822+0000 7fd7a08fd640 -1 osd.1 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:06:41.691 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:41.688+0000 7f94db546640 -1 osd.15 21 handle_osd_map injecting map crc failure 2024-09-21T18:06:41.727 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:41.722+0000 7f440980f640 -1 osd.13 21 handle_osd_map injecting map crc failure 2024-09-21T18:06:42.342 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T18:06:42.680 INFO:teuthology.misc.health.smithi029.stdout: 2024-09-21T18:06:42.680 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":23,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:06:41.831877+0000","last_up_change":"2024-09-21T18:06:41.831877+0000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":501927900}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":501927900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":501927900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":501927900}]},"public_addr":"172.21.15.160:6801/501927900","cluster_addr":"172.21.15.160:6802/501927900","heartbeat_back_addr":"172.21.15.160:6804/501927900","heartbeat_front_addr":"172.21.15.160:6803/501927900","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.888021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.971774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.573107+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.734463+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.956194+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.774747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.809566+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.936207+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.419469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.578534+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.001270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.397216+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.219010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:06:42.698 DEBUG:teuthology.misc:16 of 16 OSDs are up 2024-09-21T18:06:42.699 INFO:tasks.ceph:Creating RBD pool 2024-09-21T18:06:42.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T18:06:43.851 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'rbd' created 2024-09-21T18:06:43.880 DEBUG:teuthology.orchestra.run.smithi029:> rbd --cluster ceph pool init rbd 2024-09-21T18:06:44.323 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:44.318+0000 7f0b6a212640 -1 osd.9 23 handle_osd_map injecting map crc failure 2024-09-21T18:06:44.327 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:44.324+0000 7f6a3cca9640 -1 osd.3 23 handle_osd_map injecting map crc failure 2024-09-21T18:06:44.859 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:44.855+0000 7fb260c0e640 -1 osd.11 25 handle_osd_map injecting map crc failure 2024-09-21T18:06:44.860 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:06:44.854+0000 7f0b6c216640 -1 osd.9 25 handle_osd_map injecting map crc failure 2024-09-21T18:06:44.860 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:06:44.856+0000 7f3ce4680640 -1 osd.6 25 handle_osd_map injecting map crc failure 2024-09-21T18:06:45.889 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:45.888+0000 7fca50942640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-21T18:06:46.875 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:46.872+0000 7fca50942640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-09-21T18:06:46.898 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:06:46.896+0000 7f7283507640 -1 osd.14 26 handle_osd_map injecting map crc failure 2024-09-21T18:06:47.114 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T18:06:47.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-21T18:06:47.115 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T18:06:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:47.462 INFO:teuthology.orchestra.run.smithi029.stdout:[{"version":9,"timestamp":"2024-09-21T18:06:40.719838+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"5717.925353"}]}] 2024-09-21T18:06:47.462 INFO:tasks.ceph_manager:config epoch is 9 2024-09-21T18:06:47.465 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T18:06:47.465 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T18:06:47.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T18:06:47.822 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:47.838 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6800","nonce":3307717321}]},"active_addr":"172.21.15.160:6800/3307717321","active_change":"2024-09-21T18:06:33.030917+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540719139924082687,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.8","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":1992863820}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":3972287121}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":2947132670}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.160:0","nonce":4061789456}]}]} 2024-09-21T18:06:47.844 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T18:06:47.844 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T18:06:47.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:06:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:48.163 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:06:46.871691+0000","last_up_change":"2024-09-21T18:06:41.831877+0000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":501927900}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":501927900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":501927900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":501927900}]},"public_addr":"172.21.15.160:6801/501927900","cluster_addr":"172.21.15.160:6802/501927900","heartbeat_back_addr":"172.21.15.160:6804/501927900","heartbeat_front_addr":"172.21.15.160:6803/501927900","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.888021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.308804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.971774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.573107+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.734463+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.956194+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.774747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.809566+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.936207+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.419469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.578534+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.001270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.397216+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.571531+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.219010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.336388+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:06:48.183 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T18:06:48.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:06:48.485 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:48.485 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":28,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:06:46.871691+0000","last_up_change":"2024-09-21T18:06:41.831877+0000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":16,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":501927900}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":501927900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":501927900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":501927900}]},"public_addr":"172.21.15.160:6801/501927900","cluster_addr":"172.21.15.160:6802/501927900","heartbeat_back_addr":"172.21.15.160:6804/501927900","heartbeat_front_addr":"172.21.15.160:6803/501927900","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.888021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.308804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.971774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.573107+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.734463+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.956194+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.774747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.809566+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.936207+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.419469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.578534+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.001270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.397216+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.571531+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.219010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.336388+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:06:48.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T18:06:48.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T18:06:48.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T18:06:48.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T18:06:48.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T18:06:48.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T18:06:48.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T18:06:48.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T18:06:48.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-21T18:06:48.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-21T18:06:48.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-21T18:06:48.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-21T18:06:48.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-21T18:06:48.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-21T18:06:48.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-21T18:06:48.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-21T18:06:48.891 INFO:teuthology.orchestra.run.smithi029.stdout:90194313220 2024-09-21T18:06:48.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T18:06:48.950 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:48.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-21T18:06:48.954 INFO:teuthology.orchestra.run.smithi029.stdout:98784247813 2024-09-21T18:06:48.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-21T18:06:48.972 INFO:teuthology.orchestra.run.smithi029.stdout:98784247812 2024-09-21T18:06:48.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T18:06:48.983 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:48.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-21T18:06:48.990 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:48.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-21T18:06:48.992 INFO:teuthology.orchestra.run.smithi029.stdout:90194313221 2024-09-21T18:06:48.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T18:06:49.019 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:49.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T18:06:49.027 INFO:teuthology.orchestra.run.smithi029.stdout:98784247811 2024-09-21T18:06:49.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-21T18:06:49.034 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-21T18:06:49.067 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-21T18:06:49.071 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T18:06:49.150 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:49.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T18:06:49.151 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:49.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T18:06:49.152 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:49.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-21T18:06:49.160 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T18:06:49.737 INFO:teuthology.orchestra.run.smithi029.stdout:98784247811 2024-09-21T18:06:49.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247811 for osd.15 2024-09-21T18:06:49.861 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.874 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:49.898 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.14 2024-09-21T18:06:49.948 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.5 2024-09-21T18:06:50.014 INFO:teuthology.orchestra.run.smithi029.stdout:98784247809 2024-09-21T18:06:50.020 INFO:teuthology.orchestra.run.smithi029.stdout:90194313219 2024-09-21T18:06:50.069 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247809 for osd.13 2024-09-21T18:06:50.071 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313219 for osd.4 2024-09-21T18:06:50.098 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:50.109 INFO:teuthology.orchestra.run.smithi029.stdout:94489280514 2024-09-21T18:06:50.143 INFO:teuthology.orchestra.run.smithi029.stdout:90194313221 2024-09-21T18:06:50.146 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.153 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.12 2024-09-21T18:06:50.153 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.158 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.3 2024-09-21T18:06:50.181 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.9 2024-09-21T18:06:50.182 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.182 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313221 got 90194313221 for osd.6 2024-09-21T18:06:50.185 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.189 INFO:teuthology.orchestra.run.smithi029.stdout:98784247812 2024-09-21T18:06:50.198 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.204 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:50.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.2 2024-09-21T18:06:50.211 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247812 got 98784247812 for osd.1 2024-09-21T18:06:50.220 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.228 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.230 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.10 2024-09-21T18:06:50.230 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.234 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.8 2024-09-21T18:06:50.234 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.235 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:50.251 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.7 2024-09-21T18:06:50.256 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.11 2024-09-21T18:06:50.261 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:50.278 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.0 2024-09-21T18:06:50.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-21T18:06:50.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-21T18:06:50.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T18:06:51.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-21T18:06:51.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T18:06:51.129 INFO:teuthology.orchestra.run.smithi029.stdout:98784247813 2024-09-21T18:06:51.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247813 got 98784247813 for osd.15 2024-09-21T18:06:51.147 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:51.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T18:06:51.214 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:51.236 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.14 2024-09-21T18:06:51.236 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:51.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T18:06:51.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-21T18:06:51.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T18:06:51.370 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:51.377 INFO:teuthology.orchestra.run.smithi029.stdout:98784247811 2024-09-21T18:06:51.398 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247811 got 98784247811 for osd.13 2024-09-21T18:06:51.398 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:51.404 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.5 2024-09-21T18:06:51.404 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:51.530 INFO:teuthology.orchestra.run.smithi029.stdout:90194313220 2024-09-21T18:06:51.555 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313220 got 90194313220 for osd.4 2024-09-21T18:06:51.555 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:51.575 INFO:teuthology.orchestra.run.smithi029.stdout:94489280514 2024-09-21T18:06:51.592 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280514 for osd.3 2024-09-21T18:06:51.653 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:51.736 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.11 2024-09-21T18:06:51.737 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:51.737 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.0 2024-09-21T18:06:51.738 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:51.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280515 for osd.7 2024-09-21T18:06:52.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T18:06:52.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-21T18:06:52.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T18:06:52.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T18:06:52.798 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:06:52.794+0000 7f440980f640 -1 osd.13 28 handle_osd_map injecting map crc failure 2024-09-21T18:06:52.916 INFO:teuthology.orchestra.run.smithi029.stdout:94489280515 2024-09-21T18:06:52.936 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280515 got 94489280515 for osd.3 2024-09-21T18:06:52.937 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:53.030 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:53.027+0000 7f6a3cca9640 -1 osd.3 28 handle_osd_map injecting map crc failure 2024-09-21T18:06:53.116 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:53.119 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:53.123 INFO:teuthology.orchestra.run.smithi029.stdout:94489280516 2024-09-21T18:06:53.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.7 2024-09-21T18:06:53.139 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:53.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.11 2024-09-21T18:06:53.141 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:53.144 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280516 got 94489280516 for osd.0 2024-09-21T18:06:53.144 DEBUG:teuthology.parallel:result is None 2024-09-21T18:06:53.144 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T18:06:53.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:06:53.194 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:53.191+0000 7f94d9542640 -1 osd.15 28 handle_osd_map injecting map crc failure 2024-09-21T18:06:53.474 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:53.474 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:06:53.491 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T18:06:53.050460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":133,"num_write_kb":2412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":453044,"kb_used_data":3916,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496396,"statfs":{"total":1546188226560,"available":1545724309504,"internally_reserved":0,"allocated":4009984,"data_stored":2494672,"data_compressed":58700,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":29,"num_write":69,"num_write_kb":1567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.176070"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.124693+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:06:47.124693+0000","last_peered":"2024-09-21T18:06:47.124693+0000","last_clean":"2024-09-21T18:06:47.124693+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:06:47.124693+0000","last_undegraded":"2024-09-21T18:06:47.124693+0000","last_fullsized":"2024-09-21T18:06:47.124693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.324681+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:06:47.324681+0000","last_peered":"2024-09-21T18:06:47.324681+0000","last_clean":"2024-09-21T18:06:47.324681+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:06:47.324681+0000","last_undegraded":"2024-09-21T18:06:47.324681+0000","last_fullsized":"2024-09-21T18:06:47.324681+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.980470+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:06:46.980470+0000","last_peered":"2024-09-21T18:06:46.980470+0000","last_clean":"2024-09-21T18:06:46.980470+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:06:46.980470+0000","last_undegraded":"2024-09-21T18:06:46.980470+0000","last_fullsized":"2024-09-21T18:06:46.980470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.878313+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:06:46.878313+0000","last_peered":"2024-09-21T18:06:46.878313+0000","last_clean":"2024-09-21T18:06:46.878313+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:06:46.878313+0000","last_undegraded":"2024-09-21T18:06:46.878313+0000","last_fullsized":"2024-09-21T18:06:46.878313+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.982554+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:06:46.982554+0000","last_peered":"2024-09-21T18:06:46.982554+0000","last_clean":"2024-09-21T18:06:46.982554+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:06:46.982554+0000","last_undegraded":"2024-09-21T18:06:46.982554+0000","last_fullsized":"2024-09-21T18:06:46.982554+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.988258+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:06:47.988258+0000","last_peered":"2024-09-21T18:06:47.988258+0000","last_clean":"2024-09-21T18:06:47.988258+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:06:47.988258+0000","last_undegraded":"2024-09-21T18:06:47.988258+0000","last_fullsized":"2024-09-21T18:06:47.988258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.889135+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:06:46.889135+0000","last_peered":"2024-09-21T18:06:46.889135+0000","last_clean":"2024-09-21T18:06:46.889135+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:06:46.889135+0000","last_undegraded":"2024-09-21T18:06:46.889135+0000","last_fullsized":"2024-09-21T18:06:46.889135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.096800+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:06:47.096800+0000","last_peered":"2024-09-21T18:06:47.096800+0000","last_clean":"2024-09-21T18:06:47.096800+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:06:47.096800+0000","last_undegraded":"2024-09-21T18:06:47.096800+0000","last_fullsized":"2024-09-21T18:06:47.096800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'100","reported_seq":139,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:51.413279+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:06:51.413279+0000","last_peered":"2024-09-21T18:06:51.413279+0000","last_clean":"2024-09-21T18:06:51.413279+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:06:51.413279+0000","last_undegraded":"2024-09-21T18:06:51.413279+0000","last_fullsized":"2024-09-21T18:06:51.413279+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":131,"num_write_kb":2410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":131,"num_write_kb":2410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":59574,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":212992,"data_stored":96458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":94628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":101522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":96869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":192512,"data_stored":92136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":568372,"data_compressed":29350,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":93328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31432,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":249856,"data_stored":96794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":564892,"data_compressed":29350,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":262144,"data_stored":100349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":262144,"data_stored":100912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":29787,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":29787,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:06:53.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:06:53.782 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:06:53.799 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T18:06:53.050460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":133,"num_write_kb":2412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":103,"ondisk_log_size":103,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":453044,"kb_used_data":3916,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496396,"statfs":{"total":1546188226560,"available":1545724309504,"internally_reserved":0,"allocated":4009984,"data_stored":2494672,"data_compressed":58700,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":11,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":29,"num_write":69,"num_write_kb":1567,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.176070"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.124693+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:06:47.124693+0000","last_peered":"2024-09-21T18:06:47.124693+0000","last_clean":"2024-09-21T18:06:47.124693+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:06:47.124693+0000","last_undegraded":"2024-09-21T18:06:47.124693+0000","last_fullsized":"2024-09-21T18:06:47.124693+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.324681+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:06:47.324681+0000","last_peered":"2024-09-21T18:06:47.324681+0000","last_clean":"2024-09-21T18:06:47.324681+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:06:47.324681+0000","last_undegraded":"2024-09-21T18:06:47.324681+0000","last_fullsized":"2024-09-21T18:06:47.324681+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.980470+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:06:46.980470+0000","last_peered":"2024-09-21T18:06:46.980470+0000","last_clean":"2024-09-21T18:06:46.980470+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:06:46.980470+0000","last_undegraded":"2024-09-21T18:06:46.980470+0000","last_fullsized":"2024-09-21T18:06:46.980470+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.878313+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:06:46.878313+0000","last_peered":"2024-09-21T18:06:46.878313+0000","last_clean":"2024-09-21T18:06:46.878313+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:06:46.878313+0000","last_undegraded":"2024-09-21T18:06:46.878313+0000","last_fullsized":"2024-09-21T18:06:46.878313+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"28'2","reported_seq":22,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.982554+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:06:46.982554+0000","last_peered":"2024-09-21T18:06:46.982554+0000","last_clean":"2024-09-21T18:06:46.982554+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:06:46.982554+0000","last_undegraded":"2024-09-21T18:06:46.982554+0000","last_fullsized":"2024-09-21T18:06:46.982554+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.988258+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:06:47.988258+0000","last_peered":"2024-09-21T18:06:47.988258+0000","last_clean":"2024-09-21T18:06:47.988258+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:06:47.988258+0000","last_undegraded":"2024-09-21T18:06:47.988258+0000","last_fullsized":"2024-09-21T18:06:47.988258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:46.889135+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:06:46.889135+0000","last_peered":"2024-09-21T18:06:46.889135+0000","last_clean":"2024-09-21T18:06:46.889135+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:06:46.889135+0000","last_undegraded":"2024-09-21T18:06:46.889135+0000","last_fullsized":"2024-09-21T18:06:46.889135+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"26'1","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.096800+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:06:47.096800+0000","last_peered":"2024-09-21T18:06:47.096800+0000","last_clean":"2024-09-21T18:06:47.096800+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:06:47.096800+0000","last_undegraded":"2024-09-21T18:06:47.096800+0000","last_fullsized":"2024-09-21T18:06:47.096800+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"28'100","reported_seq":139,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:51.413279+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:06:51.413279+0000","last_peered":"2024-09-21T18:06:51.413279+0000","last_clean":"2024-09-21T18:06:51.413279+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:06:51.413279+0000","last_undegraded":"2024-09-21T18:06:51.413279+0000","last_fullsized":"2024-09-21T18:06:51.413279+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":131,"num_write_kb":2410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":131,"num_write_kb":2410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":934976,"data_compressed":59574,"data_compressed_allocated":73728,"data_compressed_original":901120,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313221,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":212992,"data_stored":96458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313222,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":94628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":266240,"data_stored":101522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280516,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31436,"kb_used_data":248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340404,"statfs":{"total":96636764160,"available":96604573696,"internally_reserved":0,"allocated":253952,"data_stored":96869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":192512,"data_stored":92136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280517,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27344,"kb_used_data":312,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344496,"statfs":{"total":96636764160,"available":96608763904,"internally_reserved":0,"allocated":319488,"data_stored":568372,"data_compressed":29350,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27280,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344560,"statfs":{"total":96636764160,"available":96608829440,"internally_reserved":0,"allocated":258048,"data_stored":100330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":100912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280516,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":200704,"data_stored":93328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280516,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31432,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":249856,"data_stored":96794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":311296,"data_stored":564892,"data_compressed":29350,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,13],"upper_bound":512},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":262144,"data_stored":100349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":262144,"data_stored":100912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":29787,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":467488,"data_compressed":29787,"data_compressed_allocated":36864,"data_compressed_original":450560,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:06:53.800 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T18:06:53.801 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T18:06:53.801 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T18:06:53.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T18:06:54.135 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:54.135 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T18:06:54.153 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T18:06:54.154 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T18:06:54.169 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T18:06:54.169 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:06:54.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/balancer/log_level debug' 2024-09-21T18:06:54.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-09-21T18:06:54.777 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "active": true, 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_duration": "0:00:00.000300", 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_started": "Sat Sep 21 18:06:33 2024", 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "upmap", 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "no_optimization_needed": false, 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "optimize_result": "No pools available", 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout: "plans": [] 2024-09-21T18:06:54.778 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:06:54.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph osd set-require-min-compat-client reef' 2024-09-21T18:06:56.070 INFO:teuthology.orchestra.run.smithi029.stderr:set require_min_compat_client to reef 2024-09-21T18:06:56.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode upmap-read' 2024-09-21T18:06:56.347 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:06:56.343+0000 7f6a3cca9640 -1 osd.3 29 handle_osd_map injecting map crc failure 2024-09-21T18:06:56.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-09-21T18:06:56.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status' 2024-09-21T18:06:56.839 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:06:56.837+0000 7fb924fb6640 -1 osd.8 29 handle_osd_map injecting map crc failure 2024-09-21T18:06:57.081 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:06:57.081 INFO:teuthology.orchestra.run.smithi029.stdout: "active": true, 2024-09-21T18:06:57.081 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_duration": "0:00:00.001415", 2024-09-21T18:06:57.081 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_started": "Sat Sep 21 18:06:56 2024", 2024-09-21T18:06:57.082 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "upmap-read", 2024-09-21T18:06:57.082 INFO:teuthology.orchestra.run.smithi029.stdout: "no_optimization_needed": true, 2024-09-21T18:06:57.082 INFO:teuthology.orchestra.run.smithi029.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-09-21T18:06:57.082 INFO:teuthology.orchestra.run.smithi029.stdout: "plans": [] 2024-09-21T18:06:57.082 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:06:57.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer status detail' 2024-09-21T18:06:57.416 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:06:57.416 INFO:teuthology.orchestra.run.smithi029.stdout: "active": true, 2024-09-21T18:06:57.416 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_duration": "0:00:00.001415", 2024-09-21T18:06:57.416 INFO:teuthology.orchestra.run.smithi029.stdout: "last_optimize_started": "Sat Sep 21 18:06:56 2024", 2024-09-21T18:06:57.416 INFO:teuthology.orchestra.run.smithi029.stdout: "mode": "upmap-read", 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "no_optimization_needed": true, 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items_added": [], 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items_removed": [], 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries_added": [], 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries_removed": [], 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout: "plans": [] 2024-09-21T18:06:57.417 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:06:57.436 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T18:06:57.452 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T18:06:57.452 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-rf-qprio-w-qa'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-qprio-w-qa', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}} 2024-09-21T18:06:57.452 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgnum_shrink': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'map_discontinuity_sleep_time': 200, 'min_in': 2, 'thrash_hosts': True, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-21T18:06:57.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T18:06:57.792 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T18:06:57.792 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T18:06:57.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:06:58.103 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:58.104 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:06:58.123 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-21T18:06:57.051688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":453228,"kb_used_data":4096,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496212,"statfs":{"total":1546188226560,"available":1545724121088,"internally_reserved":0,"allocated":4194304,"data_stored":2721024,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":82,"num_write_kb":1828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.840284+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:06:52.840284+0000","last_peered":"2024-09-21T18:06:52.840284+0000","last_clean":"2024-09-21T18:06:52.840284+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:06:52.840284+0000","last_undegraded":"2024-09-21T18:06:52.840284+0000","last_fullsized":"2024-09-21T18:06:52.840284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.799299+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:06:52.799299+0000","last_peered":"2024-09-21T18:06:52.799299+0000","last_clean":"2024-09-21T18:06:52.799299+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:06:52.799299+0000","last_undegraded":"2024-09-21T18:06:52.799299+0000","last_fullsized":"2024-09-21T18:06:52.799299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.349419+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:06:56.349419+0000","last_peered":"2024-09-21T18:06:56.349419+0000","last_clean":"2024-09-21T18:06:56.349419+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:06:56.349419+0000","last_undegraded":"2024-09-21T18:06:56.349419+0000","last_fullsized":"2024-09-21T18:06:56.349419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.945495+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:06:52.945495+0000","last_peered":"2024-09-21T18:06:52.945495+0000","last_clean":"2024-09-21T18:06:52.945495+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:06:52.945495+0000","last_undegraded":"2024-09-21T18:06:52.945495+0000","last_fullsized":"2024-09-21T18:06:52.945495+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.097829+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:06:56.097829+0000","last_peered":"2024-09-21T18:06:56.097829+0000","last_clean":"2024-09-21T18:06:56.097829+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:06:56.097829+0000","last_undegraded":"2024-09-21T18:06:56.097829+0000","last_fullsized":"2024-09-21T18:06:56.097829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.988258+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:06:47.988258+0000","last_peered":"2024-09-21T18:06:47.988258+0000","last_clean":"2024-09-21T18:06:47.988258+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:06:47.988258+0000","last_undegraded":"2024-09-21T18:06:47.988258+0000","last_fullsized":"2024-09-21T18:06:47.988258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.097820+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:06:56.097820+0000","last_peered":"2024-09-21T18:06:56.097820+0000","last_clean":"2024-09-21T18:06:56.097820+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:06:56.097820+0000","last_undegraded":"2024-09-21T18:06:56.097820+0000","last_fullsized":"2024-09-21T18:06:56.097820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.349129+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:06:56.349129+0000","last_peered":"2024-09-21T18:06:56.349129+0000","last_clean":"2024-09-21T18:06:56.349129+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:06:56.349129+0000","last_undegraded":"2024-09-21T18:06:56.349129+0000","last_fullsized":"2024-09-21T18:06:56.349129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"29'112","reported_seq":158,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.449147+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:06:56.449147+0000","last_peered":"2024-09-21T18:06:56.449147+0000","last_clean":"2024-09-21T18:06:56.449147+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:06:56.449147+0000","last_undegraded":"2024-09-21T18:06:56.449147+0000","last_fullsized":"2024-09-21T18:06:56.449147+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":103398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":101568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":103809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":105444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":335872,"data_stored":636752,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":107270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":100268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":110618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":327680,"data_stored":633272,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:06:58.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:06:58.428 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:06:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:06:58.445 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-21T18:06:57.051688+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":147,"num_write_kb":2674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":12,"kb":1509949440,"kb_used":453228,"kb_used_data":4096,"kb_used_omap":24,"kb_used_meta":448743,"kb_avail":1509496212,"statfs":{"total":1546188226560,"available":1545724121088,"internally_reserved":0,"allocated":4194304,"data_stored":2721024,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":25520,"internal_metadata":459512912},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":42,"num_read_kb":38,"num_write":82,"num_write_kb":1828,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.840284+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:06:52.840284+0000","last_peered":"2024-09-21T18:06:52.840284+0000","last_clean":"2024-09-21T18:06:52.840284+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:06:52.840284+0000","last_undegraded":"2024-09-21T18:06:52.840284+0000","last_fullsized":"2024-09-21T18:06:52.840284+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.799299+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:06:52.799299+0000","last_peered":"2024-09-21T18:06:52.799299+0000","last_clean":"2024-09-21T18:06:52.799299+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:06:52.799299+0000","last_undegraded":"2024-09-21T18:06:52.799299+0000","last_fullsized":"2024-09-21T18:06:52.799299+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.349419+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:06:56.349419+0000","last_peered":"2024-09-21T18:06:56.349419+0000","last_clean":"2024-09-21T18:06:56.349419+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:06:56.349419+0000","last_undegraded":"2024-09-21T18:06:56.349419+0000","last_fullsized":"2024-09-21T18:06:56.349419+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T18:06:52.945495+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:06:52.945495+0000","last_peered":"2024-09-21T18:06:52.945495+0000","last_clean":"2024-09-21T18:06:52.945495+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:06:52.945495+0000","last_undegraded":"2024-09-21T18:06:52.945495+0000","last_fullsized":"2024-09-21T18:06:52.945495+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":26,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.097829+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:06:56.097829+0000","last_peered":"2024-09-21T18:06:56.097829+0000","last_clean":"2024-09-21T18:06:56.097829+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:06:56.097829+0000","last_undegraded":"2024-09-21T18:06:56.097829+0000","last_fullsized":"2024-09-21T18:06:56.097829+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T18:06:47.988258+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:06:47.988258+0000","last_peered":"2024-09-21T18:06:47.988258+0000","last_clean":"2024-09-21T18:06:47.988258+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:06:47.988258+0000","last_undegraded":"2024-09-21T18:06:47.988258+0000","last_fullsized":"2024-09-21T18:06:47.988258+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":24,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.097820+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:06:56.097820+0000","last_peered":"2024-09-21T18:06:56.097820+0000","last_clean":"2024-09-21T18:06:56.097820+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:06:56.097820+0000","last_undegraded":"2024-09-21T18:06:56.097820+0000","last_fullsized":"2024-09-21T18:06:56.097820+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.349129+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:06:56.349129+0000","last_peered":"2024-09-21T18:06:56.349129+0000","last_clean":"2024-09-21T18:06:56.349129+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:06:56.349129+0000","last_undegraded":"2024-09-21T18:06:56.349129+0000","last_fullsized":"2024-09-21T18:06:56.349129+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"1.0","version":"29'112","reported_seq":158,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-09-21T18:06:56.449147+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:06:56.449147+0000","last_peered":"2024-09-21T18:06:56.449147+0000","last_clean":"2024-09-21T18:06:56.449147+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:06:56.449147+0000","last_undegraded":"2024-09-21T18:06:56.449147+0000","last_fullsized":"2024-09-21T18:06:56.449147+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247814,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,9,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247812,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27272,"kb_used_data":244,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344568,"statfs":{"total":96636764160,"available":96608837632,"internally_reserved":0,"allocated":249856,"data_stored":94564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247813,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27284,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":258048,"data_stored":98042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313222,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":225280,"data_stored":103398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":212992,"data_stored":101568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":108462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280517,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31448,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340392,"statfs":{"total":96636764160,"available":96604561408,"internally_reserved":0,"allocated":266240,"data_stored":103809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":105444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280518,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":335872,"data_stored":636752,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":107270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280517,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":100268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280517,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31456,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":274432,"data_stored":110618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280518,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":327680,"data_stored":633272,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280518,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107289,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280518,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27296,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":274432,"data_stored":107852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:06:58.446 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T18:06:58.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-21T18:06:58.742 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 30 2024-09-21T18:06:58.742 INFO:teuthology.orchestra.run.smithi029.stdout:fsid ea312541-f204-4ac1-b2ae-f5daa6b7f078 2024-09-21T18:06:58.742 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-09-21T18:06:28.087618+0000 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-09-21T18:06:56.061134+0000 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 5 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client reef 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client jewel 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 25 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 15.38 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'rbd' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 28 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 4.00 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 16 2024-09-21T18:06:58.743 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6808/3976261094 v1:172.21.15.29:6809/3976261094 exists,up f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6808/408295210 v1:172.21.15.40:6809/408295210 exists,up 020df4b2-13cb-4e14-88a1-a8219b5b34b0 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6804/1064027255 v1:172.21.15.70:6805/1064027255 exists,up 57a83527-6610-4eb4-a01e-e22c3ade98a5 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6813/1036992123 v1:172.21.15.160:6814/1036992123 exists,up 42758eff-818c-4e2b-98c8-5f9c01c883c9 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6800/2670503742 v1:172.21.15.29:6801/2670503742 exists,up c358d178-864a-4a0a-a1a8-0c42d1b10d86 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6800/3912255253 v1:172.21.15.40:6801/3912255253 exists,up 73790513-6074-4d9b-80b2-0403d3c1f0c6 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6800/3623713846 v1:172.21.15.70:6801/3623713846 exists,up e588347d-7853-4ac0-a871-03a7311247e4 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6801/501927900 v1:172.21.15.160:6802/501927900 exists,up f47cab83-6e0b-478f-88e0-0cf069bb3f3c 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6804/4039331719 v1:172.21.15.29:6805/4039331719 exists,up 427789c1-98e4-43d1-a12c-2606a8a6cd4b 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6812/3437338221 v1:172.21.15.40:6813/3437338221 exists,up 268cb87f-9f31-4543-8fc5-4e05b6d9ed0d 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6808/3389947982 v1:172.21.15.70:6809/3389947982 exists,up 85e66756-8ef8-4ba9-911f-08d74f04e0a1 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6809/524032079 v1:172.21.15.160:6810/524032079 exists,up 952e9412-fdb5-4b68-b109-def00c946b31 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6812/2242191432 v1:172.21.15.29:6813/2242191432 exists,up 1b4bd0f6-ed9f-426e-bb82-4e4328199d79 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6804/707207003 v1:172.21.15.40:6805/707207003 exists,up a535d39d-7bee-4185-a28d-4155f9b17207 2024-09-21T18:06:58.744 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6812/2255369090 v1:172.21.15.70:6813/2255369090 exists,up 95b8504b-67c0-4ed9-be7e-81eb3dacacf6 2024-09-21T18:06:58.745 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6805/4046424455 v1:172.21.15.160:6806/4046424455 exists,up 84b8cc82-bc18-4cce-901e-ed48e85ba7e3 2024-09-21T18:06:58.761 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6808/3976261094 v1:172.21.15.29:6809/3976261094 exists,up f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7', 'osd.1 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6808/408295210 v1:172.21.15.40:6809/408295210 exists,up 020df4b2-13cb-4e14-88a1-a8219b5b34b0', 'osd.2 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6804/1064027255 v1:172.21.15.70:6805/1064027255 exists,up 57a83527-6610-4eb4-a01e-e22c3ade98a5', 'osd.3 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6813/1036992123 v1:172.21.15.160:6814/1036992123 exists,up 42758eff-818c-4e2b-98c8-5f9c01c883c9', 'osd.4 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6800/2670503742 v1:172.21.15.29:6801/2670503742 exists,up c358d178-864a-4a0a-a1a8-0c42d1b10d86', 'osd.5 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6800/3912255253 v1:172.21.15.40:6801/3912255253 exists,up 73790513-6074-4d9b-80b2-0403d3c1f0c6', 'osd.6 up in weight 1 up_from 21 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6800/3623713846 v1:172.21.15.70:6801/3623713846 exists,up e588347d-7853-4ac0-a871-03a7311247e4', 'osd.7 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6801/501927900 v1:172.21.15.160:6802/501927900 exists,up f47cab83-6e0b-478f-88e0-0cf069bb3f3c', 'osd.8 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6804/4039331719 v1:172.21.15.29:6805/4039331719 exists,up 427789c1-98e4-43d1-a12c-2606a8a6cd4b', 'osd.9 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6812/3437338221 v1:172.21.15.40:6813/3437338221 exists,up 268cb87f-9f31-4543-8fc5-4e05b6d9ed0d', 'osd.10 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6808/3389947982 v1:172.21.15.70:6809/3389947982 exists,up 85e66756-8ef8-4ba9-911f-08d74f04e0a1', 'osd.11 up in weight 1 up_from 22 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6809/524032079 v1:172.21.15.160:6810/524032079 exists,up 952e9412-fdb5-4b68-b109-def00c946b31', 'osd.12 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.29:6812/2242191432 v1:172.21.15.29:6813/2242191432 exists,up 1b4bd0f6-ed9f-426e-bb82-4e4328199d79', 'osd.13 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.40:6804/707207003 v1:172.21.15.40:6805/707207003 exists,up a535d39d-7bee-4185-a28d-4155f9b17207', 'osd.14 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) v1:172.21.15.70:6812/2255369090 v1:172.21.15.70:6813/2255369090 exists,up 95b8504b-67c0-4ed9-be7e-81eb3dacacf6', 'osd.15 up in weight 1 up_from 23 up_thru 25 down_at 0 last_clean_interval [0,0) v1:172.21.15.160:6805/4046424455 v1:172.21.15.160:6806/4046424455 exists,up 84b8cc82-bc18-4cce-901e-ed48e85ba7e3'] 2024-09-21T18:06:58.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-21T18:06:58.876 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:06:58.876 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mon.a", 2024-09-21T18:06:58.876 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster": "ceph", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "admin_socket_mode": "", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_client_required": "cephx, none", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_cluster_required": "cephx", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_debug": "true", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_service_required": "cephx", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_supported": "cephx", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio": "true", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T18:06:58.877 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_reap_max": "16", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_async_discard_threads": "0", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_block_size": "4096", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio": "false", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_enable_discard": "false", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_flock_retry": "3", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_inject_crash": "0", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T18:06:58.878 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring_hipri": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_type": "", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_allocator": "hybrid", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_buffered_io": "true", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T18:06:58.879 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_replay_recovery": "false", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_sync_write": "false", 2024-09-21T18:06:58.880 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_allocator": "hybrid", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_async_db_compaction": "true", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T18:06:58.881 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_create": "true", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_create": "false", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_path": "", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_size": "0", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_path": "", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_size": "107374182400", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_create": "false", 2024-09-21T18:06:58.882 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_path": "", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs": "true", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_autotune": "true", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T18:06:58.883 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size": "0", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_type": "2q", 2024-09-21T18:06:58.884 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_clone_cow": "true", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_mode": "none", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T18:06:58.885 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_freelist": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T18:06:58.886 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T18:06:58.887 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fail_eio": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T18:06:58.888 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T18:06:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size": "0", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T18:06:58.890 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_mem": "512", 2024-09-21T18:06:58.891 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_tracing": "false", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T18:06:58.892 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_write_v2": "false", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_write_v2_random": "false", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T18:06:58.893 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_cluster_require_version": "2", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_require_signatures": "false", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_require_version": "2", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_service_require_signatures": "false", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_service_require_version": "2", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_sign_messages": "true", 2024-09-21T18:06:58.894 INFO:teuthology.orchestra.run.smithi029.stdout: "chdir": "", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_acl_type": "", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_asio_thread_count": "2", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_cache_mid": "0.750000", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_cache_size": "16384", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_caps_release_delay": "5", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_check_pool_perm": "true", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_force_sync_read": "false", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_getattr_caps": "false", 2024-09-21T18:06:58.895 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_inject_features": "", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_die_on_failed_remount": "false", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_dirsize_rbytes": "true", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_force_lazyio": "false", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_fs": "", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_inject_release_failure": "false", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_max_inline_size": "4096", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T18:06:58.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mds_namespace": "", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_metadata": "", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_gid": "-1", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_timeout": "300", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_uid": "-1", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mountpoint": "/", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_notify_timeout": "10", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc": "true", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_objects": "1000", 2024-09-21T18:06:58.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_size": "209715200", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_permissions": "true", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_quota": "true", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_quota_df": "true", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_max_bytes": "0", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_max_periods": "4", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_min": "131072", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_reconnect_stale": "false", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_shutdown_timeout": "30", 2024-09-21T18:06:58.898 INFO:teuthology.orchestra.run.smithi029.stdout: "client_snapdir": ".snap", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "client_tick_interval": "1", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "client_trace": "", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "client_use_faked_inos": "false", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "client_use_random_mds": "false", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog": "false", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog_port": "12201", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_monitors": "default=true", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog": "false", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog_level": "info", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "-", 2024-09-21T18:06:58.899 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_network": "", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_network_interface": "", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_isal": "false", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_level": "5", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zstd_level": "1", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T18:06:58.900 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location": "", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location_hook": "", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location_hook_timeout": "10", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "daemonize": "false", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "daos_pool": "tank", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_alienstore": "0/5", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asok": "1/5", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asok_assert_abort": "false", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_auth": "20/20", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bdev": "1/3", 2024-09-21T18:06:58.901 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bluefs": "1/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bluestore": "1/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_buffer": "0/1", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_ceph_dedup": "0/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cephsqlite": "0/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_client": "0/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_compressor": "1/5", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_context": "0/1", 2024-09-21T18:06:58.902 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crush": "1/1", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crypto": "1/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cyanstore": "0/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_dpdk": "1/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_eventtrace": "1/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_filer": "0/1", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_filestore": "1/3", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_finisher": "1/1", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_fuse": "1/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T18:06:58.903 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_javaclient": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_journal": "1/3", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_journaler": "0/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_kstore": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_lockdep": "0/1", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mclock": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_balancer": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_locker": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_log": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_migrator": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_memstore": "1/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mgr": "2/5", 2024-09-21T18:06:58.904 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mgrc": "1/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mon": "20/20", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_monc": "0/10", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_ms": "1/1", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_none": "0/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objclass": "0/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objectcacher": "0/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objecter": "0/1", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_optracker": "0/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_osd": "1/5", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_paxos": "20/20", 2024-09-21T18:06:58.905 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_perfcounter": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_prioritycache": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rados": "0/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd": "0/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_replay": "0/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_refs": "0/0", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_reserver": "1/1", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_access": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T18:06:58.906 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_flight": "1/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_notification": "1/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_sync": "1/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rocksdb": "4/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_backref": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_cache": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_device": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_epm": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_journal": "0/5", 2024-09-21T18:06:58.907 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_lba": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_odata": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_omap": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_onode": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_t": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_tm": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_striper": "0/1", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_test": "0/5", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_throttle": "1/1", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_timer": "0/1", 2024-09-21T18:06:58.908 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_tp": "0/5", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_trace": "1/5", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "device_failure_prediction_mode": "none", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_graylog": "false", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_journald": "false", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_stderr": "true", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_syslog": "false", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event_tracing": "false", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_cert_file": "", 2024-09-21T18:06:58.909 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_http_port": "9926", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_key_file": "", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_prio_limit": "5", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_sort_metrics": "true", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_stats_period": "5", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "fake_statfs_for_testing": "0", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "fatal_signal_handlers": "true", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "filer_max_purge_ops": "10", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "filer_max_truncate_ops": "128", 2024-09-21T18:06:58.910 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_blackhole": "false", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_btrfs_snap": "true", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_caller_concurrency": "10", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_omap_check": "false", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_verify_split": "false", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_dump_file": "", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fadvise": "true", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fail_eio": "true", 2024-09-21T18:06:58.911 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fd_cache_size": "128", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fiemap": "false", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_inject_stall": "0", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_parallel": "false", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_trailing": "false", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_writeahead": "false", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_kill_at": "0", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T18:06:58.912 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T18:06:58.913 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_merge_threshold": "-10", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_odsync_write": "false", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_backend_path": "", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T18:06:58.914 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_threads": "2", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_punch_hole": "false", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T18:06:58.915 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_ops": "50", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_seek_data_hole": "false", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_sloppy_crc": "false", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_splice": "false", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_split_multiple": "2", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_split_rand_factor": "20", 2024-09-21T18:06:58.916 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_update_to": "1000", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T18:06:58.917 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_xfs_extsize": "false", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_zfs_snap": "false", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fio_dir": "/tmp/fio", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "ea312541-f204-4ac1-b2ae-f5daa6b7f078", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_allow_other": "true", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_big_writes": "true", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_debug": "false", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_default_permissions": "false", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_disable_pagecache": "false", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_max_write": "0", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_multithreaded": "true", 2024-09-21T18:06:58.918 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_require_active_mds": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_set_user_groups": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_move": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_read": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_write": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T18:06:58.919 INFO:teuthology.orchestra.run.smithi029.stdout: "gss_target_name": "ceph", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_file": "", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_inject_failure": "0", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_interval": "5", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "host": "smithi029", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T18:06:58.920 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T18:06:58.921 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "inject_early_sigterm": "false", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "jaeger_agent_port": "6799", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "jaeger_tracing_enable": "false", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_aio": "true", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_align_min_size": "65536", 2024-09-21T18:06:58.922 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_block_align": "true", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_block_size": "4096", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_dio": "true", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_discard": "false", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_force_aio": "false", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_ignore_corruption": "false", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_max_write_entries": "100", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_replay_from": "0", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T18:06:58.923 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_write_header_frequency": "0", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_zero_on_create": "false", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_prefetch_periods": "10", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_prezero_periods": "5", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_write_head_interval": "15", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "keyfile": "", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_backend": "rocksdb", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T18:06:58.924 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_max_bytes": "67108864", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_max_ops": "512", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_onode_map_size": "1024", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_sync_transaction": "false", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "librados_thread_count": "2", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "lockdep": "false", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "lockdep_force_backtrace": "false", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "log_coarse_timestamps": "true", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T18:06:58.925 INFO:teuthology.orchestra.run.smithi029.stdout: "log_flush_on_exit": "false", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_graylog_port": "12201", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_max_new": "1000", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_max_recent": "10000", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_stderr_prefix": "", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": "true", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_graylog": "false", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_journald": "false", 2024-09-21T18:06:58.926 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_stderr": "false", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_syslog": "false", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_action_on_write_error": "1", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_alternate_name_max": "8192", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_asio_thread_count": "2", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_export_pin": "true", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T18:06:58.927 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_interval": "10", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_max": "-1", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_max_until": "-1", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_merge_size": "50", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_mode": "0", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_bits": "3", 2024-09-21T18:06:58.928 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_size": "10000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_mid": "0.700000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T18:06:58.929 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_interval": "1", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_auth_pins": "false", 2024-09-21T18:06:58.930 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_frag": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_scatterstat": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_subtrees": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_default_dir_hash": "2", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_defer_session_stale": "true", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_max_entries": "0", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_prefetch": "true", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T18:06:58.931 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_early_reply": "true", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_enable_op_tracker": "true", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_enforce_unique_name": "true", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T18:06:58.932 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_health_dummy": "false", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T18:06:58.933 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_join_fs": "", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_journal_format": "1", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_create_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_dirfrag_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_export_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_import_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_link_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_openc_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_rename_at": "0", 2024-09-21T18:06:58.934 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_max_events": "-1", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_max_segments": "128", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_pause": "false", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_segment_size": "0", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_threshold": "128", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T18:06:58.935 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_completed_requests": "100000", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_export_size": "20971520", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_file_recover": "32", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_files": "64", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_ops": "8192", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_metrics_update_interval": "2", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_min_caps_per_client": "100", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_numa_node": "-1", 2024-09-21T18:06:58.936 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_duration": "600", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_size": "20", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_log_threshold": "5", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_ping_grace": "15", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_ping_interval": "5", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_caps": "30000", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T18:06:58.937 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_replay_interval": "1.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_root_ino_gid": "0", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_root_ino_uid": "0", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_scrub_stats_review_period": "1", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T18:06:58.938 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_shutdown_check": "0", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_skip_ino": "0", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_min_uid": "0", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_rstat": "false", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_symlink_recovery": "true", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T18:06:58.939 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_thrash_exports": "0", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_thrash_fragments": "0", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_tick_interval": "5.000000", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_valgrind_exit": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_verify_backtrace": "1", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_verify_scatter": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_wipe_sessions": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool_debug": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_page_set": "false", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_page_size": "65536", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_bytes": "134217728", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_messages": "512", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T18:06:58.940 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_disabled_modules": "", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_max_pg_creating": "1024", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mds_messages": "128", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mon_messages": "128", 2024-09-21T18:06:58.941 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_history_duration": "600", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_history_size": "20", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_op_log_threshold": "5", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_osd_messages": "8192", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_pool": "true", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_standby_modules": "true", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_stats_period": "5", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_stats_threshold": "5", 2024-09-21T18:06:58.942 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_tick_period": "2", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_allow_pool_delete": "true", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_allow_pool_size_one": "true", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_bytes": "104857600", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T18:06:58.943 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_target_rank": "-1", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_level": "debug", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T18:06:58.944 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_journald": "false", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_start": "false", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_trim": "true", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cpu_threads": "4", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T18:06:58.945 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_avail_crit": "5", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_avail_warn": "5", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_json": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_extra_checks": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T18:06:58.946 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_down_mkfs_grace": "120", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_down_uptime_grace": "60", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_election_default_strategy": "1", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_election_timeout": "5.000000", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_enable_op_tracker": "true", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_fake_pool_delete": "false", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_force_quorum_join": "false", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_detail_to_clog": "false", 2024-09-21T18:06:58.947 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_log_update_period": "5", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_max_detail": "50", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog": "true", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_host": "[v1:172.21.15.29:6789],[v1:172.21.15.40:6789],[v1:172.21.15.70:6789]", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_host_override": "", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_initial_members": "", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T18:06:58.948 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease": "5.000000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_full_interval": "50", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_max": "10000", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_max_summary": "50", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_log_epochs": "500", 2024-09-21T18:06:58.949 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_nvmeof_epochs": "500", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_osd": "10000", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_memory_autotune": "true", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_memory_target": "2147483648", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T18:06:58.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_digest_period": "5", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_min_osdmap_epochs": "50", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_complaint_time": "30", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_duration": "600", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_size": "20", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_log_threshold": "5", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T18:06:58.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_cache_size": "500", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T18:06:58.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_report_timeout": "900", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T18:06:58.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T18:06:58.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_max_osds": "4", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_interval": "300", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_max_keys": "100", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_timeout": "300", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_session_timeout": "300", 2024-09-21T18:06:58.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_smart_report_timeout": "5", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_pool_size": "4", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_debug": "false", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T18:06:58.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_tick_interval": "5", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_older_version": "true", 2024-09-21T18:06:58.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "monmap": "", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T18:06:58.958 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_tracing_enabled": "false", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_op_threads": "3", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_cm": "false", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_device_name": "", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T18:06:58.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_sl": "3", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_type": "ib", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_reap_threshold": "5", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_before_connect": "false", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_ipv4": "true", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_ipv6": "false", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_msgr1": "true", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_msgr2": "false", 2024-09-21T18:06:58.960 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_port_max": "7568", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_port_min": "6800", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_retry_count": "3", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_retry_delay": "5", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_client": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mds": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mgr": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mon": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_osd": "false", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_client_mode": "crc secure", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T18:06:58.961 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_cluster_type": "", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_compress_secure": "false", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_crc_data": "true", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_crc_header": "true", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T18:06:58.962 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_bug": "true", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_old_message": "true", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hugepages": "", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_lro": "true", 2024-09-21T18:06:58.963 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_pmd": "", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_port_id": "0", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dump_on_send": "false", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_type": "", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_network_congestion": "0", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_socket_failures": "0", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_max_accept_failures": "4", 2024-09-21T18:06:58.964 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_max_backoff": "15.000000", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compress_mode": "none", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_pq_min_cost": "65536", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_public_type": "", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_service_mode": "crc secure", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_nodelay": "true", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T18:06:58.965 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_time_events_min_wait_interval": "1000", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_type": "async", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "no_config_file": "false", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "nvmeof_mon_client_tick_period": "2", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inflight_ops": "1024", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_timeout": "10.000000", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objectstore_blackhole": "false", 2024-09-21T18:06:58.966 INFO:teuthology.orchestra.run.smithi029.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "openssl_engine_opts": "", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_max_ops": "4", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_slop": "0.020000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_asio_thread_count": "2", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_async_recovery_min_cost": "1", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_scan_max": "512", 2024-09-21T18:06:58.967 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_scan_min": "64", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_peering": "false", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_beacon_report_interval": "300", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_duration": "30", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_blkin_trace_all": "false", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T18:06:58.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_update_on_start": "true", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_message_cap": "256", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_op_priority": "63", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_watch_timeout": "30", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_command_max_records": "256", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_compact_on_start": "false", 2024-09-21T18:06:58.969 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_update_on_start": "true", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_dispatch_delay_probability": "0.100000", 2024-09-21T18:06:58.970 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_op_order": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_shutdown": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_trim_objects": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T18:06:58.971 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_default_notify_timeout": "30", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T18:06:58.972 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_ec_partial_reads": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_enable_op_tracker": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_info": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T18:06:58.973 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_function_tracing": "false", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_grace": "20", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_interval": "6", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_stale": "600", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T18:06:58.974 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal_size": "5120", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_kill_backfill_at": "0", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_cache_size": "50", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_dedup": "true", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_message_max": "40", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_attr_name_len": "100", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_attr_size": "0", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_backfills": "1", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_markdown_count": "5", 2024-09-21T18:06:58.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_markdown_period": "600", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_name_len": "2048", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_size": "134217728", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pgls": "1024", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_push_cost": "8388608", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_push_objects": "10", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_scrubs": "3", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T18:06:58.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_write_size": "90", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_profile": "balanced", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T18:06:58.977 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_base": "805306368", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target": "4294967296", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target_autotune": "false", 2024-09-21T18:06:58.978 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_min_recovery_priority": "0", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_report_interval": "5", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_num_cache_shards": "32", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_node": "-1", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T18:06:58.979 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_object_clean_region_max_num_intervals": "1000", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objecter_finishers": "1", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore": "bluestore", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore_fuse": "false", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore_ideal_list_max": "64", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore_tracing": "false", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_duration": "600", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_size": "20", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_log_threshold": "5", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards": "0", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T18:06:58.980 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_thread_timeout": "15", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_open_classes_on_start": "true", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_os_flags": "0", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_peering_op_priority": "255", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T18:06:58.981 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_delete_priority": "5", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T18:06:58.982 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_crimson": "false", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T18:06:58.983 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flags": "0", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_min_size": "2", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_size": "2", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_type": "replicated", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T18:06:58.984 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_cost": "20971520", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active": "0", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_op_priority": "3", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_priority": "5", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T18:06:58.985 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_repair_during_recovery": "false", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_requested_scrub_priority": "5", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_chunk_max": "15", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T18:06:58.986 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_cost": "52428800", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_end_hour": "0", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_priority": "5", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_retry_after_noscrub": "60", 2024-09-21T18:06:58.987 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_retry_delay": "30", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_retry_new_interval": "10", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_retry_pg_state": "60", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_retry_trimming": "10", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_skip_data_digest": "false", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_smart_report_timeout": "5", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T18:06:58.988 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_priority": "5", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_target_transaction_size": "30", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T18:06:58.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tracing": "false", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_use_stale_snap": "false", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_kill_at": "0", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_max_join_drift": "10", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_min": "500", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_min_wait": "0.050000", 2024-09-21T18:06:58.990 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_max": "500", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_min": "10", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_stash_full_interval": "25", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_trim_max": "500", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_trim_min": "250", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "perf": "true", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "-", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrv": "", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "public_bind_addr": "-", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "public_network": "", 2024-09-21T18:06:58.991 INFO:teuthology.orchestra.run.smithi029.stdout: "public_network_interface": "", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "qat_compressor_enabled": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_mon_op_timeout": "0", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_osd_op_timeout": "0", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_tracing": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_atime_update_interval": "60", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache": "true", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T18:06:58.992 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_policy": "writearound", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_size": "33554432", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_compression_hint": "none", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_clone_format": "auto", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_data_pool": "", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_features": "61", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_format": "2", 2024-09-21T18:06:58.993 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_map_options": "", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_order": "22", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_pool": "rbd", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_stripe_count": "0", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_io_scheduler": "simple", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T18:06:58.994 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_order": "24", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_pool": "", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_splay_width": "4", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T18:06:58.995 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T18:06:58.996 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T18:06:58.997 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_op_threads": "1", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T18:06:58.998 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_plugins": "", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T18:06:58.999 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_tracing": "false", 2024-09-21T18:06:59.000 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_validate_names": "true", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_validate_pool": "true", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "restapi_base_url": "", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "restapi_log_level": "", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_admin_entry": "admin", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_asio_assert_yielding": "false", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_backend_store": "rados", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_barbican_url": "", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_beast_enable_async": "true", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T18:06:59.001 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_enabled": "true", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_config_store": "rados", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_content_length_compat": "false", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T18:06:59.002 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cross_domain_policy": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T18:06:59.003 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T18:06:59.004 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T18:06:59.005 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-09-21T18:06:59.006 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_l1_fadvise": "4", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_window": "30", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T18:06:59.007 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_disable_s3select": "false", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T18:06:59.008 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dns_name": "", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dns_s3website_name": "", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T18:06:59.009 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_mdsearch": "true", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_ops_log": "false", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_static_website": "false", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_usage_log": "false", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_expose_bucket": "false", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_extended_http_attrs": "", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_filter": "none", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T18:06:59.010 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_objs": "32", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_init_timeout": "300", 2024-09-21T18:06:59.011 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_password": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_project": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_token": "", 2024-09-21T18:06:59.012 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_user": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_url": "", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T18:06:59.013 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_objs": "32", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_worker": "3", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_uri": "", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lfuda_sync_frequency": "60", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T18:06:59.014 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_http_headers": "", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attr_size": "0", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T18:06:59.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_listing_results": "5000", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_notify_retries": "10", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T18:06:59.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T18:06:59.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_num_control_oids": "8", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_numa_node": "-1", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T18:06:59.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_tracing": "false", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_token": "", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_url": "", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_rados": "false", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T18:06:59.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_print_continue": "true", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T18:06:59.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_tracing": "false", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm": "", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm_id": "", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_region": "", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_request_uri": "", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T18:06:59.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_debug_interval": "-1", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_progress_judge_interval": "120", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshardlog_threshold": "30000", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_resolve_cname": "false", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_run_sync_thread": "true", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T18:06:59.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_script_uri": "", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_service_provider_name": "", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_client_id": "", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_client_secret": "", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_entry": "sts", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_key": "sts", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T18:06:59.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_auth_url": "", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_custom_header": "", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_need_stats": "true", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_tenant_name": "", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_url": "", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T18:06:59.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_lease_period": "120", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_thread_pool_size": "512", 2024-09-21T18:06:59.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_comment": "", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_createby": "", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_encoding": "", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_flag": "false", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_origin": "", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_tracker": "", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T18:06:59.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_max_shards": "32", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_use_opa_authz": "false", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_counters_cache": "false", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_unique_email": "true", 2024-09-21T18:06:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_verify_ssl": "true", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone": "", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone_id": "", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup": "", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup_id": "", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_block_size": "4096", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T18:06:59.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_partition_filters": "false", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_perf": "false", 2024-09-21T18:06:59.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "run_dir": "/var/run/ceph", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_block_create": "true", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_device_size": "53687091200", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_full_integrity_check": "false", 2024-09-21T18:06:59.030 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_main_device_type": "SSD", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T18:06:59.031 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_segment_size": "67108864", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "set_keepcaps": "false", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "setgroup": "", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "setuser": "", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "setuser_match_path": "", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "thp": "false", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "threadpool_default_timeout": "60", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "throttler_perf_counter": "true", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tmp_dir": "/tmp", 2024-09-21T18:06:59.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2024-09-21T18:06:59.033 INFO:teuthology.orchestra.run.smithi029.stdout: "uadk_compressor_enabled": "false", 2024-09-21T18:06:59.033 INFO:teuthology.orchestra.run.smithi029.stdout: "uadk_wd_sync_ctx_num": "2" 2024-09-21T18:06:59.033 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:06:59.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T18:06:59.183 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a: {} 2024-09-21T18:06:59.183 INFO:teuthology.orchestra.run.smithi029.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:06:59.196 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b: {} 2024-09-21T18:06:59.196 INFO:teuthology.orchestra.run.smithi029.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:06:59.210 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c: {} 2024-09-21T18:06:59.210 INFO:teuthology.orchestra.run.smithi029.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:06:59.226 DEBUG:teuthology.orchestra.run.smithi029:> type ceph-objectstore-tool 2024-09-21T18:06:59.230 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T18:06:59.230 DEBUG:teuthology.orchestra.run.smithi160:> type ceph-objectstore-tool 2024-09-21T18:06:59.236 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T18:06:59.236 DEBUG:teuthology.orchestra.run.smithi040:> type ceph-objectstore-tool 2024-09-21T18:06:59.241 INFO:teuthology.orchestra.run.smithi040.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T18:06:59.242 DEBUG:teuthology.orchestra.run.smithi070:> type ceph-objectstore-tool 2024-09-21T18:06:59.247 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T18:06:59.247 INFO:teuthology.run_tasks:Running task exec... 2024-09-21T18:06:59.266 INFO:teuthology.task.exec:Executing custom commands... 2024-09-21T18:06:59.266 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:06:59.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create base 4' 2024-09-21T18:06:59.267 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T18:06:59.267 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] out_osds: [] dead_osds: [] live_osds: [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-21T18:06:59.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T18:06:59.267 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T18:06:59.268 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T18:06:59.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:06:59.268 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T18:06:59.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:06:59.268 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T18:06:59.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:06:59.284 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:06:59.282+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:06:59.369 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:06:59.367+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:06:59.398 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:06:59.398 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:06:59.398 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:06:59.398 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:06:59.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:06:59.414 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:06:59.414 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:06:59.435 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:06:59.435 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:06:59.456 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:06:59.456 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:06:59.471 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:06:59.467+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:06:59.475 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:06:59.475 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:06:59.507 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:06:59.508 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:06:59.525 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:06:59.525 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:06:59.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:06:59.542 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:06:59.543 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:06:59.563 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:06:59.563 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:06:59.572 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:59.571+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:06:59.597 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:06:59.597 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:06:59.615 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:06:59.615 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:06:59.650 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:06:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:06:59.654 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:06:59.654 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:06:59.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:06:59.670 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:06:59.670 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:06:59.672 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:06:59.670+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:06:59.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:06:59.691 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:06:59.706 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:06:59.707 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:06:59.727 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:06:59.727 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:06:59.747 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:06:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:06:59.768 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:06:59.768 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:06:59.773 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:59.771+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:06:59.778 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:06:59.779 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:06:59.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:06:59.874 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:06:59.871+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:06:59.953 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:06:59.953 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:06:59.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:06:59.974 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:06:59.971+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:00.075 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:00.071+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:00.089 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:07:00.090 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'base' created 2024-09-21T18:07:00.090 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.090 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T18:07:00.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:00.115 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool application enable base rados' 2024-09-21T18:07:00.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T18:07:00.146 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:00.143+0000 7fb260c0e640 -1 osd.11 30 handle_osd_map injecting map crc failure 2024-09-21T18:07:00.176 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:00.175+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:00.241 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.241 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:00.298 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:00.297+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:00.378 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:00.375+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:00.378 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.378 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:00.479 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:00.475+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.555 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:00.579 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:00.575+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:00.680 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:00.675+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:00.692 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.692 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:00.781 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:00.777+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:00.831 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:00.831 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:00.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:00.882 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:00.879+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:00.983 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:00.982+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:01.014 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.015 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:01.084 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:01.079+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:01.095 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T18:07:01.100 INFO:teuthology.orchestra.run.smithi160.stderr:enabled application 'rados' on pool 'base' 2024-09-21T18:07:01.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T18:07:01.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool create cache 4' 2024-09-21T18:07:01.168 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.168 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.182 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:01.186 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:01.182+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:01.283 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:01.281+0000 7f8c3a0ca640 -1 osd.12 31 handle_osd_map injecting map crc failure 2024-09-21T18:07:01.285 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:01.281+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:01.309 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.309 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.324 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:01.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:01.382+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:01.451 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.451 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.465 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:01.486 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:01.485+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:01.588 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:01.586+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:01.634 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.635 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.649 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:01.689 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:01.683+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:01.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:01.789 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:01.787+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:01.818 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.818 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:01.890 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:01.887+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:01.941 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:01.960 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:01.960 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:01.960 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:01.960 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:01.974 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:01.981 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:01.991 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:01.987+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:02.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:02.002 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:02.020 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:02.020 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:02.036 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:02.037 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:02.057 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:02.057 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:02.077 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:02.077 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:02.091 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:02.090+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:02.094 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T18:07:02.097 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'cache' created 2024-09-21T18:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:02.104 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:02.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:07:02.116 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:02.117 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:02.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T18:07:02.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:02.122 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier add base cache' 2024-09-21T18:07:02.138 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:02.138 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:02.156 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:02.157 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:02.175 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:02.175 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:02.192 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:02.187+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:02.196 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:02.218 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:02.218 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:02.239 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:02.239 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:02.290 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:02.290 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:02.290 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:02.294 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:02.290+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:02.305 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:02.395 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:02.391+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:02.477 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:02.477 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:02.477 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:02.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:02.496 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:02.491+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:02.597 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:02.593+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:02.609 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:02.610 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:02.610 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:02.610 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:02.624 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:02.698 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:02.695+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:02.742 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:02.742 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:02.742 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:02.742 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:02.742 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:02.757 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:02.799 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:02.795+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:02.875 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:02.875 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:02.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:02.875 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:02.875 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:02.888 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:02.900 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:02.895+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:03.001 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:02.999+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:03.020 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.020 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:03.020 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:03.020 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:03.100 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:07:03.100 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T18:07:03.100 INFO:teuthology.orchestra.run.smithi160.stderr:pool 'cache' is now (or already was) a tier of 'base' 2024-09-21T18:07:03.101 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:03.097+0000 7f8c3c0ce640 -1 osd.12 32 handle_osd_map injecting map crc failure 2024-09-21T18:07:03.121 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:03.119+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:03.126 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier cache-mode cache writeback' 2024-09-21T18:07:03.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T18:07:03.200 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.200 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:03.200 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:03.200 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:03.200 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.203 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:03.199+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:03.214 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:03.280 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:03.278+0000 7fd79a4fd640 -1 osd.1 32 handle_osd_map injecting map crc failure 2024-09-21T18:07:03.304 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:03.299+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:03.348 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:03.343+0000 7f66f041c640 -1 osd.7 32 handle_osd_map injecting map crc failure 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4347.0:1 2.3 e26/25 2:c4c92e5a:::rbd_trash:head v 26'1, mlcod=0'0)", 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:44.876483+0000", 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.506284400999998, 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101731356, 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876483+0000", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876483+0000", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876485+0000", 2024-09-21T18:07:03.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.513e-06 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876497+0000", 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2303000000000001e-05 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876535+0000", 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7605000000000003e-05 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.876571+0000", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6180999999999998e-05 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.976810+0000", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100238981 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.976875+0000", 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4518999999999997e-05 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.978106+0000", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001231717 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.978215+0000", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010853699999999999 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:44.984938+0000", 2024-09-21T18:07:03.388 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.397829449, 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052829699999999995, 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 2 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984938+0000", 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984938+0000", 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984942+0000", 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.963e-06 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984954+0000", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1533e-05 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984976+0000", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1712000000000002e-05 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.984994+0000", 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8355e-05 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.985162+0000", 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016829099999999999 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.985467+0000", 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000304443 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e26)", 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:44.986101+0000", 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.396667154999999, 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000426909, 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "reached pg", 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:03.393 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 3 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986101+0000", 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986101+0000", 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986104+0000", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5599999999999998e-06 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986115+0000", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1122e-05 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986145+0000", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9575000000000001e-05 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986161+0000", 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6016e-05 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986289+0000", 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012804599999999999 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:44.986528+0000", 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023859 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 17.493646841, 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10982987, 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 5 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889126+0000", 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6820000000000002e-06 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889143+0000", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7246000000000001e-05 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889177+0000", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3584999999999998e-05 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889196+0000", 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9658999999999999e-05 2024-09-21T18:07:03.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889338+0000", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000141996 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op must wait for map", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889348+0000", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0261999999999999e-05 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.895716+0000", 2024-09-21T18:07:03.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063680359999999997 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.895936+0000", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021955799999999999 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 12", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.896153+0000", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000216481 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.897514+0000", 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013610479999999999 2024-09-21T18:07:03.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998806+0000", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10129218500000001 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998885+0000", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9565000000000001e-05 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998951+0000", 2024-09-21T18:07:03.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5567000000000004e-05 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.4347.0:5 2.2 e27/25 ondisk, result = 0)", 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:45.998545+0000", 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 17.384223051999999, 2024-09-21T18:07:03.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00042278699999999999, 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998545+0000", 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998545+0000", 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998548+0000", 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4060000000000001e-06 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998559+0000", 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0465999999999999e-05 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998585+0000", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6216000000000001e-05 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998601+0000", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5832e-05 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998780+0000", 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017932000000000001 2024-09-21T18:07:03.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998797+0000", 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6456e-05 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998968+0000", 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017109100000000001 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.502736435999999, 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20442987400000001, 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 7 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880035+0000", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0629999999999999e-06 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880049+0000", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3253e-05 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880082+0000", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3408000000000003e-05 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880103+0000", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0628999999999998e-05 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.888592+0000", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0084889909999999996 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op must wait for map", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.888603+0000", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1369000000000001e-05 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981159+0000", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.092556303000000006 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:03.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981307+0000", 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000147319 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 12", 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981546+0000", 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000239368 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983148+0000", 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0016015700000000001 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084293+0000", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101145496 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084384+0000", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1259000000000002e-05 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084461+0000", 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6846000000000003e-05 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop_reply(client.4347.0:7 2.2 e27/25 ondisk, result = 0)", 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:46.983886+0000", 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.398881678999999, 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100588313, 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983886+0000", 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983886+0000", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983895+0000", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3529999999999995e-06 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983905+0000", 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0869e-05 2024-09-21T18:07:03.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983927+0000", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1472e-05 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084077+0000", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150183 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084269+0000", 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019232 2024-09-21T18:07:03.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084284+0000", 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4326e-05 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084475+0000", 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00019079000000000001 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:03.417 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.419 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:03.419 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:03.417+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:03.520 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:03.517+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:03.588 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.588 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:03.589 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:03.589 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:03.649 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:03.647+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:03.722 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:03.719+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:03.726 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.726 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:03.726 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:03.726 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:03.726 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:03.823 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:03.817+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:03.899 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:03.900 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:03.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:03.900 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:07:03.900 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:03.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:03.958 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:03.955+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:04.024 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:04.021+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:04.029 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:04.029 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:04.029 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:04.029 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:04.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:04.113 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T18:07:04.116 INFO:teuthology.orchestra.run.smithi160.stderr:set cache-mode for pool 'cache' to writeback 2024-09-21T18:07:04.125 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:04.119+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:04.140 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd tier set-overlay base cache' 2024-09-21T18:07:04.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T18:07:04.193 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:04.193 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:04.193 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:04.193 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:04.193 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:04.205 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:04.229 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:04.227+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:04.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:04.312 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:04.313 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:04.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:04.313 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:07:04.313 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:04.327 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:04.328 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:04.325+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:04.414 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:04.414 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:04.428 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:04.425+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:04.434 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:04.434 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:04.447 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:04.443+0000 7f6a3cca9640 -1 osd.3 34 handle_osd_map injecting map crc failure 2024-09-21T18:07:04.451 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:04.451 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:04.451 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:04.455 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:04.455 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:04.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:04.475 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:04.491 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:04.511 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:04.511 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:04.529 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:04.527+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:04.531 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:04.531 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:04.549 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:04.549 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:04.565 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:04.586 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:04.586 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:04.588 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:04.588 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:04.588 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:04.588 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:04.588 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:04.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:04.604 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:04.604 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:04.624 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:04.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:04.631 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:04.627+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:04.641 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:04.641 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:04.662 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:04.681 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:04.703 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:04.703 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:04.762 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:04.761+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:04.769 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:04.769 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:04.783 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:04.832 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:04.827+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:04.914 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:04.914 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:04.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:04.933 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:04.929+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:05.034 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:05.031+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:05.047 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.048 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.061 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:05.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:07:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T18:07:05.134 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:05.129+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:05.135 INFO:teuthology.orchestra.run.smithi160.stderr:overlay for 'base' is now (or already was) 'cache' 2024-09-21T18:07:05.156 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_type bloom' 2024-09-21T18:07:05.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T18:07:05.232 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.232 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.236 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:05.231+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:05.247 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:05.337 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:05.331+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:05.408 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.408 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.423 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:05.438 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:05.433+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:05.538 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:05.533+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:05.590 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.590 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.604 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:05.649 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:05.647+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:05.722 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.722 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:05.740 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:05.737+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:05.841 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:05.837+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:05.901 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:05.901 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:05.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:05.942 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:05.937+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:05.984 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:05.981+0000 7f8c3a0ca640 -1 osd.12 35 handle_osd_map injecting map crc failure 2024-09-21T18:07:06.033 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.033 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.042 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:06.037+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:06.045 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:07:06.140 INFO:teuthology.orchestra.run.smithi160.stderr:set pool 4 hit_set_type to bloom 2024-09-21T18:07:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T18:07:06.143 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:06.139+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:06.160 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_count 8' 2024-09-21T18:07:06.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T18:07:06.168 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.168 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.182 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:06.258 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:06.257+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:06.345 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:06.339+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:06.345 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.345 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:06.445 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:06.441+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:06.484 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:06.479+0000 7f6a3cca9640 -1 osd.3 36 handle_osd_map injecting map crc failure 2024-09-21T18:07:06.484 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:06.481+0000 7f094d1de640 -1 osd.0 36 handle_osd_map injecting map crc failure 2024-09-21T18:07:06.485 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:06.481+0000 7f4976023640 -1 osd.5 36 handle_osd_map injecting map crc failure 2024-09-21T18:07:06.523 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.523 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.537 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:06.546 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:06.545+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:06.647 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:06.645+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:06.661 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.661 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:06.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:06.747 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:06.745+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:06.806 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:06.806 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:06.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:06.848 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:06.845+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:06.861 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:06.881 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:06.881 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:06.903 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:06.903 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:06.926 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:06.927 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:06.938 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:06.939 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:06.939 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:06.939 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:06.939 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:06.943 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:06.943 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:06.949 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:06.947+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:06.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:06.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:06.984 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:06.985 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:07.005 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:07.023 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:07.023 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:07.044 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:07.051 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:07.047+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:07.066 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:07.084 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.085 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:07.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:07.085 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:07.085 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.087 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:07.087 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:07.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:07.107 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:07.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:07.130 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:07.130 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T18:07:07.141 INFO:teuthology.orchestra.run.smithi160.stderr:set pool 4 hit_set_count to 8 2024-09-21T18:07:07.151 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:07.147+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:07.152 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:07.152 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:07.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-21T18:07:07.165 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache hit_set_period 3600' 2024-09-21T18:07:07.172 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:07.245 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.245 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:07.245 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:07.245 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.253 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:07.249+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:07.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:07.354 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:07.353+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:07.418 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.418 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:07.418 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:07.418 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:07.418 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:07.478 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:07.477+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:07.556 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:07.551+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:07.593 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.593 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:07.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:07.594 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:07.594 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:07.656 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:07.653+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:07.712 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:07.707+0000 7f66f041c640 -1 osd.7 37 handle_osd_map injecting map crc failure 2024-09-21T18:07:07.732 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.732 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:07.732 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:07.732 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:07.757 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:07.751+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:07.858 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:07.857+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:07.906 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:07.906 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:07.906 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:07.907 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:07.907 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:07.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:07.959 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:07.955+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:08.060 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:08.055+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:08.083 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:08.083 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:08.084 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:08.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:08.150 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.8 to 1 (10000) 2024-09-21T18:07:08.150 INFO:teuthology.orchestra.run.smithi160.stderr:set pool 4 hit_set_period to 3600 2024-09-21T18:07:08.160 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:08.157+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:08.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-21T18:07:08.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:07:08.173 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache min_read_recency_for_promote 0' 2024-09-21T18:07:08.224 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:08.224 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:08.224 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:08.224 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:08.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:08.286 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:08.285+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:08.288 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:08.283+0000 7f9bb588f640 -1 osd.10 38 handle_osd_map injecting map crc failure 2024-09-21T18:07:08.362 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:08.359+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:08.404 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:08.405 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:08.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:08.463 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:08.459+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.640403361000001, 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:07:08.537 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:07:08.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 21.555010972000002, 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:08.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:08.546 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:08.546 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:08.550 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:08.564 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:08.559+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:08.666 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:08.663+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:08.684 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:08.684 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:08.684 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:08.685 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:08.699 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:08.767 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:08.763+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:08.825 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:08.825 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:08.825 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:08.825 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:08.825 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:08.839 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:08.868 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:08.863+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:08.966 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:08.966 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:08.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:08.967 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:08.967 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:08.968 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:08.963+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:08.981 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:09.069 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:09.065+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:09.109 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:09.109 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:09.109 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:09.109 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:09.123 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:07:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.9 to 1 (10000) 2024-09-21T18:07:09.154 INFO:teuthology.orchestra.run.smithi160.stderr:set pool 4 min_read_recency_for_promote to 0 2024-09-21T18:07:09.170 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:09.169+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:09.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph osd pool set cache min_write_recency_for_promote 0' 2024-09-21T18:07:09.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-21T18:07:09.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:09.272 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:09.269+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:09.294 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:09.295 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:09.295 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:09.295 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:09.295 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:09.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:09.368 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:09.372 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:09.367+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:09.385 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:09.386 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:09.406 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:09.406 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:09.426 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:21 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'11, mlcod=25'9)", 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.138241101999999, 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20317162799999999, 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297877+0000", 2024-09-21T18:07:09.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0130000000000002e-06 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298015+0000", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013840599999999999 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298043+0000", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7702999999999999e-05 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298059+0000", 2024-09-21T18:07:09.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5512e-05 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298175+0000", 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011620699999999999 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298212+0000", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7409e-05 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500956+0000", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20274424699999999 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.501046+0000", 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9130999999999998e-05 2024-09-21T18:07:09.443 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.137063104999999, 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:07:09.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:07:09.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.135849046000001, 2024-09-21T18:07:09.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.452 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.134498126, 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.453 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:07:09.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:25 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'15, mlcod=25'9)", 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.401872+0000", 2024-09-21T18:07:09.458 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.034242975000002, 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.201402625, 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401872+0000", 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401872+0000", 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401874+0000", 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401887+0000", 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3023000000000001e-05 2024-09-21T18:07:09.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401903+0000", 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5472999999999999e-05 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401913+0000", 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06e-05 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601811+0000", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19989710299999999 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601841+0000", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0156000000000001e-05 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.603204+0000", 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001363792 2024-09-21T18:07:09.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.603275+0000", 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0208000000000003e-05 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:56 1.0 e26/23 1:30306672:devicehealth::main.db.0000000000000000:head v 26'38, mlcod=26'33)", 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:46.514435+0000", 2024-09-21T18:07:09.463 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 22.921679826999998, 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20197416800000001, 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514435+0000", 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514435+0000", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514439+0000", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4800000000000001e-06 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514502+0000", 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3814999999999998e-05 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514530+0000", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7642000000000001e-05 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.514544+0000", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3570000000000001e-05 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.714909+0000", 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20036552699999999 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.714942+0000", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2345999999999998e-05 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.716335+0000", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001393554 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:46.716409+0000", 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4233999999999994e-05 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.467 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:76 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'54, mlcod=28'47)", 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:47.612086+0000", 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 21.824029203999999, 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20003839200000001, 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612086+0000", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612086+0000", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612089+0000", 2024-09-21T18:07:09.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.994e-06 2024-09-21T18:07:09.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612101+0000", 2024-09-21T18:07:09.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2466000000000001e-05 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612128+0000", 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6276000000000001e-05 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.612143+0000", 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5469e-05 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.810740+0000", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19859695899999999 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.810772+0000", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2382999999999998e-05 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.812039+0000", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012664200000000001 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:47.812124+0000", 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5425000000000003e-05 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:93 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'64)", 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.235875214, 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203107066, 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200243+0000", 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.01e-06 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200311+0000", 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8039000000000004e-05 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200338+0000", 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6934999999999999e-05 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200352+0000", 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.377e-05 2024-09-21T18:07:09.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.299647+0000", 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099295849000000005 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.299678+0000", 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0565000000000003e-05 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403249+0000", 2024-09-21T18:07:09.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10357140400000001 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403347+0000", 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7493999999999999e-05 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:94 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'64)", 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.234653589000001, 2024-09-21T18:07:09.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20356700799999999, 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:09.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201464+0000", 2024-09-21T18:07:09.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.033e-06 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201477+0000", 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2743e-05 2024-09-21T18:07:09.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201504+0000", 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6435999999999999e-05 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201517+0000", 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3336000000000001e-05 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300456+0000", 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098939036999999994 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300486+0000", 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0295999999999999e-05 2024-09-21T18:07:09.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.404948+0000", 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104461295 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405028+0000", 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0832000000000001e-05 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.489 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:95 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'69, mlcod=28'64)", 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.233617057, 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20260623799999999, 2024-09-21T18:07:09.490 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:09.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202501+0000", 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202512+0000", 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1590000000000001e-05 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202533+0000", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0665999999999998e-05 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202546+0000", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2806000000000001e-05 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300947+0000", 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098401269999999999 2024-09-21T18:07:09.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300977+0000", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0079000000000001e-05 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405035+0000", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10405750800000001 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405104+0000", 2024-09-21T18:07:09.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9555999999999996e-05 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:96 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'70, mlcod=28'64)", 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.203593+0000", 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.232522107000001, 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20157095899999999, 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203593+0000", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203593+0000", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203601+0000", 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6639999999999998e-06 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203713+0000", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111927 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203739+0000", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6733e-05 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.203752+0000", 2024-09-21T18:07:09.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3196e-05 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.401630+0000", 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.197877523 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.401665+0000", 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4956000000000002e-05 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405113+0000", 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0034480610000000001 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405164+0000", 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0899000000000002e-05 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.500 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:97 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'71, mlcod=28'64)", 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.204813+0000", 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.231301605999999, 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.201767421, 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204813+0000", 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204813+0000", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204816+0000", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204829+0000", 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2717e-05 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204855+0000", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5939e-05 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.204868+0000", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3377000000000001e-05 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.402820+0000", 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.197951191 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.402855+0000", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5666e-05 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.406495+0000", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0036397230000000001 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.406581+0000", 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5808000000000002e-05 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:98 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'72, mlcod=28'64)", 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.230142319999999, 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203347474, 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205976+0000", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.033e-06 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205989+0000", 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2847000000000001e-05 2024-09-21T18:07:09.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.206015+0000", 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6265999999999999e-05 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.206028+0000", 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3199e-05 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403443+0000", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.197415325 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403473+0000", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9543000000000002e-05 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409245+0000", 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057717460000000003 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409320+0000", 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5514999999999998e-05 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:99 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'73, mlcod=28'64)", 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.207034+0000", 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.229081329, 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20232905200000001, 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207034+0000", 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207034+0000", 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207041+0000", 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5240000000000003e-06 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207158+0000", 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117007 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207186+0000", 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.207199+0000", 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3263e-05 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403966+0000", 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19676740000000001 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403999+0000", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2388999999999998e-05 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409324+0000", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053258239999999998 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409363+0000", 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8309000000000002e-05 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:100 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'74, mlcod=28'64)", 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.208241+0000", 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 20.227873567, 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20115551600000001, 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208241+0000", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208241+0000", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208244+0000", 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:07:09.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208257+0000", 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2785999999999999e-05 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208284+0000", 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6653e-05 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.208297+0000", 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3379e-05 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405219+0000", 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19692162399999999 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405257+0000", 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7605000000000003e-05 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409365+0000", 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0041080960000000003 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409397+0000", 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2373000000000003e-05 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.846917488999999, 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:07:09.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:07:09.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:117 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'87, mlcod=28'80)", 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.746130562000001, 2024-09-21T18:07:09.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20472252399999999, 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689988+0000", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2600000000000001e-06 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690085+0000", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7133999999999995e-05 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690105+0000", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9826e-05 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690118+0000", 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.314e-05 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.790369+0000", 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100250926 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.790444+0000", 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5288000000000002e-05 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894614+0000", 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104170426 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894707+0000", 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2523999999999997e-05 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:118 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'88, mlcod=28'80)", 2024-09-21T18:07:09.527 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.745966699, 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20465380899999999, 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690150+0000", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.72e-06 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690160+0000", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0406e-05 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690177+0000", 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6296000000000001e-05 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690187+0000", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.039e-05 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.891400+0000", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20121330400000001 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.891457+0000", 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6508999999999997e-05 2024-09-21T18:07:09.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894718+0000", 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032612829999999998 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894802+0000", 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3900999999999995e-05 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.531 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:119 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'89, mlcod=28'80)", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.745910924, 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.204678522, 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:09.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690205+0000", 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690246+0000", 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0209000000000001e-05 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690261+0000", 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5596000000000001e-05 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690269+0000", 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1529999999999997e-06 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892340+0000", 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20207024700000001 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892378+0000", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8173000000000002e-05 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894808+0000", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024301629999999999 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894883+0000", 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4740999999999995e-05 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.535 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:120 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'90, mlcod=28'80)", 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.745829195999999, 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20471718799999999, 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:09.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690287+0000", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0640000000000001e-06 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690344+0000", 2024-09-21T18:07:09.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7000999999999999e-05 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690385+0000", 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1526000000000002e-05 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690394+0000", 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2060000000000007e-06 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:09.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892905+0000", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20251159399999999 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892933+0000", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7572e-05 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894888+0000", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019556840000000001 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.895003+0000", 2024-09-21T18:07:09.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114541 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:09.540 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:09.541 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:09.541 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:09.542 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:09.543 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:09.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:09.543 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:09.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:09.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:09.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:09.545 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:09.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:09.545 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:09.539+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:09.564 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:09.565 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:09.584 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:09.584 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:09.601 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:09.602 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:09.624 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:09.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:09.642 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:09.639+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:09.644 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:09.644 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:09.665 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:09.666 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:09.677 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:09.677 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:09.693 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:09.743 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:09.741+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:09.819 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:09.820 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:09.834 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:09.883 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:09.881+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:09.945 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:09.941+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:09.963 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:09.963 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:09.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:09.983 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:09.981+0000 7ff409995640 -1 osd.4 39 handle_osd_map injecting map crc failure 2024-09-21T18:07:09.983 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:09.979+0000 7fb25ec0a640 -1 osd.11 39 handle_osd_map injecting map crc failure 2024-09-21T18:07:10.046 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:10.043+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:10.139 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:10.140 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:10.147 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:10.145+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:10.154 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:10.163 INFO:teuthology.orchestra.run.smithi160.stderr:set pool 4 min_write_recency_for_promote to 0 2024-09-21T18:07:10.168 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.10 to 1 (10000) 2024-09-21T18:07:10.186 INFO:teuthology.run_tasks:Running task rados... 2024-09-21T18:07:10.202 INFO:tasks.rados:Beginning rados... 2024-09-21T18:07:10.202 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']} 2024-09-21T18:07:10.202 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-rf-qprio-w-qa'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async', 'osd debug inject dispatch delay duration': 0.1, 'osd debug inject dispatch delay probability': 0.1, 'osd_async_recovery_min_cost': 1, 'osd_object_clean_region_max_num_intervals': 1000, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd beacon report interval': 30, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd inject bad map crc probability': 0.1, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd max scrubs': 5, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd_max_pg_log_entries': 3000, 'osd_min_pg_log_entries': 3000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', 'failed to encode map', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'must scrub before tier agent can activate', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rf-qprio-w-qa', 'sha1': 'c69a6b39fef6dd04706886608793361ef510a986'}} 2024-09-21T18:07:10.202 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 500, 'op_weights': {'cache_evict': 50, 'cache_flush': 50, 'cache_try_flush': 50, 'copy_from': 50, 'delete': 50, 'read': 100, 'write': 100}, 'ops': 4000, 'pools': ['base']} 2024-09-21T18:07:10.202 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-21T18:07:10.291 INFO:tasks.rados:joining rados 2024-09-21T18:07:10.292 INFO:tasks.rados:clients are ['client.0'] 2024-09-21T18:07:10.292 INFO:tasks.rados:starting run 0 out of 1 2024-09-21T18:07:10.292 DEBUG:teuthology.orchestra.run.smithi160:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op copy_from 50 --op cache_flush 50 --op cache_try_flush 50 --op cache_evict 50 --op write_excl 50 --pool base 2024-09-21T18:07:10.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-21T18:07:10.295 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:10.291+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:10.313 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight read -> 100 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight write -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight delete -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight copy_from -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight cache_flush -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight cache_try_flush -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight cache_evict -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:adding op weight write_excl -> 50 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:ceph version 19.3.0-5073-gc69a6b39 (c69a6b39fef6dd04706886608793361ef510a986) squid (dev) 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout:Configuration: 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout: Number of operations: 4000 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout: Number of objects: 500 2024-09-21T18:07:10.314 INFO:tasks.rados.rados.0.smithi160.stdout: Max in flight operations: 16 2024-09-21T18:07:10.315 INFO:tasks.rados.rados.0.smithi160.stdout: Object size (in bytes): 4000000 2024-09-21T18:07:10.315 INFO:tasks.rados.rados.0.smithi160.stdout: Write stride min: 400000 2024-09-21T18:07:10.315 INFO:tasks.rados.rados.0.smithi160.stdout: Write stride max: 800000 2024-09-21T18:07:10.395 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:10.391+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:10.425 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:10.425 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:10.439 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:10.495 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:10.493+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:10.567 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:10.567 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:10.581 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:10.597 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:10.593+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:10.698 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:10.697+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:10.712 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:10.712 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:10.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:10.799 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:10.795+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:10.849 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:10.849 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:10.845+0000 7ff409995640 -1 osd.4 40 handle_osd_map injecting map crc failure 2024-09-21T18:07:10.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:10.900 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:10.897+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:11.000 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:10.995+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:11.029 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.029 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:11.134 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:11.131+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:11.180 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.11 to 1 (10000) 2024-09-21T18:07:11.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:07:11.186 INFO:tasks.rados.rados.0.smithi160.stdout:1: write initial oid 1 2024-09-21T18:07:11.186 INFO:tasks.rados.rados.0.smithi160.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-21T18:07:11.193 INFO:tasks.rados.rados.0.smithi160.stdout:1: writing smithi16030472-1 from 416807 to 892056 tid 1 2024-09-21T18:07:11.201 INFO:tasks.rados.rados.0.smithi160.stdout:1: writing smithi16030472-1 from 1542129 to 2016807 tid 2 2024-09-21T18:07:11.202 INFO:tasks.rados.rados.0.smithi160.stdout:2: write initial oid 2 2024-09-21T18:07:11.202 INFO:tasks.rados.rados.0.smithi160.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-21T18:07:11.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-21T18:07:11.209 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:11.205+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:11.209 INFO:tasks.rados.rados.0.smithi160.stdout:2: writing smithi16030472-2 from 416807 to 892056 tid 1 2024-09-21T18:07:11.210 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.211 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.216 INFO:tasks.rados.rados.0.smithi160.stdout:2: writing smithi16030472-2 from 1542129 to 2016807 tid 2 2024-09-21T18:07:11.216 INFO:tasks.rados.rados.0.smithi160.stdout:3: write initial oid 3 2024-09-21T18:07:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-21T18:07:11.224 INFO:tasks.rados.rados.0.smithi160.stdout:3: writing smithi16030472-3 from 433614 to 984112 tid 1 2024-09-21T18:07:11.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:11.231 INFO:tasks.rados.rados.0.smithi160.stdout:3: writing smithi16030472-3 from 1600611 to 2033614 tid 2 2024-09-21T18:07:11.232 INFO:tasks.rados.rados.0.smithi160.stdout:4: write initial oid 4 2024-09-21T18:07:11.232 INFO:tasks.rados.rados.0.smithi160.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-21T18:07:11.240 INFO:tasks.rados.rados.0.smithi160.stdout:4: writing smithi16030472-4 from 450421 to 1076168 tid 1 2024-09-21T18:07:11.247 INFO:tasks.rados.rados.0.smithi160.stdout:4: writing smithi16030472-4 from 1659093 to 2050421 tid 2 2024-09-21T18:07:11.247 INFO:tasks.rados.rados.0.smithi160.stdout:5: write initial oid 5 2024-09-21T18:07:11.247 INFO:tasks.rados.rados.0.smithi160.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-21T18:07:11.255 INFO:tasks.rados.rados.0.smithi160.stdout:5: writing smithi16030472-5 from 467228 to 1168224 tid 1 2024-09-21T18:07:11.261 INFO:tasks.rados.rados.0.smithi160.stdout:5: writing smithi16030472-5 from 1717575 to 2067228 tid 2 2024-09-21T18:07:11.261 INFO:tasks.rados.rados.0.smithi160.stdout:6: write initial oid 6 2024-09-21T18:07:11.261 INFO:tasks.rados.rados.0.smithi160.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-21T18:07:11.268 INFO:tasks.rados.rados.0.smithi160.stdout:6: writing smithi16030472-6 from 484035 to 1260280 tid 1 2024-09-21T18:07:11.272 INFO:tasks.rados.rados.0.smithi160.stdout:6: writing smithi16030472-6 from 2059704 to 2084035 tid 2 2024-09-21T18:07:11.272 INFO:tasks.rados.rados.0.smithi160.stdout:7: write initial oid 7 2024-09-21T18:07:11.273 INFO:tasks.rados.rados.0.smithi160.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-21T18:07:11.278 INFO:tasks.rados.rados.0.smithi160.stdout:7: writing smithi16030472-7 from 500842 to 952336 tid 1 2024-09-21T18:07:11.285 INFO:tasks.rados.rados.0.smithi160.stdout:7: writing smithi16030472-7 from 1718186 to 2100842 tid 2 2024-09-21T18:07:11.285 INFO:tasks.rados.rados.0.smithi160.stdout:8: write initial oid 8 2024-09-21T18:07:11.285 INFO:tasks.rados.rados.0.smithi160.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-21T18:07:11.290 INFO:tasks.rados.rados.0.smithi160.stdout:8: writing smithi16030472-8 from 517649 to 1044392 tid 1 2024-09-21T18:07:11.297 INFO:tasks.rados.rados.0.smithi160.stdout:8: writing smithi16030472-8 from 1776668 to 2117649 tid 2 2024-09-21T18:07:11.297 INFO:tasks.rados.rados.0.smithi160.stdout:9: write initial oid 9 2024-09-21T18:07:11.297 INFO:tasks.rados.rados.0.smithi160.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-21T18:07:11.304 INFO:tasks.rados.rados.0.smithi160.stdout:9: writing smithi16030472-9 from 534456 to 1252801 tid 1 2024-09-21T18:07:11.309 INFO:tasks.rados.rados.0.smithi160.stdout:9: writing smithi16030472-9 from 1951503 to 2134456 tid 2 2024-09-21T18:07:11.309 INFO:tasks.rados.rados.0.smithi160.stdout:10: write initial oid 10 2024-09-21T18:07:11.309 INFO:tasks.rados.rados.0.smithi160.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-21T18:07:11.317 INFO:tasks.rados.rados.0.smithi160.stdout:10: writing smithi16030472-10 from 551263 to 1344857 tid 1 2024-09-21T18:07:11.317 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:11.315+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:11.321 INFO:tasks.rados.rados.0.smithi160.stdout:10: writing smithi16030472-10 from 1893632 to 2151263 tid 2 2024-09-21T18:07:11.321 INFO:tasks.rados.rados.0.smithi160.stdout:11: write initial oid 11 2024-09-21T18:07:11.322 INFO:tasks.rados.rados.0.smithi160.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-21T18:07:11.327 INFO:tasks.rados.rados.0.smithi160.stdout:11: writing smithi16030472-11 from 568070 to 1036913 tid 1 2024-09-21T18:07:11.334 INFO:tasks.rados.rados.0.smithi160.stdout:11: writing smithi16030472-11 from 1552114 to 2168070 tid 2 2024-09-21T18:07:11.334 INFO:tasks.rados.rados.0.smithi160.stdout:12: write initial oid 12 2024-09-21T18:07:11.334 INFO:tasks.rados.rados.0.smithi160.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-21T18:07:11.340 INFO:tasks.rados.rados.0.smithi160.stdout:12: writing smithi16030472-12 from 584877 to 1128969 tid 1 2024-09-21T18:07:11.346 INFO:tasks.rados.rados.0.smithi160.stdout:12: writing smithi16030472-12 from 1610596 to 2172599 tid 2 2024-09-21T18:07:11.346 INFO:tasks.rados.rados.0.smithi160.stdout:13: write initial oid 13 2024-09-21T18:07:11.347 INFO:tasks.rados.rados.0.smithi160.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-21T18:07:11.353 INFO:tasks.rados.rados.0.smithi160.stdout:13: writing smithi16030472-13 from 601684 to 1221025 tid 1 2024-09-21T18:07:11.359 INFO:tasks.rados.rados.0.smithi160.stdout:13: writing smithi16030472-13 from 1669078 to 2201684 tid 2 2024-09-21T18:07:11.359 INFO:tasks.rados.rados.0.smithi160.stdout:14: write initial oid 14 2024-09-21T18:07:11.359 INFO:tasks.rados.rados.0.smithi160.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-21T18:07:11.366 INFO:tasks.rados.rados.0.smithi160.stdout:14: writing smithi16030472-14 from 618491 to 1313081 tid 1 2024-09-21T18:07:11.372 INFO:tasks.rados.rados.0.smithi160.stdout:14: writing smithi16030472-14 from 2011207 to 2218491 tid 2 2024-09-21T18:07:11.372 INFO:tasks.rados.rados.0.smithi160.stdout:15: write initial oid 15 2024-09-21T18:07:11.372 INFO:tasks.rados.rados.0.smithi160.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-21T18:07:11.380 INFO:tasks.rados.rados.0.smithi160.stdout:15: writing smithi16030472-15 from 635298 to 1405137 tid 1 2024-09-21T18:07:11.384 INFO:tasks.rados.rados.0.smithi160.stdout:15: writing smithi16030472-15 from 2069689 to 2235298 tid 2 2024-09-21T18:07:11.384 INFO:tasks.rados.rados.0.smithi160.stdout:16: write initial oid 16 2024-09-21T18:07:11.385 INFO:tasks.rados.rados.0.smithi160.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-21T18:07:11.391 INFO:tasks.rados.rados.0.smithi160.stdout:16: writing smithi16030472-16 from 652105 to 1097193 tid 1 2024-09-21T18:07:11.392 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.392 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.397 INFO:tasks.rados.rados.0.smithi160.stdout:16: writing smithi16030472-16 from 1728171 to 2181159 tid 2 2024-09-21T18:07:11.397 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:11.404 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:11.399+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:11.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:11.499 INFO:tasks.rados.rados.0.smithi160.stdout:1: finishing write tid 1 to smithi16030472-1 2024-09-21T18:07:11.505 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:11.501+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:11.511 INFO:tasks.rados.rados.0.smithi160.stdout:6: finishing write tid 1 to smithi16030472-6 2024-09-21T18:07:11.577 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.577 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:11.602 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:11.599+0000 7f3ce267c640 -1 osd.6 40 handle_osd_map injecting map crc failure 2024-09-21T18:07:11.605 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:11.603+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:11.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:11.703 INFO:tasks.rados.rados.0.smithi160.stdout:6: finishing write tid 2 to smithi16030472-6 2024-09-21T18:07:11.706 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:11.701+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:11.711 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.711 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:11.741 INFO:tasks.rados.rados.0.smithi160.stdout:4: finishing write tid 1 to smithi16030472-4 2024-09-21T18:07:11.742 INFO:tasks.rados.rados.0.smithi160.stdout:7: finishing write tid 1 to smithi16030472-7 2024-09-21T18:07:11.753 INFO:tasks.rados.rados.0.smithi160.stdout:7: finishing write tid 2 to smithi16030472-7 2024-09-21T18:07:11.753 INFO:tasks.rados.rados.0.smithi160.stdout:7: finishing write tid 3 to smithi16030472-7 2024-09-21T18:07:11.754 INFO:tasks.rados.rados.0.smithi160.stdout:7: finishing write tid 4 to smithi16030472-7 2024-09-21T18:07:11.754 INFO:tasks.rados.rados.0.smithi160.stdout:7: oid 7 updating version 0 to 3 2024-09-21T18:07:11.754 INFO:tasks.rados.rados.0.smithi160.stdout:7: oid 7 version 3 is already newer than 2 2024-09-21T18:07:11.754 INFO:tasks.rados.rados.0.smithi160.stdout:7: oid 7 updating version 3 to 4 2024-09-21T18:07:11.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 7 v 4 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-21T18:07:11.755 INFO:tasks.rados.rados.0.smithi160.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-21T18:07:11.755 INFO:tasks.rados.rados.0.smithi160.stdout:7: done (15 left) 2024-09-21T18:07:11.755 INFO:tasks.rados.rados.0.smithi160.stdout:17: write initial oid 17 2024-09-21T18:07:11.755 INFO:tasks.rados.rados.0.smithi160.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-21T18:07:11.761 INFO:tasks.rados.rados.0.smithi160.stdout:17: writing smithi16030472-17 from 668912 to 1305602 tid 1 2024-09-21T18:07:11.767 INFO:tasks.rados.rados.0.smithi160.stdout:17: writing smithi16030472-17 from 1903006 to 2268912 tid 2 2024-09-21T18:07:11.767 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:11.804 INFO:tasks.rados.rados.0.smithi160.stdout:6: finishing write tid 3 to smithi16030472-6 2024-09-21T18:07:11.804 INFO:tasks.rados.rados.0.smithi160.stdout:6: finishing write tid 4 to smithi16030472-6 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:6: oid 6 updating version 0 to 3 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:6: oid 6 version 3 is already newer than 2 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:6: oid 6 updating version 3 to 4 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 4 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:1: finishing write tid 2 to smithi16030472-1 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:6: done (15 left) 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:18: write initial oid 18 2024-09-21T18:07:11.805 INFO:tasks.rados.rados.0.smithi160.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-21T18:07:11.807 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:11.805+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:11.812 INFO:tasks.rados.rados.0.smithi160.stdout:18: writing smithi16030472-18 from 685719 to 1397658 tid 1 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:18: writing smithi16030472-18 from 1845135 to 2285719 tid 2 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: finishing write tid 3 to smithi16030472-1 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: finishing write tid 4 to smithi16030472-1 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: oid 1 updating version 0 to 5 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: oid 1 updating version 5 to 6 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: oid 1 version 6 is already newer than 1 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 1 v 6 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-21T18:07:11.818 INFO:tasks.rados.rados.0.smithi160.stdout:1: done (15 left) 2024-09-21T18:07:11.819 INFO:tasks.rados.rados.0.smithi160.stdout:19: write initial oid 19 2024-09-21T18:07:11.819 INFO:tasks.rados.rados.0.smithi160.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-21T18:07:11.828 INFO:tasks.rados.rados.0.smithi160.stdout:19: writing smithi16030472-19 from 702526 to 1489714 tid 1 2024-09-21T18:07:11.832 INFO:tasks.rados.rados.0.smithi160.stdout:19: writing smithi16030472-19 from 1903617 to 2302526 tid 2 2024-09-21T18:07:11.832 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:11.838 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:11.838 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:11.859 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:11.869 INFO:tasks.rados.rados.0.smithi160.stdout:2: finishing write tid 1 to smithi16030472-2 2024-09-21T18:07:11.869 INFO:tasks.rados.rados.0.smithi160.stdout:3: finishing write tid 1 to smithi16030472-3 2024-09-21T18:07:11.869 INFO:tasks.rados.rados.0.smithi160.stdout:8: finishing write tid 1 to smithi16030472-8 2024-09-21T18:07:11.881 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:11.881 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:11.896 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:11.897 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:11.903 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:11.909 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:11.907+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:11.911 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:11.912 INFO:tasks.rados.rados.0.smithi160.stdout:9: finishing write tid 1 to smithi16030472-9 2024-09-21T18:07:11.924 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:11.930 INFO:tasks.rados.rados.0.smithi160.stdout:14: finishing write tid 1 to smithi16030472-14 2024-09-21T18:07:11.947 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:11.947 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:11.966 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:11.967 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:11.980 INFO:tasks.rados.rados.0.smithi160.stdout:16: finishing write tid 1 to smithi16030472-16 2024-09-21T18:07:11.980 INFO:tasks.rados.rados.0.smithi160.stdout:4: finishing write tid 2 to smithi16030472-4 2024-09-21T18:07:11.984 INFO:tasks.rados.rados.0.smithi160.stdout:18: finishing write tid 1 to smithi16030472-18 2024-09-21T18:07:11.990 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:11.991 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:12.004 INFO:tasks.rados.rados.0.smithi160.stdout:5: finishing write tid 1 to smithi16030472-5 2024-09-21T18:07:12.010 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:12.007+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:12.010 INFO:tasks.rados.rados.0.smithi160.stdout:2: finishing write tid 2 to smithi16030472-2 2024-09-21T18:07:12.011 INFO:tasks.rados.rados.0.smithi160.stdout:2: finishing write tid 3 to smithi16030472-2 2024-09-21T18:07:12.011 INFO:tasks.rados.rados.0.smithi160.stdout:2: finishing write tid 4 to smithi16030472-2 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:2: oid 2 updating version 0 to 1 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:2: oid 2 updating version 1 to 2 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:2: oid 2 updating version 2 to 3 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:2: done (15 left) 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:20: write initial oid 20 2024-09-21T18:07:12.012 INFO:tasks.rados.rados.0.smithi160.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-21T18:07:12.012 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:12.013 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:12.020 INFO:tasks.rados.rados.0.smithi160.stdout:20: writing smithi16030472-20 from 719333 to 1181770 tid 1 2024-09-21T18:07:12.027 INFO:tasks.rados.rados.0.smithi160.stdout:20: writing smithi16030472-20 from 1962099 to 2319333 tid 2 2024-09-21T18:07:12.028 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.028 INFO:tasks.rados.rados.0.smithi160.stdout:13: finishing write tid 1 to smithi16030472-13 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: finishing write tid 2 to smithi16030472-3 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: finishing write tid 3 to smithi16030472-3 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: finishing write tid 4 to smithi16030472-3 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: oid 3 updating version 0 to 1 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: oid 3 updating version 1 to 3 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: oid 3 updating version 3 to 4 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 3 v 4 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:3: done (15 left) 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:21: write initial oid 21 2024-09-21T18:07:12.038 INFO:tasks.rados.rados.0.smithi160.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-21T18:07:12.039 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:12.039 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:12.044 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:12.044 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:12.045 INFO:tasks.rados.rados.0.smithi160.stdout:21: writing smithi16030472-21 from 736140 to 1273826 tid 1 2024-09-21T18:07:12.051 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:12.052 INFO:tasks.rados.rados.0.smithi160.stdout:21: writing smithi16030472-21 from 2020581 to 2336140 tid 2 2024-09-21T18:07:12.052 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.092 INFO:tasks.rados.rados.0.smithi160.stdout:4: finishing write tid 3 to smithi16030472-4 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:4: finishing write tid 4 to smithi16030472-4 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:4: oid 4 updating version 0 to 1 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:4: oid 4 updating version 1 to 5 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:4: oid 4 updating version 5 to 6 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 4 v 6 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-21T18:07:12.093 INFO:tasks.rados.rados.0.smithi160.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-21T18:07:12.094 INFO:tasks.rados.rados.0.smithi160.stdout:4: done (15 left) 2024-09-21T18:07:12.094 INFO:tasks.rados.rados.0.smithi160.stdout:22: write initial oid 22 2024-09-21T18:07:12.094 INFO:tasks.rados.rados.0.smithi160.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-21T18:07:12.105 INFO:tasks.rados.rados.0.smithi160.stdout:22: writing smithi16030472-22 from 752947 to 1365882 tid 1 2024-09-21T18:07:12.111 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:12.109+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:12.114 INFO:tasks.rados.rados.0.smithi160.stdout:22: writing smithi16030472-22 from 1962710 to 2352947 tid 2 2024-09-21T18:07:12.114 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.114 INFO:tasks.rados.rados.0.smithi160.stdout:10: finishing write tid 1 to smithi16030472-10 2024-09-21T18:07:12.118 INFO:tasks.rados.rados.0.smithi160.stdout:5: finishing write tid 2 to smithi16030472-5 2024-09-21T18:07:12.118 INFO:tasks.rados.rados.0.smithi160.stdout:5: finishing write tid 3 to smithi16030472-5 2024-09-21T18:07:12.129 INFO:tasks.rados.rados.0.smithi160.stdout:9: finishing write tid 2 to smithi16030472-9 2024-09-21T18:07:12.129 INFO:tasks.rados.rados.0.smithi160.stdout:9: finishing write tid 3 to smithi16030472-9 2024-09-21T18:07:12.130 INFO:tasks.rados.rados.0.smithi160.stdout:9: finishing write tid 4 to smithi16030472-9 2024-09-21T18:07:12.130 INFO:tasks.rados.rados.0.smithi160.stdout:9: oid 9 updating version 0 to 9 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:9: oid 9 version 9 is already newer than 8 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:9: oid 9 version 9 is already newer than 3 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 9 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:9: done (15 left) 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:23: write initial oid 23 2024-09-21T18:07:12.131 INFO:tasks.rados.rados.0.smithi160.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-21T18:07:12.143 INFO:tasks.rados.rados.0.smithi160.stdout:23: writing smithi16030472-23 from 769754 to 1457938 tid 1 2024-09-21T18:07:12.152 INFO:tasks.rados.rados.0.smithi160.stdout:23: writing smithi16030472-23 from 2021192 to 2369754 tid 2 2024-09-21T18:07:12.152 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.152 INFO:tasks.rados.rados.0.smithi160.stdout:19: finishing write tid 1 to smithi16030472-19 2024-09-21T18:07:12.152 INFO:tasks.rados.rados.0.smithi160.stdout:20: finishing write tid 1 to smithi16030472-20 2024-09-21T18:07:12.153 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:12.154 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:12.179 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:12.179 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:12.179 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:07:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.12 to 1 (10000) 2024-09-21T18:07:12.195 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:12.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-21T18:07:12.208 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:12.209 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: finishing write tid 4 to smithi16030472-5 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: oid 5 updating version 0 to 5 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: oid 5 version 5 is already newer than 2 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: oid 5 updating version 5 to 6 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 5 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:5: done (15 left) 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:24: write initial oid 24 2024-09-21T18:07:12.219 INFO:tasks.rados.rados.0.smithi160.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-21T18:07:12.231 INFO:tasks.rados.rados.0.smithi160.stdout:24: writing smithi16030472-24 from 786561 to 1266347 tid 1 2024-09-21T18:07:12.237 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:12.235+0000 7fb25ec0a640 -1 osd.11 42 handle_osd_map injecting map crc failure 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:24: writing smithi16030472-24 from 1796027 to 2263691 tid 2 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:11: finishing write tid 1 to smithi16030472-11 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:12: finishing write tid 1 to smithi16030472-12 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:10: finishing write tid 2 to smithi16030472-10 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:14: finishing write tid 2 to smithi16030472-14 2024-09-21T18:07:12.241 INFO:tasks.rados.rados.0.smithi160.stdout:13: finishing write tid 2 to smithi16030472-13 2024-09-21T18:07:12.246 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:12.243+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:12.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:12.313 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:12.307+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:12.317 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: finishing write tid 3 to smithi16030472-13 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: finishing write tid 4 to smithi16030472-13 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: oid 13 updating version 0 to 11 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: oid 13 version 11 is already newer than 4 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: oid 13 updating version 11 to 12 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 13 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:10: finishing write tid 3 to smithi16030472-10 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:10: finishing write tid 4 to smithi16030472-10 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:10: oid 10 updating version 0 to 3 2024-09-21T18:07:12.325 INFO:tasks.rados.rados.0.smithi160.stdout:10: oid 10 updating version 3 to 8 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:10: oid 10 updating version 8 to 9 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:10: done (15 left) 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:13: done (14 left) 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:25: write initial oid 25 2024-09-21T18:07:12.326 INFO:tasks.rados.rados.0.smithi160.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-21T18:07:12.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:12.332 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:12.335 INFO:tasks.rados.rados.0.smithi160.stdout:25: writing smithi16030472-25 from 403368 to 958403 tid 1 2024-09-21T18:07:12.346 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:12.346 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:12.348 INFO:tasks.rados.rados.0.smithi160.stdout:25: writing smithi16030472-25 from 1454509 to 2182184 tid 2 2024-09-21T18:07:12.348 INFO:tasks.rados.rados.0.smithi160.stdout:26: write initial oid 26 2024-09-21T18:07:12.348 INFO:tasks.rados.rados.0.smithi160.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-21T18:07:12.359 INFO:tasks.rados.rados.0.smithi160.stdout:26: writing smithi16030472-26 from 420175 to 1050459 tid 1 2024-09-21T18:07:12.367 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:12.361+0000 7f0b6a212640 -1 osd.9 42 handle_osd_map injecting map crc failure 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:26: writing smithi16030472-26 from 1796638 to 2267971 tid 2 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: finishing write tid 2 to smithi16030472-20 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: finishing write tid 3 to smithi16030472-20 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: finishing write tid 4 to smithi16030472-20 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: oid 20 updating version 0 to 14 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: oid 20 version 14 is already newer than 5 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: oid 20 updating version 14 to 15 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 20 v 15 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:18: finishing write tid 2 to smithi16030472-18 2024-09-21T18:07:12.372 INFO:tasks.rados.rados.0.smithi160.stdout:18: finishing write tid 3 to smithi16030472-18 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:14: finishing write tid 3 to smithi16030472-14 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:8: finishing write tid 2 to smithi16030472-8 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:8: finishing write tid 3 to smithi16030472-8 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:20: done (15 left) 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:27: write initial oid 27 2024-09-21T18:07:12.373 INFO:tasks.rados.rados.0.smithi160.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-21T18:07:12.383 INFO:tasks.rados.rados.0.smithi160.stdout:27: writing smithi16030472-27 from 436982 to 1142515 tid 1 2024-09-21T18:07:12.386 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:12.381+0000 7f4976023640 -1 osd.5 42 handle_osd_map injecting map crc failure 2024-09-21T18:07:12.397 INFO:tasks.rados.rados.0.smithi160.stdout:27: writing smithi16030472-27 from 1855120 to 2436982 tid 2 2024-09-21T18:07:12.397 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.397 INFO:tasks.rados.rados.0.smithi160.stdout:26: finishing write tid 1 to smithi16030472-26 2024-09-21T18:07:12.421 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:12.419+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:12.432 INFO:tasks.rados.rados.0.smithi160.stdout:15: finishing write tid 1 to smithi16030472-15 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: finishing write tid 4 to smithi16030472-18 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: oid 18 updating version 0 to 9 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: oid 18 updating version 9 to 10 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: oid 18 version 10 is already newer than 4 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 18 v 10 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:18: done (15 left) 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:28: write initial oid 28 2024-09-21T18:07:12.435 INFO:tasks.rados.rados.0.smithi160.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-21T18:07:12.450 INFO:tasks.rados.rados.0.smithi160.stdout:28: writing smithi16030472-28 from 453789 to 1234571 tid 1 2024-09-21T18:07:12.458 INFO:tasks.rados.rados.0.smithi160.stdout:28: writing smithi16030472-28 from 1913602 to 2388604 tid 2 2024-09-21T18:07:12.458 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.458 INFO:tasks.rados.rados.0.smithi160.stdout:16: finishing write tid 2 to smithi16030472-16 2024-09-21T18:07:12.458 INFO:tasks.rados.rados.0.smithi160.stdout:16: finishing write tid 3 to smithi16030472-16 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: finishing write tid 4 to smithi16030472-16 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: oid 16 updating version 0 to 11 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: oid 16 version 11 is already newer than 3 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: oid 16 updating version 11 to 12 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 16 v 12 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:17: finishing write tid 1 to smithi16030472-17 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:21: finishing write tid 1 to smithi16030472-21 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:16: done (15 left) 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:29: write initial oid 29 2024-09-21T18:07:12.459 INFO:tasks.rados.rados.0.smithi160.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-21T18:07:12.467 INFO:tasks.rados.rados.0.smithi160.stdout:29: writing smithi16030472-29 from 470596 to 926627 tid 1 2024-09-21T18:07:12.479 INFO:tasks.rados.rados.0.smithi160.stdout:29: writing smithi16030472-29 from 1572084 to 2190744 tid 2 2024-09-21T18:07:12.479 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.498 INFO:tasks.rados.rados.0.smithi160.stdout:26: finishing write tid 2 to smithi16030472-26 2024-09-21T18:07:12.498 INFO:tasks.rados.rados.0.smithi160.stdout:26: finishing write tid 3 to smithi16030472-26 2024-09-21T18:07:12.498 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:12.498 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:12.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:12.498 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:07:12.499 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:12.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:12.514 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:12.511+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:12.592 INFO:tasks.rados.rados.0.smithi160.stdout:27: finishing write tid 1 to smithi16030472-27 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: finishing write tid 4 to smithi16030472-26 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: oid 26 updating version 0 to 17 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: oid 26 version 17 is already newer than 6 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: oid 26 updating version 17 to 18 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 18 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:26: done (15 left) 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:30: write initial oid 30 2024-09-21T18:07:12.593 INFO:tasks.rados.rados.0.smithi160.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-21T18:07:12.599 INFO:tasks.rados.rados.0.smithi160.stdout:30: writing smithi16030472-30 from 487403 to 1018683 tid 1 2024-09-21T18:07:12.606 INFO:tasks.rados.rados.0.smithi160.stdout:30: writing smithi16030472-30 from 1514213 to 1992884 tid 2 2024-09-21T18:07:12.606 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.615 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:12.613+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:12.648 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:12.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:12.649 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:12.649 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:12.664 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:8: finishing write tid 4 to smithi16030472-8 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:8: oid 8 updating version 0 to 8 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:8: oid 8 version 8 is already newer than 2 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:8: oid 8 updating version 8 to 9 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 8 v 9 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:19: finishing write tid 2 to smithi16030472-19 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:19: finishing write tid 3 to smithi16030472-19 2024-09-21T18:07:12.682 INFO:tasks.rados.rados.0.smithi160.stdout:19: finishing write tid 4 to smithi16030472-19 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:19: oid 19 updating version 0 to 11 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:19: oid 19 updating version 11 to 12 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:19: oid 19 version 12 is already newer than 4 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 19 v 12 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: finishing write tid 4 to smithi16030472-14 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: oid 14 updating version 0 to 3 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: oid 14 updating version 3 to 6 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: oid 14 updating version 6 to 7 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 7 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:22: finishing write tid 1 to smithi16030472-22 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:8: done (15 left) 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:14: done (14 left) 2024-09-21T18:07:12.683 INFO:tasks.rados.rados.0.smithi160.stdout:19: done (13 left) 2024-09-21T18:07:12.684 INFO:tasks.rados.rados.0.smithi160.stdout:31: write initial oid 31 2024-09-21T18:07:12.684 INFO:tasks.rados.rados.0.smithi160.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-21T18:07:12.690 INFO:tasks.rados.rados.0.smithi160.stdout:31: writing smithi16030472-31 from 504210 to 1110739 tid 1 2024-09-21T18:07:12.699 INFO:tasks.rados.rados.0.smithi160.stdout:31: writing smithi16030472-31 from 1572695 to 2195024 tid 2 2024-09-21T18:07:12.699 INFO:tasks.rados.rados.0.smithi160.stdout:32: write initial oid 32 2024-09-21T18:07:12.699 INFO:tasks.rados.rados.0.smithi160.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-21T18:07:12.709 INFO:tasks.rados.rados.0.smithi160.stdout:32: writing smithi16030472-32 from 521017 to 1319148 tid 1 2024-09-21T18:07:12.715 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:12.711+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:12.717 INFO:tasks.rados.rados.0.smithi160.stdout:32: writing smithi16030472-32 from 1747530 to 2229870 tid 2 2024-09-21T18:07:12.717 INFO:tasks.rados.rados.0.smithi160.stdout:33: write initial oid 33 2024-09-21T18:07:12.717 INFO:tasks.rados.rados.0.smithi160.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-21T18:07:12.725 INFO:tasks.rados.rados.0.smithi160.stdout:33: writing smithi16030472-33 from 537824 to 1011204 tid 1 2024-09-21T18:07:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:33: writing smithi16030472-33 from 1806012 to 2432010 tid 2 2024-09-21T18:07:12.736 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:23: finishing write tid 1 to smithi16030472-23 2024-09-21T18:07:12.786 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:12.786 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:12.786 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:12.787 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:12.787 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:12.803 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:12.837 INFO:tasks.rados.rados.0.smithi160.stdout:12: finishing write tid 2 to smithi16030472-12 2024-09-21T18:07:12.837 INFO:tasks.rados.rados.0.smithi160.stdout:12: finishing write tid 3 to smithi16030472-12 2024-09-21T18:07:12.837 INFO:tasks.rados.rados.0.smithi160.stdout:24: finishing write tid 1 to smithi16030472-24 2024-09-21T18:07:12.837 INFO:tasks.rados.rados.0.smithi160.stdout:12: finishing write tid 4 to smithi16030472-12 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:12: oid 12 updating version 0 to 5 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:12: oid 12 updating version 5 to 11 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:12: oid 12 updating version 11 to 12 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: finishing write tid 2 to smithi16030472-11 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: finishing write tid 3 to smithi16030472-11 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: finishing write tid 4 to smithi16030472-11 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: oid 11 updating version 0 to 4 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: oid 11 updating version 4 to 15 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: oid 11 updating version 15 to 16 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 11 v 16 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-21T18:07:12.838 INFO:tasks.rados.rados.0.smithi160.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-21T18:07:12.839 INFO:tasks.rados.rados.0.smithi160.stdout:15: finishing write tid 2 to smithi16030472-15 2024-09-21T18:07:12.839 INFO:tasks.rados.rados.0.smithi160.stdout:11: done (15 left) 2024-09-21T18:07:12.839 INFO:tasks.rados.rados.0.smithi160.stdout:12: done (14 left) 2024-09-21T18:07:12.839 INFO:tasks.rados.rados.0.smithi160.stdout:34: write initial oid 34 2024-09-21T18:07:12.839 INFO:tasks.rados.rados.0.smithi160.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-21T18:07:12.839 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:12.835+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:12.843 INFO:tasks.rados.rados.0.smithi160.stdout:34: writing smithi16030472-34 from 554631 to 1103260 tid 1 2024-09-21T18:07:12.850 INFO:tasks.rados.rados.0.smithi160.stdout:34: writing smithi16030472-34 from 1748141 to 2234150 tid 2 2024-09-21T18:07:12.850 INFO:tasks.rados.rados.0.smithi160.stdout:35: write initial oid 35 2024-09-21T18:07:12.850 INFO:tasks.rados.rados.0.smithi160.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-21T18:07:12.857 INFO:tasks.rados.rados.0.smithi160.stdout:35: writing smithi16030472-35 from 571438 to 1195316 tid 1 2024-09-21T18:07:12.867 INFO:tasks.rados.rados.0.smithi160.stdout:35: writing smithi16030472-35 from 1806623 to 2436290 tid 2 2024-09-21T18:07:12.868 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:12.868 INFO:tasks.rados.rados.0.smithi160.stdout:32: finishing write tid 1 to smithi16030472-32 2024-09-21T18:07:12.917 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:12.913+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:12.965 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:12.965 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:12.965 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:12.979 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:13.017 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:13.015+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:13.018 INFO:tasks.rados.rados.0.smithi160.stdout:23: finishing write tid 2 to smithi16030472-23 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: finishing write tid 3 to smithi16030472-23 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: finishing write tid 4 to smithi16030472-23 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: oid 23 updating version 0 to 15 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: oid 23 version 15 is already newer than 14 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: oid 23 version 15 is already newer than 5 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 23 v 15 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:23: done (15 left) 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:36: write initial oid 36 2024-09-21T18:07:13.049 INFO:tasks.rados.rados.0.smithi160.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-21T18:07:13.058 INFO:tasks.rados.rados.0.smithi160.stdout:36: writing smithi16030472-36 from 588245 to 1287372 tid 1 2024-09-21T18:07:13.067 INFO:tasks.rados.rados.0.smithi160.stdout:36: writing smithi16030472-36 from 1865105 to 2354783 tid 2 2024-09-21T18:07:13.067 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.067 INFO:tasks.rados.rados.0.smithi160.stdout:32: finishing write tid 2 to smithi16030472-32 2024-09-21T18:07:13.067 INFO:tasks.rados.rados.0.smithi160.stdout:32: finishing write tid 3 to smithi16030472-32 2024-09-21T18:07:13.067 INFO:tasks.rados.rados.0.smithi160.stdout:32: finishing write tid 4 to smithi16030472-32 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:32: oid 32 updating version 0 to 17 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:32: oid 32 version 17 is already newer than 6 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:32: oid 32 updating version 17 to 18 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 32 v 18 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:32: done (15 left) 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:37: write initial oid 37 2024-09-21T18:07:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-21T18:07:13.077 INFO:tasks.rados.rados.0.smithi160.stdout:37: writing smithi16030472-37 from 605052 to 1379428 tid 1 2024-09-21T18:07:13.087 INFO:tasks.rados.rados.0.smithi160.stdout:37: writing smithi16030472-37 from 1923587 to 2556923 tid 2 2024-09-21T18:07:13.087 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.100 INFO:tasks.rados.rados.0.smithi160.stdout:31: finishing write tid 1 to smithi16030472-31 2024-09-21T18:07:13.118 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:13.115+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:13.145 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:13.145 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:13.145 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:13.145 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:13.147 INFO:tasks.rados.rados.0.smithi160.stdout:27: finishing write tid 2 to smithi16030472-27 2024-09-21T18:07:13.147 INFO:tasks.rados.rados.0.smithi160.stdout:27: finishing write tid 3 to smithi16030472-27 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: finishing write tid 4 to smithi16030472-27 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: oid 27 updating version 0 to 20 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: oid 27 updating version 20 to 21 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: oid 27 version 21 is already newer than 7 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 21 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:27: done (15 left) 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:38: write initial oid 38 2024-09-21T18:07:13.148 INFO:tasks.rados.rados.0.smithi160.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-21T18:07:13.157 INFO:tasks.rados.rados.0.smithi160.stdout:38: writing smithi16030472-38 from 621859 to 1071484 tid 1 2024-09-21T18:07:13.161 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:13.170 INFO:tasks.rados.rados.0.smithi160.stdout:38: writing smithi16030472-38 from 1865716 to 2621859 tid 2 2024-09-21T18:07:13.170 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.170 INFO:tasks.rados.rados.0.smithi160.stdout:25: finishing write tid 1 to smithi16030472-25 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: finishing write tid 2 to smithi16030472-25 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: finishing write tid 3 to smithi16030472-15 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: finishing write tid 4 to smithi16030472-15 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: oid 15 updating version 0 to 6 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: oid 15 updating version 6 to 17 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: oid 15 updating version 17 to 18 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 18 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:30: finishing write tid 1 to smithi16030472-30 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: finishing write tid 3 to smithi16030472-25 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: finishing write tid 4 to smithi16030472-25 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: oid 25 updating version 0 to 5 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: oid 25 updating version 5 to 7 2024-09-21T18:07:13.171 INFO:tasks.rados.rados.0.smithi160.stdout:25: oid 25 version 7 is already newer than 6 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 25 v 7 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:28: finishing write tid 1 to smithi16030472-28 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:28: finishing write tid 2 to smithi16030472-28 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:30: finishing write tid 2 to smithi16030472-30 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:15: done (15 left) 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:25: done (14 left) 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:39: write initial oid 39 2024-09-21T18:07:13.172 INFO:tasks.rados.rados.0.smithi160.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-21T18:07:13.178 INFO:tasks.rados.rados.0.smithi160.stdout:39: writing smithi16030472-39 from 638666 to 1163540 tid 1 2024-09-21T18:07:13.185 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.13 to 1 (10000) 2024-09-21T18:07:13.189 INFO:tasks.rados.rados.0.smithi160.stdout:39: writing smithi16030472-39 from 1924198 to 2561203 tid 2 2024-09-21T18:07:13.189 INFO:tasks.rados.rados.0.smithi160.stdout:40: write initial oid 40 2024-09-21T18:07:13.189 INFO:tasks.rados.rados.0.smithi160.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-21T18:07:13.199 INFO:tasks.rados.rados.0.smithi160.stdout:40: writing smithi16030472-40 from 655473 to 1371949 tid 1 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:40: writing smithi16030472-40 from 2099033 to 2655473 tid 2 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:21: finishing write tid 2 to smithi16030472-21 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: finishing write tid 2 to smithi16030472-31 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: finishing write tid 3 to smithi16030472-31 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: finishing write tid 4 to smithi16030472-31 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: oid 31 updating version 0 to 24 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: oid 31 version 24 is already newer than 23 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: oid 31 version 24 is already newer than 8 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 24 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-21T18:07:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:31: done (15 left) 2024-09-21T18:07:13.210 INFO:tasks.rados.rados.0.smithi160.stdout:41: write initial oid 41 2024-09-21T18:07:13.210 INFO:tasks.rados.rados.0.smithi160.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-21T18:07:13.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-21T18:07:13.218 INFO:tasks.rados.rados.0.smithi160.stdout:41: writing smithi16030472-41 from 672280 to 1464005 tid 1 2024-09-21T18:07:13.219 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:13.215+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:13.226 INFO:tasks.rados.rados.0.smithi160.stdout:41: writing smithi16030472-41 from 2157515 to 2672280 tid 2 2024-09-21T18:07:13.226 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.249 INFO:tasks.rados.rados.0.smithi160.stdout:28: finishing write tid 3 to smithi16030472-28 2024-09-21T18:07:13.249 INFO:tasks.rados.rados.0.smithi160.stdout:28: finishing write tid 4 to smithi16030472-28 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:28: oid 28 updating version 0 to 8 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:28: oid 28 updating version 8 to 9 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:28: oid 28 updating version 9 to 10 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 28 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: finishing write tid 1 to smithi16030472-29 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: finishing write tid 2 to smithi16030472-29 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: finishing write tid 3 to smithi16030472-29 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: finishing write tid 4 to smithi16030472-29 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: oid 29 updating version 0 to 13 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: oid 29 version 13 is already newer than 11 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: oid 29 version 13 is already newer than 12 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 29 v 13 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-21T18:07:13.250 INFO:tasks.rados.rados.0.smithi160.stdout:30: finishing write tid 3 to smithi16030472-30 2024-09-21T18:07:13.251 INFO:tasks.rados.rados.0.smithi160.stdout:28: done (15 left) 2024-09-21T18:07:13.251 INFO:tasks.rados.rados.0.smithi160.stdout:29: done (14 left) 2024-09-21T18:07:13.251 INFO:tasks.rados.rados.0.smithi160.stdout:42: write initial oid 42 2024-09-21T18:07:13.251 INFO:tasks.rados.rados.0.smithi160.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-21T18:07:13.257 INFO:tasks.rados.rados.0.smithi160.stdout:42: writing smithi16030472-42 from 689087 to 1156061 tid 1 2024-09-21T18:07:13.266 INFO:tasks.rados.rados.0.smithi160.stdout:42: writing smithi16030472-42 from 1699644 to 2483976 tid 2 2024-09-21T18:07:13.266 INFO:tasks.rados.rados.0.smithi160.stdout:43: write initial oid 43 2024-09-21T18:07:13.267 INFO:tasks.rados.rados.0.smithi160.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-21T18:07:13.274 INFO:tasks.rados.rados.0.smithi160.stdout:43: writing smithi16030472-43 from 705894 to 1248117 tid 1 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:43: writing smithi16030472-43 from 1758126 to 2402469 tid 2 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:30: finishing write tid 4 to smithi16030472-30 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:30: oid 30 updating version 0 to 12 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:30: oid 30 version 12 is already newer than 11 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:30: oid 30 updating version 12 to 13 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 30 v 13 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-21T18:07:13.282 INFO:tasks.rados.rados.0.smithi160.stdout:33: finishing write tid 1 to smithi16030472-33 2024-09-21T18:07:13.283 INFO:tasks.rados.rados.0.smithi160.stdout:30: done (15 left) 2024-09-21T18:07:13.283 INFO:tasks.rados.rados.0.smithi160.stdout:44: write initial oid 44 2024-09-21T18:07:13.283 INFO:tasks.rados.rados.0.smithi160.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-21T18:07:13.290 INFO:tasks.rados.rados.0.smithi160.stdout:44: writing smithi16030472-44 from 722701 to 1340173 tid 1 2024-09-21T18:07:13.291 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:13.291 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:13.291 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:13.291 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:13.291 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:13.299 INFO:tasks.rados.rados.0.smithi160.stdout:44: writing smithi16030472-44 from 1816608 to 2604609 tid 2 2024-09-21T18:07:13.299 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.305 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:13.320 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:13.317+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:13.348 INFO:tasks.rados.rados.0.smithi160.stdout:21: finishing write tid 3 to smithi16030472-21 2024-09-21T18:07:13.348 INFO:tasks.rados.rados.0.smithi160.stdout:21: finishing write tid 4 to smithi16030472-21 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:21: oid 21 updating version 0 to 8 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:21: oid 21 updating version 8 to 22 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:21: oid 21 version 22 is already newer than 21 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 22 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:17: finishing write tid 2 to smithi16030472-17 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:21: done (15 left) 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:45: write initial oid 45 2024-09-21T18:07:13.349 INFO:tasks.rados.rados.0.smithi160.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-21T18:07:13.357 INFO:tasks.rados.rados.0.smithi160.stdout:45: writing smithi16030472-45 from 739508 to 1432229 tid 1 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout:45: writing smithi16030472-45 from 1875090 to 2523102 tid 2 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout:17: finishing write tid 3 to smithi16030472-17 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout:17: finishing write tid 4 to smithi16030472-17 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout:17: oid 17 updating version 0 to 23 2024-09-21T18:07:13.364 INFO:tasks.rados.rados.0.smithi160.stdout:17: oid 17 updating version 23 to 24 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:17: oid 17 version 24 is already newer than 7 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 17 v 24 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:22: finishing write tid 2 to smithi16030472-22 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:17: done (15 left) 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:46: write initial oid 46 2024-09-21T18:07:13.365 INFO:tasks.rados.rados.0.smithi160.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-21T18:07:13.373 INFO:tasks.rados.rados.0.smithi160.stdout:46: writing smithi16030472-46 from 756315 to 1524285 tid 1 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:46: writing smithi16030472-46 from 1933572 to 2725242 tid 2 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: finishing write tid 3 to smithi16030472-22 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: finishing write tid 4 to smithi16030472-22 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: oid 22 updating version 0 to 27 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: oid 22 version 27 is already newer than 9 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: oid 22 version 27 is already newer than 26 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 22 v 27 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:07:13.381 INFO:tasks.rados.rados.0.smithi160.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-21T18:07:13.382 INFO:tasks.rados.rados.0.smithi160.stdout:24: finishing write tid 2 to smithi16030472-24 2024-09-21T18:07:13.382 INFO:tasks.rados.rados.0.smithi160.stdout:34: finishing write tid 1 to smithi16030472-34 2024-09-21T18:07:13.382 INFO:tasks.rados.rados.0.smithi160.stdout:22: done (15 left) 2024-09-21T18:07:13.382 INFO:tasks.rados.rados.0.smithi160.stdout:47: write initial oid 47 2024-09-21T18:07:13.382 INFO:tasks.rados.rados.0.smithi160.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-21T18:07:13.390 INFO:tasks.rados.rados.0.smithi160.stdout:47: writing smithi16030472-47 from 773122 to 1332694 tid 1 2024-09-21T18:07:13.399 INFO:tasks.rados.rados.0.smithi160.stdout:47: writing smithi16030472-47 from 1992054 to 2643735 tid 2 2024-09-21T18:07:13.399 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.420 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:13.415+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: finishing write tid 3 to smithi16030472-24 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: finishing write tid 4 to smithi16030472-24 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: oid 24 updating version 0 to 10 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: oid 24 updating version 10 to 29 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: oid 24 updating version 29 to 30 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 24 v 30 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-21T18:07:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: finishing write tid 2 to smithi16030472-34 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: finishing write tid 3 to smithi16030472-34 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: finishing write tid 4 to smithi16030472-34 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: oid 34 updating version 0 to 15 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: oid 34 updating version 15 to 16 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: oid 34 version 16 is already newer than 14 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 16 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: finishing write tid 2 to smithi16030472-33 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: finishing write tid 3 to smithi16030472-33 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: finishing write tid 4 to smithi16030472-33 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: oid 33 updating version 0 to 15 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: oid 33 version 15 is already newer than 14 2024-09-21T18:07:13.462 INFO:tasks.rados.rados.0.smithi160.stdout:33: oid 33 updating version 15 to 16 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 33 v 16 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:24: done (15 left) 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:33: done (14 left) 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:34: done (13 left) 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:48: write initial oid 48 2024-09-21T18:07:13.463 INFO:tasks.rados.rados.0.smithi160.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-21T18:07:13.472 INFO:tasks.rados.rados.0.smithi160.stdout:48: writing smithi16030472-48 from 789929 to 1424750 tid 1 2024-09-21T18:07:13.481 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:13.481 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:13.481 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:13.482 INFO:tasks.rados.rados.0.smithi160.stdout:48: writing smithi16030472-48 from 2050536 to 2789929 tid 2 2024-09-21T18:07:13.482 INFO:tasks.rados.rados.0.smithi160.stdout:49: write initial oid 49 2024-09-21T18:07:13.482 INFO:tasks.rados.rados.0.smithi160.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-21T18:07:13.491 INFO:tasks.rados.rados.0.smithi160.stdout:49: writing smithi16030472-49 from 406736 to 1116806 tid 1 2024-09-21T18:07:13.496 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:13.501 INFO:tasks.rados.rados.0.smithi160.stdout:49: writing smithi16030472-49 from 1709018 to 2364368 tid 2 2024-09-21T18:07:13.501 INFO:tasks.rados.rados.0.smithi160.stdout:50: write initial oid 50 2024-09-21T18:07:13.501 INFO:tasks.rados.rados.0.smithi160.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-21T18:07:13.510 INFO:tasks.rados.rados.0.smithi160.stdout:50: writing smithi16030472-50 from 423543 to 1208862 tid 1 2024-09-21T18:07:13.519 INFO:tasks.rados.rados.0.smithi160.stdout:50: writing smithi16030472-50 from 1767500 to 2566508 tid 2 2024-09-21T18:07:13.519 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.519 INFO:tasks.rados.rados.0.smithi160.stdout:35: finishing write tid 1 to smithi16030472-35 2024-09-21T18:07:13.521 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:13.517+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:13.577 INFO:tasks.rados.rados.0.smithi160.stdout:41: finishing write tid 1 to smithi16030472-41 2024-09-21T18:07:13.622 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:13.617+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:138 4.3 4:cd35ea38:::smithi16030472-35:head [write 1806623~629667 in=629667b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144130+0000", 2024-09-21T18:07:13.636 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48819347000000002, 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.488242226, 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 138 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144130+0000", 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144130+0000", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144134+0000", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4630000000000001e-06 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144899+0000", 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00076531699999999995 2024-09-21T18:07:13.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144924+0000", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5082000000000001e-05 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144938+0000", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3597e-05 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245094+0000", 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10015653200000001 2024-09-21T18:07:13.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:23 3.2 3:6abed166:::smithi16030472-39:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.459654+0000", 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.172669504, 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.172823855, 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459654+0000", 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459654+0000", 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459655+0000", 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.06e-06 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459661+0000", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7239999999999997e-06 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459680+0000", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8746e-05 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459693+0000", 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2802999999999999e-05 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.464002+0000", 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0043095609999999999 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.464073+0000", 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0970000000000007e-05 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.643 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.487357918, 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.487601907, 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:07:13.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:24 3.2 3:6abed166:::smithi16030472-39:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.459714+0000", 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.17260951499999999, 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17294853199999999, 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459714+0000", 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459714+0000", 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459715+0000", 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3399999999999997e-07 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459720+0000", 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6399999999999996e-06 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459730+0000", 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0426e-05 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459738+0000", 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9160000000000001e-06 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.464112+0000", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0043743260000000004 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.464200+0000", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7478000000000005e-05 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:13.651 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48731160899999998, 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.487751661, 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:13.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:07:13.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:07:13.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:141 4.3 4:d903dcdd:::smithi16030472-36:head [write 588245~699127 in=699127b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145065+0000", 2024-09-21T18:07:13.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.4872591, 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48779294400000001, 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 141 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145065+0000", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145065+0000", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145066+0000", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.08e-06 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145723+0000", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00065707799999999996 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145737+0000", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4263999999999999e-05 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145746+0000", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7900000000000005e-06 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245568+0000", 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099822606999999994 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:153 3.2 e44/31)", 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.560534+0000", 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.071789417999999994, 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.072398199999999996, 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560534+0000", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560534+0000", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560537+0000", 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8940000000000002e-06 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560547+0000", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7720000000000006e-06 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560572+0000", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4654000000000001e-05 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.560588+0000", 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6589e-05 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.661 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:142 4.3 4:d903dcdd:::smithi16030472-36:head [write 1865105~489678 in=489678b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145760+0000", 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48656403199999998, 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48723927299999997, 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 142 2024-09-21T18:07:13.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145760+0000", 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145760+0000", 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145761+0000", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1000000000000001e-06 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146227+0000", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000466568 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146241+0000", 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3173000000000001e-05 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146248+0000", 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5440000000000001e-06 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245771+0000", 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099523158 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:143 4.3 4:d903dcdd:::smithi16030472-36:head [setxattr _header (56) in=63b,truncate 2588245] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.486063674, 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.486828027, 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 143 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146261+0000", 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0330000000000001e-06 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146266+0000", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6229999999999996e-06 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146276+0000", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0326000000000001e-05 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146295+0000", 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9097e-05 2024-09-21T18:07:13.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246122+0000", 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099826456999999993 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246156+0000", 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4924e-05 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.669 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:144 4.3 4:d903dcdd:::smithi16030472-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48601829800000002, 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48687239599999999, 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 144 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:13.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146306+0000", 2024-09-21T18:07:13.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:07:13.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146310+0000", 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7239999999999998e-06 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146318+0000", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2689999999999999e-06 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146325+0000", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0400000000000004e-06 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246212+0000", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099886246999999997 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246253+0000", 2024-09-21T18:07:13.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1752999999999998e-05 2024-09-21T18:07:13.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.682 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:145 4.3 4:f074207e:::smithi16030472-37:head [write 605052~774376 in=774376b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.146336+0000", 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48598719299999998, 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48693067299999998, 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 145 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146336+0000", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146336+0000", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146337+0000", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9400000000000004e-07 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147073+0000", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00073565700000000002 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147086+0000", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3426e-05 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147096+0000", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4700000000000008e-06 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246328+0000", 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099232290000000001 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48521569599999997, 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48623954800000002, 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.689 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48455258099999998, 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48566604499999999, 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:07:13.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:07:13.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:07:13.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.693 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48451608200000001, 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48572970700000001, 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:07:13.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:162 4.1 4:bf36a652:::smithi16030472-41:head [write 2157515~514765 in=514765b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.249027+0000", 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38329670399999999, 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38460115700000003, 2024-09-21T18:07:13.697 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 162 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249027+0000", 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249027+0000", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249028+0000", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3200000000000004e-07 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249347+0000", 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031880299999999999 2024-09-21T18:07:13.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249354+0000", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9200000000000004e-06 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249362+0000", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4019999999999996e-06 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349522+0000", 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10015993099999999 2024-09-21T18:07:13.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349531+0000", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0100000000000001e-06 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451009+0000", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101478082 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451169+0000", 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016008 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.583530+0000", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.132360964 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.583665+0000", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013526599999999999 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.583842+0000", 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000177023 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.585987+0000", 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021448019999999999 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:163 4.1 4:bf36a652:::smithi16030472-41:head [setxattr _header (56) in=63b,truncate 2672280] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.249374+0000", 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38295001299999998, 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38439738800000001, 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.703 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 163 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249374+0000", 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249374+0000", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249374+0000", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0900000000000001e-07 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249378+0000", 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5860000000000001e-06 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249382+0000", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3970000000000004e-06 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249387+0000", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9579999999999998e-06 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349734+0000", 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10034689300000001 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349751+0000", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6463000000000001e-05 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451287+0000", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10153640799999999 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451527+0000", 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000239979 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.584853+0000", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13332549099999999 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.585111+0000", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025800600000000002 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.585437+0000", 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00032618399999999998 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.586241+0000", 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00080383899999999999 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:164 4.1 4:bf36a652:::smithi16030472-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.249405+0000", 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38291882500000002, 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38450074699999998, 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:13.709 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 164 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249405+0000", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249405+0000", 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249405+0000", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1500000000000005e-07 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249413+0000", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6850000000000001e-06 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249418+0000", 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0799999999999996e-06 2024-09-21T18:07:13.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.249422+0000", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1330000000000001e-06 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451763+0000", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.202340725 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.451887+0000", 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012415700000000001 2024-09-21T18:07:13.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.586031+0000", 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.134143874 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.586131+0000", 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100132 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:169 4.3 4:e85d446d:::smithi16030472-43:head [write 705894~542223 in=542223b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.273695+0000", 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.35862840800000001, 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.360361608, 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 169 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273695+0000", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273695+0000", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273698+0000", 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.056e-06 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273890+0000", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019160800000000001 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273907+0000", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7427000000000001e-05 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.273918+0000", 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0509999999999999e-05 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448468+0000", 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.174549807 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.35087791000000002, 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35269281400000002, 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:07:13.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.720 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.350166119, 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35207079200000002, 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:07:13.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:07:13.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:07:13.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.724 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.35012855599999998, 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35212116700000001, 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.725 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:07:13.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.728 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:153 3.2 3:6abed166:::smithi16030472-39:head [write 638666~524874 in=524874b,stat out=16b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.459394+0000", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.172929789, 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17501546600000001, 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459394+0000", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:13.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459394+0000", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459395+0000", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.47e-07 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459620+0000", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022492899999999999 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:13.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459633+0000", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3658999999999999e-05 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.459642+0000", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3899999999999993e-06 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.462271+0000", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026295619999999998 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:13.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.462406+0000", 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013454800000000001 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.462568+0000", 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161757 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.466641+0000", 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040735290000000002 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.732 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:13.733 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:13.733 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 22 2024-09-21T18:07:13.733 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:13.734 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: finishing write tid 2 to smithi16030472-41 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: finishing write tid 3 to smithi16030472-41 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: finishing write tid 4 to smithi16030472-41 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: oid 41 updating version 0 to 18 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: oid 41 version 18 is already newer than 17 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: oid 41 updating version 18 to 19 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 41 v 19 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:41: done (15 left) 2024-09-21T18:07:13.736 INFO:tasks.rados.rados.0.smithi160.stdout:51: write initial oid 51 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:51: writing smithi16030472-51 from 440350 to 900918 tid 1 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:51: writing smithi16030472-51 from 1309629 to 1852295 tid 2 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:51: writing smithi16030472-51 from 2323973 to 2840350 tid 3 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:38: finishing write tid 1 to smithi16030472-38 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:39: finishing write tid 1 to smithi16030472-39 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:40: finishing write tid 1 to smithi16030472-40 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:42: finishing write tid 1 to smithi16030472-42 2024-09-21T18:07:13.737 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:13.731+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:13.737 INFO:tasks.rados.rados.0.smithi160.stdout:42: finishing write tid 2 to smithi16030472-42 2024-09-21T18:07:13.741 INFO:tasks.rados.rados.0.smithi160.stdout:44: finishing write tid 1 to smithi16030472-44 2024-09-21T18:07:13.741 INFO:tasks.rados.rados.0.smithi160.stdout:44: finishing write tid 2 to smithi16030472-44 2024-09-21T18:07:13.741 INFO:tasks.rados.rados.0.smithi160.stdout:44: finishing write tid 3 to smithi16030472-44 2024-09-21T18:07:13.835 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:13.831+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:13.865 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:13.866 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:13.866 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:13.866 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:13.866 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:13.884 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:13.935 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:13.931+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:13.942 INFO:tasks.rados.rados.0.smithi160.stdout:45: finishing write tid 1 to smithi16030472-45 2024-09-21T18:07:13.952 INFO:tasks.rados.rados.0.smithi160.stdout:46: finishing write tid 1 to smithi16030472-46 2024-09-21T18:07:13.984 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:13.981+0000 7fb924fb6640 -1 osd.8 43 handle_osd_map injecting map crc failure 2024-09-21T18:07:14.006 INFO:tasks.rados.rados.0.smithi160.stdout:51: finishing write tid 1 to smithi16030472-51 2024-09-21T18:07:14.006 INFO:tasks.rados.rados.0.smithi160.stdout:51: finishing write tid 2 to smithi16030472-51 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:96 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:48.894069+0000", 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.113984813999998, 2024-09-21T18:07:14.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61232676900000005, 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894069+0000", 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894069+0000", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894070+0000", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2160000000000001e-06 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894165+0000", 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4537999999999995e-05 2024-09-21T18:07:14.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894175+0000", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7829999999999993e-06 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894183+0000", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4630000000000001e-06 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.203054+0000", 2024-09-21T18:07:14.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30887053399999997 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.203170+0000", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011660299999999999 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.203383+0000", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021291499999999999 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.208806+0000", 2024-09-21T18:07:14.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054226099999999996 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506230+0000", 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29742475499999999 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506301+0000", 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0241999999999996e-05 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506396+0000", 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5110000000000002e-05 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:97 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:48.894195+0000", 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.113859230999999, 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61250116799999998, 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.018 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 97 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894195+0000", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894195+0000", 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894196+0000", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1829999999999999e-06 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894203+0000", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0770000000000002e-06 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894211+0000", 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9330000000000001e-06 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894218+0000", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8900000000000001e-06 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.204383+0000", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31016569500000002 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.204495+0000", 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011116999999999999 2024-09-21T18:07:14.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.204676+0000", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181631 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.208812+0000", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0041356559999999997 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506540+0000", 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.297728194 2024-09-21T18:07:14.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506628+0000", 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7661e-05 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506696+0000", 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8077999999999994e-05 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.023 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:98 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:48.894227+0000", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.113827157999999, 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61270820100000001, 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 98 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894227+0000", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894227+0000", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894228+0000", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6600000000000005e-07 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894231+0000", 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.907e-06 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894239+0000", 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6159999999999996e-06 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894246+0000", 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9630000000000001e-06 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.205445+0000", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31119902999999999 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.205568+0000", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000123103 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.205801+0000", 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000233293 2024-09-21T18:07:14.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.208819+0000", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0030176140000000001 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506777+0000", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29795834700000001 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506846+0000", 2024-09-21T18:07:14.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8528000000000005e-05 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506935+0000", 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8933999999999999e-05 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:99 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:48.894255+0000", 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.113799269000001, 2024-09-21T18:07:14.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61290802, 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 99 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894255+0000", 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894255+0000", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894255+0000", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9999999999999996e-07 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894341+0000", 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5154000000000005e-05 2024-09-21T18:07:14.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.894361+0000", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0119000000000002e-05 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994462+0000", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100101619 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.206457+0000", 2024-09-21T18:07:14.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21199509899999999 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.206584+0000", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126897 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.206797+0000", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021229800000000001 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.208827+0000", 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020301400000000002 2024-09-21T18:07:14.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.506996+0000", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29816899099999999 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.507070+0000", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4428e-05 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.507163+0000", 2024-09-21T18:07:14.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2474999999999998e-05 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:100 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:48.994490+0000", 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.013563815000001, 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71285538999999998, 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.035 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 100 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994490+0000", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994490+0000", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994492+0000", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2400000000000002e-06 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994503+0000", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0963e-05 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994523+0000", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9298999999999999e-05 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:48.994534+0000", 2024-09-21T18:07:14.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.102e-05 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.207753+0000", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21321978599999999 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.207895+0000", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014198699999999999 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.208132+0000", 2024-09-21T18:07:14.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023639599999999999 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.211481+0000", 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033496110000000002 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.707054+0000", 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.49557245300000002 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.707226+0000", 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017267199999999999 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:49.707345+0000", 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011896300000000001 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:123 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:50.585559+0000", 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 23.422494788000002, 2024-09-21T18:07:14.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61496320800000004, 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 123 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.585559+0000", 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.585559+0000", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.585561+0000", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.646e-06 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.585640+0000", 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9314999999999995e-05 2024-09-21T18:07:14.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.585658+0000", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7618999999999999e-05 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685819+0000", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100161364 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.098996+0000", 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41317683700000002 2024-09-21T18:07:14.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.099133+0000", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137219 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.099331+0000", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000197479 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.101338+0000", 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0020070589999999998 2024-09-21T18:07:14.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.200313+0000", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098974952000000005 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.200398+0000", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5504e-05 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.200522+0000", 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012421399999999999 2024-09-21T18:07:14.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4263.0:124 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:06:50.685848+0000", 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 23.322206094999999, 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61735542899999996, 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4263", 2024-09-21T18:07:14.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/3972287121", 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 124 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685848+0000", 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685848+0000", 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685850+0000", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4669999999999998e-06 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685861+0000", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0379e-05 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685876+0000", 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5866999999999999e-05 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:50.685888+0000", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1372999999999999e-05 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.100216+0000", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.414328592 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.100342+0000", 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000125117 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.100530+0000", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018871900000000001 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.102074+0000", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0015438380000000001 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.303062+0000", 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.200987531 2024-09-21T18:07:14.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.303142+0000", 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0034e-05 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:06:51.303203+0000", 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1512000000000003e-05 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.051 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:94 4.3 4:fbf38165:::smithi16030472-24:head [write 1796027~467664] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.323602+0000", 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.684451812, 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0255045899999999, 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 94 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323602+0000", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323602+0000", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323603+0000", 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1400000000000001e-06 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323822+0000", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021861 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323834+0000", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843e-05 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323843+0000", 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0899999999999994e-06 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.425275+0000", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101432086 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.425407+0000", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000132068 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.788738+0000", 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36333109800000002 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.788873+0000", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013521799999999999 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.789019+0000", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014578900000000001 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790895+0000", 2024-09-21T18:07:14.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0018757789999999999 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349031+0000", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.55813668100000002 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349075+0000", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3980999999999998e-05 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.349107+0000", 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1207000000000001e-05 2024-09-21T18:07:14.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:95 4.3 4:fbf38165:::smithi16030472-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.684196552, 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.124057401, 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.058 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 95 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323859+0000", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.336e-06 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323864+0000", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.49e-06 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323873+0000", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4759999999999998e-06 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323880+0000", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426219+0000", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102339044 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426347+0000", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012758800000000001 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.789980+0000", 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36363283400000002 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790091+0000", 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110766 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790205+0000", 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011423 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.791431+0000", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012262099999999999 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447768+0000", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65633668700000003 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447842+0000", 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4401999999999997e-05 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447915+0000", 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2640999999999998e-05 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:96 4.3 4:fbf38165:::smithi16030472-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.6841637199999999, 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.124285918, 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.064 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:14.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323891+0000", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1299999999999998e-07 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323895+0000", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9400000000000004e-06 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323904+0000", 2024-09-21T18:07:14.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7469999999999994e-06 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323917+0000", 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2917e-05 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426413+0000", 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102496121 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426513+0000", 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100098 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790579+0000", 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36406573399999997 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790657+0000", 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7955999999999997e-05 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447958+0000", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65730143699999999 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448078+0000", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011979799999999999 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448176+0000", 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8257000000000005e-05 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:99 4.1 4:af0b33ab:::smithi16030472-25:head [setxattr _header (56) in=7b,truncate 2403368] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.348826+0000", 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.659227494, 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63627781900000002, 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 99 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.070 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348826+0000", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348826+0000", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348828+0000", 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.716e-06 2024-09-21T18:07:14.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348836+0000", 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3869999999999995e-06 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348850+0000", 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3115999999999999e-05 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348858+0000", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7770000000000008e-06 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.639860+0000", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.291001607 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.985103+0000", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.34524305199999999 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.985104+0000", 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164e-06 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:100 4.1 4:af0b33ab:::smithi16030472-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.348872+0000", 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.6591816050000001, 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63626168900000002, 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 100 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348872+0000", 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348872+0000", 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348873+0000", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.034e-06 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348877+0000", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6600000000000001e-06 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348888+0000", 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1472e-05 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.348896+0000", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5530000000000004e-06 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.640514+0000", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.29161785800000001 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.985134+0000", 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.34462011199999998 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.077 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:111 4.1 4:8cbe42fb:::smithi16030472-28:head [setxattr _header (56) in=7b,truncate 2453789] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.457929+0000", 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.5501247849999999, 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70051068100000002, 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 111 2024-09-21T18:07:14.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457929+0000", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457929+0000", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457931+0000", 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.486e-06 2024-09-21T18:07:14.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457937+0000", 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3400000000000003e-06 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457947+0000", 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6229999999999992e-06 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457957+0000", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0327e-05 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.741931+0000", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28397413500000002 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.158435+0000", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41650372400000002 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.158440+0000", 2024-09-21T18:07:14.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0459999999999996e-06 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:112 4.1 4:8cbe42fb:::smithi16030472-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.457968+0000", 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.5500858129999999, 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70119764299999998, 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.083 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 112 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457968+0000", 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457968+0000", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457969+0000", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457973+0000", 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7400000000000002e-06 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457989+0000", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6750000000000001e-05 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.457998+0000", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1000000000000004e-06 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.742146+0000", 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28414810000000001 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.159166+0000", 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41702 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:113 4.1 4:9a6092cc:::smithi16030472-29:head [write 470596~456031,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.466888+0000", 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.541165715, 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.69244035100000001, 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.089 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 113 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.466888+0000", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.466888+0000", 2024-09-21T18:07:14.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.466891+0000", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2229999999999999e-06 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.467114+0000", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022224000000000001 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.467144+0000", 2024-09-21T18:07:14.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9966000000000001e-05 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.467159+0000", 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5307000000000001e-05 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.742520+0000", 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.27536073700000002 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.159323+0000", 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41680335899999998 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.159328+0000", 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.519e-06 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:114 4.1 4:9a6092cc:::smithi16030472-29:head [write 1572084~618660] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.479152+0000", 2024-09-21T18:07:14.093 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.52890138, 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.68061677499999995, 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 114 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479152+0000", 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479152+0000", 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479156+0000", 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3500000000000001e-06 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479472+0000", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031644799999999999 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479503+0000", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0315999999999999e-05 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479518+0000", 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5475999999999999e-05 2024-09-21T18:07:14.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.742824+0000", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26330614499999999 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.159767+0000", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41694325399999999 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.159769+0000", 2024-09-21T18:07:14.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:115 4.1 4:9a6092cc:::smithi16030472-29:head [setxattr _header (56) in=7b,truncate 2470596] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.479543+0000", 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.5285105969999999, 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.68067152500000005, 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 115 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479543+0000", 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479543+0000", 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479545+0000", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3769999999999999e-06 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479551+0000", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4389999999999997e-06 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.479561+0000", 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7969999999999995e-06 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.579671+0000", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100110041 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.743180+0000", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16350953500000001 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.160210+0000", 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.417029702 2024-09-21T18:07:14.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.160215+0000", 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.634e-06 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:130 4.3 4:f5c0ad04:::smithi16030472-33:head [write 1806012~625998] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.735460+0000", 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2725939310000001, 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71673161600000002, 2024-09-21T18:07:14.102 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 130 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735460+0000", 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735460+0000", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735464+0000", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0069999999999999e-06 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735651+0000", 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018722199999999999 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735674+0000", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2580999999999999e-05 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735687+0000", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2860000000000001e-05 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790923+0000", 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.055236256999999997 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452187+0000", 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66126411900000004 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452192+0000", 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5700000000000003e-06 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.106 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:131 4.3 4:f5c0ad04:::smithi16030472-33:head [setxattr _header (56) in=7b,truncate 2537824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.735717+0000", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2723365529999999, 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71655215900000002, 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 131 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.107 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735717+0000", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735717+0000", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735718+0000", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0200000000000001e-07 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.108 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735724+0000", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9619999999999998e-06 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735731+0000", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2350000000000002e-06 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735736+0000", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.544e-06 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.109 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.883966+0000", 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.148230321 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452267+0000", 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.56830129200000001 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452269+0000", 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.103e-06 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.110 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:132 4.3 4:f5c0ad04:::smithi16030472-33:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.735743+0000", 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2723110530000001, 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71656985900000003, 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 132 2024-09-21T18:07:14.111 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735743+0000", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735743+0000", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735743+0000", 2024-09-21T18:07:14.112 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3799999999999997e-07 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735746+0000", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5670000000000002e-06 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.735767+0000", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1004000000000001e-05 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.835912+0000", 2024-09-21T18:07:14.113 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10014479900000001 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.884151+0000", 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.048239738999999997 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452313+0000", 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.56816121200000003 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.114 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.115 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:14.116 INFO:tasks.rados.rados.0.smithi160.stdout:35: finishing write tid 2 to smithi16030472-35 2024-09-21T18:07:14.116 INFO:tasks.rados.rados.0.smithi160.stdout:36: finishing write tid 1 to smithi16030472-36 2024-09-21T18:07:14.116 INFO:tasks.rados.rados.0.smithi160.stdout:36: finishing write tid 2 to smithi16030472-36 2024-09-21T18:07:14.116 INFO:tasks.rados.rados.0.smithi160.stdout:51: finishing write tid 3 to smithi16030472-51 2024-09-21T18:07:14.116 INFO:tasks.rados.rados.0.smithi160.stdout:51: finishing write tid 4 to smithi16030472-51 2024-09-21T18:07:14.117 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:46: finishing write tid 2 to smithi16030472-46 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:48: finishing write tid 1 to smithi16030472-48 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:48: finishing write tid 2 to smithi16030472-48 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:38: finishing write tid 2 to smithi16030472-38 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:38: finishing write tid 3 to smithi16030472-38 2024-09-21T18:07:14.118 INFO:tasks.rados.rados.0.smithi160.stdout:38: finishing write tid 4 to smithi16030472-38 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:38: oid 38 updating version 0 to 20 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:38: oid 38 updating version 20 to 21 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:38: oid 38 version 21 is already newer than 7 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 21 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:38: done (15 left) 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:52: write initial oid 52 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-21T18:07:14.119 INFO:tasks.rados.rados.0.smithi160.stdout:52: writing smithi16030472-52 from 457157 to 992974 tid 1 2024-09-21T18:07:14.119 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:14.113+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:14.122 INFO:tasks.rados.rados.0.smithi160.stdout:52: writing smithi16030472-52 from 1768111 to 2170788 tid 2 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:47: finishing write tid 1 to smithi16030472-47 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:49: finishing write tid 1 to smithi16030472-49 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: finishing write tid 5 to smithi16030472-51 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: oid 51 updating version 0 to 20 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: oid 51 version 20 is already newer than 18 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: oid 51 version 20 is already newer than 19 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: oid 51 updating version 20 to 21 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 21 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: left oid 51 (ObjNum 50 snap 0 seq_num 50) 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:51: done (15 left) 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:53: write initial oid 53 2024-09-21T18:07:14.123 INFO:tasks.rados.rados.0.smithi160.stdout:53: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2024-09-21T18:07:14.131 INFO:tasks.rados.rados.0.smithi160.stdout:53: writing smithi16030472-53 from 473964 to 1085030 tid 1 2024-09-21T18:07:14.140 INFO:tasks.rados.rados.0.smithi160.stdout:53: writing smithi16030472-53 from 1826593 to 2372928 tid 2 2024-09-21T18:07:14.144 INFO:tasks.rados.rados.0.smithi160.stdout:53: writing smithi16030472-53 from 2778819 to 2873964 tid 3 2024-09-21T18:07:14.145 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.145 INFO:tasks.rados.rados.0.smithi160.stdout:37: finishing write tid 1 to smithi16030472-37 2024-09-21T18:07:14.145 INFO:tasks.rados.rados.0.smithi160.stdout:43: finishing write tid 1 to smithi16030472-43 2024-09-21T18:07:14.177 INFO:tasks.rados.rados.0.smithi160.stdout:50: finishing write tid 1 to smithi16030472-50 2024-09-21T18:07:14.196 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.14 to 1 (10000) 2024-09-21T18:07:14.196 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:14.193+0000 7f4978027640 -1 osd.5 44 handle_osd_map injecting map crc failure 2024-09-21T18:07:14.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:07:14.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-21T18:07:14.218 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:14.217+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:14.221 INFO:tasks.rados.rados.0.smithi160.stdout:39: finishing write tid 2 to smithi16030472-39 2024-09-21T18:07:14.221 INFO:tasks.rados.rados.0.smithi160.stdout:39: finishing write tid 3 to smithi16030472-39 2024-09-21T18:07:14.222 INFO:tasks.rados.rados.0.smithi160.stdout:39: finishing write tid 4 to smithi16030472-39 2024-09-21T18:07:14.222 INFO:tasks.rados.rados.0.smithi160.stdout:39: oid 39 updating version 0 to 8 2024-09-21T18:07:14.222 INFO:tasks.rados.rados.0.smithi160.stdout:39: oid 39 updating version 8 to 24 2024-09-21T18:07:14.222 INFO:tasks.rados.rados.0.smithi160.stdout:39: oid 39 version 24 is already newer than 23 2024-09-21T18:07:14.223 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 24 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-21T18:07:14.223 INFO:tasks.rados.rados.0.smithi160.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-21T18:07:14.223 INFO:tasks.rados.rados.0.smithi160.stdout:39: done (15 left) 2024-09-21T18:07:14.223 INFO:tasks.rados.rados.0.smithi160.stdout:54: write initial oid 54 2024-09-21T18:07:14.223 INFO:tasks.rados.rados.0.smithi160.stdout:54: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-21T18:07:14.233 INFO:tasks.rados.rados.0.smithi160.stdout:54: writing smithi16030472-54 from 490771 to 1177086 tid 1 2024-09-21T18:07:14.243 INFO:tasks.rados.rados.0.smithi160.stdout:54: writing smithi16030472-54 from 1885075 to 2291421 tid 2 2024-09-21T18:07:14.243 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.243 INFO:tasks.rados.rados.0.smithi160.stdout:36: finishing write tid 3 to smithi16030472-36 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: finishing write tid 4 to smithi16030472-36 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: oid 36 updating version 0 to 32 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: oid 36 version 32 is already newer than 19 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: oid 36 version 32 is already newer than 20 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 32 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:36: done (15 left) 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:55: write initial oid 55 2024-09-21T18:07:14.244 INFO:tasks.rados.rados.0.smithi160.stdout:55: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-21T18:07:14.246 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:206 4.2 4:6ba39595:::smithi16030472-52:head [write 457157~535817 in=535817b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.113398+0000", 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.130012663, 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.130060329, 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 206 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.247 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113398+0000", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113398+0000", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113401+0000", 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2629999999999999e-06 2024-09-21T18:07:14.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113705+0000", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000304498 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113728+0000", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2413999999999999e-05 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.113744+0000", 2024-09-21T18:07:14.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6512e-05 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.219618+0000", 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.105873887 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:207 4.2 4:6ba39595:::smithi16030472-52:head [write 1768111~402677 in=402677b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.122242+0000", 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121167834, 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121319368, 2024-09-21T18:07:14.250 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 207 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122242+0000", 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122242+0000", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122248+0000", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3700000000000003e-06 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122555+0000", 2024-09-21T18:07:14.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030719200000000002 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122581+0000", 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5768000000000001e-05 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122597+0000", 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6685e-05 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221422+0000", 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098824270000000006 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221508+0000", 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6014999999999996e-05 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:208 4.2 4.a9a9c5d6 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.122646+0000", 2024-09-21T18:07:14.254 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.120763814, 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121007737, 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 208 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122646+0000", 2024-09-21T18:07:14.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122646+0000", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122648+0000", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6029999999999999e-06 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122654+0000", 2024-09-21T18:07:14.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7640000000000002e-06 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122664+0000", 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-06 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122671+0000", 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9460000000000001e-06 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:209 4.2 4.a9a9c5d6 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.122680+0000", 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12072996599999999, 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121046944, 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 209 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.258 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122680+0000", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122680+0000", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122681+0000", 2024-09-21T18:07:14.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6700000000000003e-07 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122684+0000", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3869999999999999e-06 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122693+0000", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2849999999999995e-06 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.122698+0000", 2024-09-21T18:07:14.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8100000000000003e-06 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.15.0:46 4.2 e44/33)", 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.223108+0000", 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.020301941, 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020690571000000001, 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.223108+0000", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.223108+0000", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.223111+0000", 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1370000000000002e-06 2024-09-21T18:07:14.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.223123+0000", 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1209999999999999e-05 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.223149+0000", 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6072999999999999e-05 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:215 4.2 4.b68a4bea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.233388+0000", 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.010022031000000001, 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010466949999999999, 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 215 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233388+0000", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233388+0000", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233391+0000", 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2629999999999999e-06 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233732+0000", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000340856 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233752+0000", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233764+0000", 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1671000000000001e-05 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.266 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:216 4.2 4.b68a4bea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.242974+0000", 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.000436718, 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00095168599999999998, 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.267 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 216 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.242974+0000", 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.242974+0000", 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.242978+0000", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9580000000000001e-06 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243178+0000", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020064 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243200+0000", 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2155999999999999e-05 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243216+0000", 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5274999999999999e-05 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:217 4.2 4.b68a4bea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.243254+0000", 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.000156188, 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00074072500000000002, 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.270 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 217 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243254+0000", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243254+0000", 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243255+0000", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.181e-06 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243262+0000", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2580000000000001e-06 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243272+0000", 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0756e-05 2024-09-21T18:07:14.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243281+0000", 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1600000000000004e-06 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:186 4.2 4:638b5b54:::smithi16030472-47:head [write 1992054~651681 in=651681b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.84497408799999996, 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84563492399999995, 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.273 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 186 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398438+0000", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9870000000000002e-06 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398753+0000", 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000314865 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398768+0000", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4508000000000001e-05 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398780+0000", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2867999999999999e-05 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398867+0000", 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6909000000000004e-05 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398876+0000", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3219999999999993e-06 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499811+0000", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100935263 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499889+0000", 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7764999999999998e-05 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.107799+0000", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.60791006800000003 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.107937+0000", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013781500000000001 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.108146+0000", 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000209213 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116917+0000", 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087711100000000004 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:218 4.2 4.b68a4bea (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:14.243293+0000", 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.000117099, 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00091408899999999998, 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:14.279 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 218 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243293+0000", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243293+0000", 2024-09-21T18:07:14.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243294+0000", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2600000000000002e-07 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243299+0000", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0340000000000004e-06 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243308+0000", 2024-09-21T18:07:14.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.628e-06 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.243317+0000", 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3669999999999997e-06 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:187 4.2 4:638b5b54:::smithi16030472-47:head [setxattr _header (56) in=63b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.84461393200000001, 2024-09-21T18:07:14.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84549085700000004, 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 187 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398797+0000", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398803+0000", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1279999999999996e-06 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398810+0000", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9940000000000003e-06 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398815+0000", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3660000000000002e-06 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452294+0000", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.053479323000000002 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452300+0000", 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1539999999999999e-06 2024-09-21T18:07:14.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499948+0000", 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.047647308999999999 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499995+0000", 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7639000000000002e-05 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117698+0000", 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61770323100000002 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117949+0000", 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025021599999999999 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.118289+0000", 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034027699999999999 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:188 4.2 4:638b5b54:::smithi16030472-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.84456239799999999, 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84565097, 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 188 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398849+0000", 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8999999999999996e-07 2024-09-21T18:07:14.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398852+0000", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.959e-06 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398860+0000", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0409999999999998e-06 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.498966+0000", 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100106057 2024-09-21T18:07:14.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599677+0000", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10071127000000001 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599763+0000", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6383000000000003e-05 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119023+0000", 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.51925935400000001 2024-09-21T18:07:14.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119129+0000", 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106513 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:194 4.2 4:798f54f8:::smithi16030472-49:head [write 1709018~655350 in=655350b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.64380211300000001, 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64511094499999999, 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.292 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 194 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599611+0000", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.994e-06 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599807+0000", 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019620299999999999 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599823+0000", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5435999999999999e-05 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699955+0000", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100132287 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700105+0000", 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015034 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700265+0000", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000160011 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.111850+0000", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41158474499999997 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.112103+0000", 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253226 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.112448+0000", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034437399999999999 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117034+0000", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004585754 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:195 4.2 4:798f54f8:::smithi16030472-49:head [setxattr _header (56) in=63b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.297 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.54342146400000002, 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54499621399999998, 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699992+0000", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9799999999999998e-06 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700003+0000", 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1260000000000001e-05 2024-09-21T18:07:14.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700020+0000", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6969999999999998e-05 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700031+0000", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0825999999999999e-05 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700362+0000", 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033120799999999999 2024-09-21T18:07:14.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700439+0000", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6794999999999999e-05 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115214+0000", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41477478200000001 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115458+0000", 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000244536 2024-09-21T18:07:14.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115815+0000", 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035693699999999999 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221530+0000", 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10571459599999999 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:196 4.2 4:798f54f8:::smithi16030472-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.543366711, 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54513837499999995, 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 196 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700045+0000", 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.063e-06 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700050+0000", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.806e-06 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700060+0000", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.764e-06 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700068+0000", 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9829999999999996e-06 2024-09-21T18:07:14.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700524+0000", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045576900000000002 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700661+0000", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137057 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116498+0000", 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41583742800000001 2024-09-21T18:07:14.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116725+0000", 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022619900000000001 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 15 2024-09-21T18:07:14.307 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:14.308 INFO:tasks.rados.rados.0.smithi160.stdout:55: writing smithi16030472-55 from 507578 to 985495 tid 1 2024-09-21T18:07:14.309 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:55: writing smithi16030472-55 from 1543557 to 2093561 tid 2 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:55: writing smithi16030472-55 from 2833665 to 2907578 tid 3 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:53: finishing write tid 1 to smithi16030472-53 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:53: finishing write tid 2 to smithi16030472-53 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:53: finishing write tid 3 to smithi16030472-53 2024-09-21T18:07:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:40: finishing write tid 2 to smithi16030472-40 2024-09-21T18:07:14.325 INFO:tasks.rados.rados.0.smithi160.stdout:47: finishing write tid 2 to smithi16030472-47 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: finishing write tid 4 to smithi16030472-53 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: finishing write tid 5 to smithi16030472-53 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: oid 53 updating version 0 to 22 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: oid 53 updating version 22 to 23 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: oid 53 version 23 is already newer than 20 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:53: oid 53 version 23 is already newer than 21 2024-09-21T18:07:14.348 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 53 v 23 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-21T18:07:14.349 INFO:tasks.rados.rados.0.smithi160.stdout:53: left oid 53 (ObjNum 52 snap 0 seq_num 52) 2024-09-21T18:07:14.349 INFO:tasks.rados.rados.0.smithi160.stdout:53: done (15 left) 2024-09-21T18:07:14.349 INFO:tasks.rados.rados.0.smithi160.stdout:56: write initial oid 56 2024-09-21T18:07:14.349 INFO:tasks.rados.rados.0.smithi160.stdout:56: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-21T18:07:14.354 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:14.353+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:14.358 INFO:tasks.rados.rados.0.smithi160.stdout:56: writing smithi16030472-56 from 524385 to 1077551 tid 1 2024-09-21T18:07:14.367 INFO:tasks.rados.rados.0.smithi160.stdout:56: writing smithi16030472-56 from 1602039 to 2012054 tid 2 2024-09-21T18:07:14.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:14.375 INFO:tasks.rados.rados.0.smithi160.stdout:56: writing smithi16030472-56 from 2577441 to 2924385 tid 3 2024-09-21T18:07:14.375 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.384 INFO:tasks.rados.rados.0.smithi160.stdout:40: finishing write tid 3 to smithi16030472-40 2024-09-21T18:07:14.385 INFO:tasks.rados.rados.0.smithi160.stdout:40: finishing write tid 4 to smithi16030472-40 2024-09-21T18:07:14.385 INFO:tasks.rados.rados.0.smithi160.stdout:40: oid 40 updating version 0 to 26 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:40: oid 40 updating version 26 to 27 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:40: oid 40 version 27 is already newer than 9 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 27 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:40: done (15 left) 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:57: write initial oid 57 2024-09-21T18:07:14.386 INFO:tasks.rados.rados.0.smithi160.stdout:57: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-21T18:07:14.398 INFO:tasks.rados.rados.0.smithi160.stdout:57: writing smithi16030472-57 from 541192 to 1169607 tid 1 2024-09-21T18:07:14.407 INFO:tasks.rados.rados.0.smithi160.stdout:57: writing smithi16030472-57 from 1660521 to 2214194 tid 2 2024-09-21T18:07:14.414 INFO:tasks.rados.rados.0.smithi160.stdout:57: writing smithi16030472-57 from 2888511 to 2941192 tid 3 2024-09-21T18:07:14.414 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.419 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:14.413+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:14.425 INFO:tasks.rados.rados.0.smithi160.stdout:49: finishing write tid 2 to smithi16030472-49 2024-09-21T18:07:14.425 INFO:tasks.rados.rados.0.smithi160.stdout:49: finishing write tid 3 to smithi16030472-49 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: finishing write tid 4 to smithi16030472-49 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: oid 49 updating version 0 to 10 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: oid 49 updating version 10 to 28 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: oid 49 updating version 28 to 29 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 29 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:47: finishing write tid 3 to smithi16030472-47 2024-09-21T18:07:14.426 INFO:tasks.rados.rados.0.smithi160.stdout:49: done (15 left) 2024-09-21T18:07:14.427 INFO:tasks.rados.rados.0.smithi160.stdout:58: write initial oid 58 2024-09-21T18:07:14.427 INFO:tasks.rados.rados.0.smithi160.stdout:58: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-21T18:07:14.440 INFO:tasks.rados.rados.0.smithi160.stdout:58: writing smithi16030472-58 from 557999 to 1261663 tid 1 2024-09-21T18:07:14.449 INFO:tasks.rados.rados.0.smithi160.stdout:58: writing smithi16030472-58 from 1719003 to 2132687 tid 2 2024-09-21T18:07:14.450 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:14.450 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:14.450 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:14.451 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:14.451 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:14.456 INFO:tasks.rados.rados.0.smithi160.stdout:58: writing smithi16030472-58 from 2632287 to 2957999 tid 3 2024-09-21T18:07:14.456 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.465 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:14.477 INFO:tasks.rados.rados.0.smithi160.stdout:56: finishing write tid 1 to smithi16030472-56 2024-09-21T18:07:14.485 INFO:tasks.rados.rados.0.smithi160.stdout:42: finishing write tid 3 to smithi16030472-42 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: finishing write tid 4 to smithi16030472-42 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: oid 42 updating version 0 to 29 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: oid 42 version 29 is already newer than 10 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: oid 42 version 29 is already newer than 11 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 29 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:42: done (15 left) 2024-09-21T18:07:14.486 INFO:tasks.rados.rados.0.smithi160.stdout:59: write initial oid 59 2024-09-21T18:07:14.487 INFO:tasks.rados.rados.0.smithi160.stdout:59: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-21T18:07:14.494 INFO:tasks.rados.rados.0.smithi160.stdout:59: writing smithi16030472-59 from 574806 to 1353719 tid 1 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:59: writing smithi16030472-59 from 2061132 to 2618474 tid 2 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:44: finishing write tid 4 to smithi16030472-44 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:44: oid 44 updating version 0 to 14 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:44: oid 44 version 14 is already newer than 13 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:44: oid 44 version 14 is already newer than 12 2024-09-21T18:07:14.501 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 44 v 14 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T18:07:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-21T18:07:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:44: done (15 left) 2024-09-21T18:07:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:60: write initial oid 60 2024-09-21T18:07:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:60: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-21T18:07:14.507 INFO:tasks.rados.rados.0.smithi160.stdout:60: writing smithi16030472-60 from 591613 to 1045775 tid 1 2024-09-21T18:07:14.508 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:14.508 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:14.513 INFO:tasks.rados.rados.0.smithi160.stdout:60: writing smithi16030472-60 from 1719614 to 2136967 tid 2 2024-09-21T18:07:14.517 INFO:tasks.rados.rados.0.smithi160.stdout:60: writing smithi16030472-60 from 2570780 to 2991613 tid 3 2024-09-21T18:07:14.517 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.518 INFO:tasks.rados.rados.0.smithi160.stdout:45: finishing write tid 2 to smithi16030472-45 2024-09-21T18:07:14.518 INFO:tasks.rados.rados.0.smithi160.stdout:45: finishing write tid 3 to smithi16030472-45 2024-09-21T18:07:14.519 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:14.517+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:14.545 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:14.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:14.566 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:14.566 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: finishing write tid 3 to smithi16030472-35 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: finishing write tid 4 to smithi16030472-35 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: oid 35 updating version 0 to 17 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: oid 35 updating version 17 to 18 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: oid 35 updating version 18 to 36 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 36 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: finishing write tid 2 to smithi16030472-37 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: finishing write tid 3 to smithi16030472-37 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: finishing write tid 4 to smithi16030472-37 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: oid 37 updating version 0 to 21 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: oid 37 updating version 21 to 38 2024-09-21T18:07:14.579 INFO:tasks.rados.rados.0.smithi160.stdout:37: oid 37 version 38 is already newer than 37 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 37 v 38 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:35: done (15 left) 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:37: done (14 left) 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:61: write initial oid 61 2024-09-21T18:07:14.580 INFO:tasks.rados.rados.0.smithi160.stdout:61: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-21T18:07:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:14.588 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:14.589 INFO:tasks.rados.rados.0.smithi160.stdout:61: writing smithi16030472-61 from 608420 to 1137831 tid 1 2024-09-21T18:07:14.597 INFO:tasks.rados.rados.0.smithi160.stdout:61: writing smithi16030472-61 from 1778096 to 2339107 tid 2 2024-09-21T18:07:14.602 INFO:tasks.rados.rados.0.smithi160.stdout:61: writing smithi16030472-61 from 2881850 to 3008420 tid 3 2024-09-21T18:07:14.602 INFO:tasks.rados.rados.0.smithi160.stdout:62: write initial oid 62 2024-09-21T18:07:14.602 INFO:tasks.rados.rados.0.smithi160.stdout:62: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:138 3.3 3:cd35ea38:::smithi16030472-35:head [write 1806623~629667] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.606 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.245361+0000", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.3560438669999999, 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.75979732200000005, 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245361+0000", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245361+0000", 2024-09-21T18:07:14.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245364+0000", 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245698+0000", 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033420699999999999 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245726+0000", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8198999999999999e-05 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245747+0000", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0112999999999999e-05 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.457629+0000", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21188206300000001 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.457769+0000", 2024-09-21T18:07:14.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140474 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.457976+0000", 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020663300000000001 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.564587+0000", 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.106611607 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.004831+0000", 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44024371000000001 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.004945+0000", 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000114021 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.005158+0000", 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000212761 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:69 3.3 3:cd35ea38:::smithi16030472-35:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.245766+0000", 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.3556385470000001, 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.75978079099999996, 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245766+0000", 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245766+0000", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245767+0000", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.61e-06 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245772+0000", 2024-09-21T18:07:14.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9860000000000002e-06 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245786+0000", 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.395e-05 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245798+0000", 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1666e-05 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.460577+0000", 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21477853799999999 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.460676+0000", 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8948000000000002e-05 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.005293+0000", 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54461697099999995 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.005419+0000", 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126005 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.005547+0000", 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012811699999999999 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:70 3.3 3:cd35ea38:::smithi16030472-35:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.616 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.245813+0000", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.3555918810000001, 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.874886214, 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245813+0000", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245813+0000", 2024-09-21T18:07:14.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245814+0000", 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245818+0000", 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9439999999999996e-06 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245829+0000", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0849000000000001e-05 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.345975+0000", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10014598199999999 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.461332+0000", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.115357181 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.461406+0000", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.347e-05 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117657+0000", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65625144800000001 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117742+0000", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4430000000000003e-05 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.120699+0000", 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002956874 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:150 4.2 4:4dd833f5:::smithi16030472-38:head [write 1865716~756143] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.257520+0000", 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.343884198, 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76105703000000002, 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 150 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257520+0000", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257520+0000", 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257523+0000", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2440000000000002e-06 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257783+0000", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025944100000000001 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257796+0000", 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2690999999999999e-05 2024-09-21T18:07:14.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257807+0000", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1022000000000001e-05 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257938+0000", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013176 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257981+0000", 2024-09-21T18:07:14.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2395999999999999e-05 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.708494+0000", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.45051332500000002 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.708731+0000", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000237295 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.709114+0000", 2024-09-21T18:07:14.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038243700000000001 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.817554+0000", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.108440016 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018480+0000", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20092665700000001 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018532+0000", 2024-09-21T18:07:14.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1489000000000001e-05 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018577+0000", 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5256999999999997e-05 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:151 4.2 4:4dd833f5:::smithi16030472-38:head [setxattr _header (56) in=7b,truncate 2621859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.257830+0000", 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.3435745379999999, 2024-09-21T18:07:14.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76099773100000001, 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 151 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257830+0000", 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257830+0000", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257831+0000", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4e-07 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257836+0000", 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5140000000000001e-06 2024-09-21T18:07:14.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257850+0000", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4083999999999999e-05 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257858+0000", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0330000000000008e-06 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258023+0000", 2024-09-21T18:07:14.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000164964 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258075+0000", 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1805999999999998e-05 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.711942+0000", 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.45386691600000001 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.712107+0000", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016519399999999999 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.712343+0000", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000236126 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.917916+0000", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20557215400000001 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018660+0000", 2024-09-21T18:07:14.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10074480600000001 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018736+0000", 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5798999999999997e-05 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018828+0000", 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1395000000000002e-05 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:152 4.2 4:4dd833f5:::smithi16030472-38:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.257885+0000", 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.343519822, 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76120679899999999, 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 152 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.634 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257885+0000", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257885+0000", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257885+0000", 2024-09-21T18:07:14.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7199999999999997e-07 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257891+0000", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5110000000000003e-06 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257902+0000", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1513e-05 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.257911+0000", 2024-09-21T18:07:14.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5150000000000006e-06 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258098+0000", 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018718399999999999 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258125+0000", 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6934e-05 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018221+0000", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76009612999999998 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018384+0000", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016302299999999999 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018901+0000", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00051699500000000002 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018990+0000", 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9190999999999994e-05 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.019091+0000", 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100931 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:154 4.2 4:6abed166:::smithi16030472-39:head [write 1924198~637005] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.258147+0000", 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.343257817, 2024-09-21T18:07:14.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96218858399999996, 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 154 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258147+0000", 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258147+0000", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258148+0000", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0330000000000001e-06 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258359+0000", 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021188199999999999 2024-09-21T18:07:14.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258369+0000", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4089999999999992e-06 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258377+0000", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2949999999999994e-06 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258419+0000", 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2187000000000001e-05 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258452+0000", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2354999999999998e-05 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.714806+0000", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.45635441100000002 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.714975+0000", 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000168834 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.715158+0000", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018293399999999999 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.917934+0000", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20277631300000001 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220160+0000", 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30222607000000001 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220252+0000", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1238999999999998e-05 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220335+0000", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3622000000000007e-05 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:155 4.2 4:6abed166:::smithi16030472-39:head [setxattr _header (56) in=7b,truncate 2638666] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.258394+0000", 2024-09-21T18:07:14.645 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.343010045, 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.96247694900000003, 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 155 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258394+0000", 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258394+0000", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258395+0000", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0299999999999997e-07 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258402+0000", 2024-09-21T18:07:14.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9020000000000002e-06 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.258411+0000", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9539999999999993e-06 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358501+0000", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100089998 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358590+0000", 2024-09-21T18:07:14.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9155999999999998e-05 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358627+0000", 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7163999999999999e-05 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.817070+0000", 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.45844216399999999 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.817164+0000", 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4011999999999994e-05 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.817282+0000", 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000118113 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.018407+0000", 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201125572 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220517+0000", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20210992699999999 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220682+0000", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016458799999999999 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.220871+0000", 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018949599999999999 2024-09-21T18:07:14.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:73 3.3 3:d903dcdd:::smithi16030472-36:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.346592+0000", 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2548120039999999, 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.759795573, 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346592+0000", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346592+0000", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346593+0000", 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.083e-06 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346607+0000", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.332e-05 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346620+0000", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2877000000000001e-05 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346629+0000", 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5300000000000002e-06 2024-09-21T18:07:14.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.564490+0000", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21786082900000001 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.564571+0000", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1298000000000007e-05 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.106117+0000", 2024-09-21T18:07:14.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54154530999999995 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.106248+0000", 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013168199999999999 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.106388+0000", 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013964399999999999 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:74 3.3 3:d903dcdd:::smithi16030472-36:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.346640+0000", 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.254764598, 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.76509126400000005, 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346640+0000", 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346640+0000", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346641+0000", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346646+0000", 2024-09-21T18:07:14.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3430000000000003e-06 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346655+0000", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6770000000000001e-06 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.446745+0000", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10008996000000001 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.664721+0000", 2024-09-21T18:07:14.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21797601 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.664799+0000", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8689000000000005e-05 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.106537+0000", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44173791200000001 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.106731+0000", 2024-09-21T18:07:14.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019398000000000001 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.111731+0000", 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0049997330000000001 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:156 4.2 4:6abed166:::smithi16030472-39:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.358557+0000", 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.242847823, 2024-09-21T18:07:14.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86279214000000004, 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 156 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358557+0000", 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358557+0000", 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358561+0000", 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7199999999999997e-06 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358578+0000", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6857000000000001e-05 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358607+0000", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8682000000000001e-05 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358627+0000", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9899999999999999e-05 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358683+0000", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5813999999999998e-05 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.358707+0000", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4584999999999999e-05 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.917977+0000", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.55927021399999999 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.918067+0000", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0109000000000001e-05 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221018+0000", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30295008800000001 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221189+0000", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017182699999999999 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221349+0000", 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015934400000000001 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:185 4.2 4:638b5b54:::smithi16030472-47:head [write 773122~559572,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.389794+0000", 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2116103469999999, 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71564153200000002, 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.667 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 185 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.389794+0000", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.389794+0000", 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.389798+0000", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.827e-06 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.390073+0000", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00027522499999999998 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.390091+0000", 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8379000000000001e-05 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.390104+0000", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.390169+0000", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4787000000000002e-05 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op must wait for map", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.390174+0000", 2024-09-21T18:07:14.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6299999999999997e-06 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499315+0000", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10914103999999999 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.105427+0000", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.60611231099999996 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.105436+0000", 2024-09-21T18:07:14.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6170000000000007e-06 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:186 4.2 4:638b5b54:::smithi16030472-47:head [write 1992054~651681 in=651681b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2029682100000001, 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.92569188199999997, 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 186 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398436+0000", 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398438+0000", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9870000000000002e-06 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398753+0000", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000314865 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398768+0000", 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4508000000000001e-05 2024-09-21T18:07:14.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398780+0000", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2867999999999999e-05 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398867+0000", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6909000000000004e-05 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398876+0000", 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3219999999999993e-06 2024-09-21T18:07:14.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499811+0000", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100935263 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499889+0000", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7764999999999998e-05 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.107799+0000", 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.60791006800000003 2024-09-21T18:07:14.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.107937+0000", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013781500000000001 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.108146+0000", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000209213 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116917+0000", 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0087711100000000004 2024-09-21T18:07:14.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.323980+0000", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20706275800000001 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.324062+0000", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2274999999999994e-05 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.324128+0000", 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6155999999999995e-05 2024-09-21T18:07:14.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:187 4.2 4:638b5b54:::smithi16030472-47:head [setxattr _header (56) in=63b,truncate 2773122] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2026080539999999, 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0267261430000001, 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.679 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 187 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398796+0000", 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398797+0000", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398803+0000", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1279999999999996e-06 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398810+0000", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9940000000000003e-06 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398815+0000", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3660000000000002e-06 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452294+0000", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.053479323000000002 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.452300+0000", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1539999999999999e-06 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499948+0000", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.047647308999999999 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.499995+0000", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7639000000000002e-05 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117698+0000", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.61770323100000002 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117949+0000", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025021599999999999 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.118289+0000", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034027699999999999 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.322123+0000", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.203834458 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425415+0000", 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103291813 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425469+0000", 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3461000000000002e-05 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425522+0000", 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3837e-05 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:188 4.2 4:638b5b54:::smithi16030472-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2025565199999999, 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1271722230000001, 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 188 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.686 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398849+0000", 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8999999999999996e-07 2024-09-21T18:07:14.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398852+0000", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.959e-06 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398860+0000", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0409999999999998e-06 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.498966+0000", 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100106057 2024-09-21T18:07:14.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599677+0000", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10071127000000001 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599763+0000", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6383000000000003e-05 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119023+0000", 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.51925935400000001 2024-09-21T18:07:14.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119129+0000", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106513 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.525671+0000", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40654210099999999 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.525825+0000", 2024-09-21T18:07:14.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015378899999999999 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.526020+0000", 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019506600000000001 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:145 3.3 3:f074207e:::smithi16030472-37:head [write 605052~774376,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.446775+0000", 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.1546297830000001, 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66523603099999995, 2024-09-21T18:07:14.691 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.446775+0000", 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.446775+0000", 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.446778+0000", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3680000000000002e-06 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447250+0000", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047159399999999998 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447270+0000", 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0214999999999999e-05 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447281+0000", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0815999999999999e-05 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.664855+0000", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.21757442099999999 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.664953+0000", 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8257000000000005e-05 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.665057+0000", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103849 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.667243+0000", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021854729999999998 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.111834+0000", 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.444591454 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.111910+0000", 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5672000000000003e-05 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.112011+0000", 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100912 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:77 3.3 3:f074207e:::smithi16030472-37:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:14.696 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.447325+0000", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.1540797490000001, 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.67027479099999998, 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447325+0000", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447325+0000", 2024-09-21T18:07:14.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447326+0000", 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447330+0000", 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7520000000000004e-06 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447337+0000", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0709999999999997e-06 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447343+0000", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7740000000000001e-06 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.666297+0000", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.218953485 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.666331+0000", 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3973999999999997e-05 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.114742+0000", 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.44841165900000002 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.114933+0000", 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019044 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117599+0000", 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0026667589999999999 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:194 4.2 4:798f54f8:::smithi16030472-49:head [write 1709018~655350 in=655350b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.001796235, 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.82525414600000002, 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.701 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 194 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599608+0000", 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599611+0000", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.994e-06 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599807+0000", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019620299999999999 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599823+0000", 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5435999999999999e-05 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699955+0000", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100132287 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700105+0000", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015034 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700265+0000", 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000160011 2024-09-21T18:07:14.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.111850+0000", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41158474499999997 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.112103+0000", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253226 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.112448+0000", 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034437399999999999 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.117034+0000", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.004585754 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.424671+0000", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30763706899999999 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.424784+0000", 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011349900000000001 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.424862+0000", 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8207999999999994e-05 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:195 4.2 4:798f54f8:::smithi16030472-49:head [setxattr _header (56) in=63b,truncate 2806736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.90141558600000005, 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72509203200000005, 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.707 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 195 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699989+0000", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.699992+0000", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9799999999999998e-06 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700003+0000", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1260000000000001e-05 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700020+0000", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6969999999999998e-05 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700031+0000", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0825999999999999e-05 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700362+0000", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033120799999999999 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700439+0000", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6794999999999999e-05 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115214+0000", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41477478200000001 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115458+0000", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000244536 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:14.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.115815+0000", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035693699999999999 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.221530+0000", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10571459599999999 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.424957+0000", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.203426881 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:14.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425017+0000", 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0912000000000002e-05 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425081+0000", 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3349000000000006e-05 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:196 4.2 4:798f54f8:::smithi16030472-49:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.713 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.90136083300000003, 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72531913800000003, 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 196 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700044+0000", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700045+0000", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.063e-06 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700050+0000", 2024-09-21T18:07:14.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.806e-06 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700060+0000", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.764e-06 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700068+0000", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9829999999999996e-06 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700524+0000", 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045576900000000002 2024-09-21T18:07:14.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.700661+0000", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000137057 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116498+0000", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41583742800000001 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.116725+0000", 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022619900000000001 2024-09-21T18:07:14.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425158+0000", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30843352800000001 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425264+0000", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010631700000000001 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.425363+0000", 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8224e-05 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.718 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:14.719 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:14.719 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:14.720 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:14.720 INFO:tasks.rados.rados.0.smithi160.stdout:62: writing smithi16030472-62 from 625227 to 1346240 tid 1 2024-09-21T18:07:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:14.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:14.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:14.722 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:14.723 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:14.724 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:14.724 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:14.725 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:14.725 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:62: writing smithi16030472-62 from 1952931 to 2657600 tid 2 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: finishing write tid 4 to smithi16030472-45 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: oid 45 updating version 0 to 32 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: oid 45 updating version 32 to 33 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: oid 45 version 33 is already newer than 15 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 45 v 33 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-21T18:07:14.726 INFO:tasks.rados.rados.0.smithi160.stdout:45: done (15 left) 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:63: write initial oid 63 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:63: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:63: writing smithi16030472-63 from 642034 to 1438296 tid 1 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:63: writing smithi16030472-63 from 1895060 to 2459740 tid 2 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:46: finishing write tid 3 to smithi16030472-46 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: finishing write tid 3 to smithi16030472-48 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: finishing write tid 4 to smithi16030472-48 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: oid 48 updating version 0 to 19 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: oid 48 version 19 is already newer than 18 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: oid 48 updating version 19 to 37 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 48 v 37 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:50: finishing write tid 2 to smithi16030472-50 2024-09-21T18:07:14.727 INFO:tasks.rados.rados.0.smithi160.stdout:47: finishing write tid 4 to smithi16030472-47 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:47: oid 47 updating version 0 to 26 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:47: oid 47 version 26 is already newer than 9 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:47: oid 47 updating version 26 to 30 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 30 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:52: finishing write tid 1 to smithi16030472-52 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:47: done (15 left) 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:48: done (14 left) 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:64: write initial oid 64 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:64: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:64: writing smithi16030472-64 from 658841 to 1130352 tid 1 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:64: writing smithi16030472-64 from 1553542 to 2261880 tid 2 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:64: writing smithi16030472-64 from 2964119 to 3058841 tid 3 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:65: write initial oid 65 2024-09-21T18:07:14.728 INFO:tasks.rados.rados.0.smithi160.stdout:65: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-21T18:07:14.729 INFO:tasks.rados.rados.0.smithi160.stdout:65: writing smithi16030472-65 from 675648 to 1222408 tid 1 2024-09-21T18:07:14.729 INFO:tasks.rados.rados.0.smithi160.stdout:65: writing smithi16030472-65 from 2012024 to 2580373 tid 2 2024-09-21T18:07:14.729 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.729 INFO:tasks.rados.rados.0.smithi160.stdout:50: finishing write tid 3 to smithi16030472-50 2024-09-21T18:07:14.729 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:14.715+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:14.732 INFO:tasks.rados.rados.0.smithi160.stdout:54: finishing write tid 1 to smithi16030472-54 2024-09-21T18:07:14.789 INFO:tasks.rados.rados.0.smithi160.stdout:50: finishing write tid 4 to smithi16030472-50 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:50: oid 50 updating version 0 to 20 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:50: oid 50 updating version 20 to 39 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:50: oid 50 updating version 39 to 40 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 50 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:50: done (15 left) 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:66: write initial oid 66 2024-09-21T18:07:14.790 INFO:tasks.rados.rados.0.smithi160.stdout:66: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-21T18:07:14.796 INFO:tasks.rados.rados.0.smithi160.stdout:66: writing smithi16030472-66 from 692455 to 1314464 tid 1 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:66: writing smithi16030472-66 from 2070506 to 2782513 tid 2 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: finishing write tid 4 to smithi16030472-46 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: oid 46 updating version 0 to 17 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: oid 46 updating version 17 to 35 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: oid 46 version 35 is already newer than 16 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 35 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-21T18:07:14.805 INFO:tasks.rados.rados.0.smithi160.stdout:46: done (15 left) 2024-09-21T18:07:14.806 INFO:tasks.rados.rados.0.smithi160.stdout:67: write initial oid 67 2024-09-21T18:07:14.806 INFO:tasks.rados.rados.0.smithi160.stdout:67: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-21T18:07:14.814 INFO:tasks.rados.rados.0.smithi160.stdout:67: writing smithi16030472-67 from 709262 to 1406520 tid 1 2024-09-21T18:07:14.821 INFO:tasks.rados.rados.0.smithi160.stdout:67: writing smithi16030472-67 from 2012635 to 2584653 tid 2 2024-09-21T18:07:14.821 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:14.817+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:14.824 INFO:tasks.rados.rados.0.smithi160.stdout:67: writing smithi16030472-67 from 3046388 to 3109262 tid 3 2024-09-21T18:07:14.825 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:14.848 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:14.848 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:14.854 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:14.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:14.922 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:14.919+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: finishing write tid 2 to smithi16030472-43 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: finishing write tid 3 to smithi16030472-43 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:56: finishing write tid 2 to smithi16030472-56 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: finishing write tid 4 to smithi16030472-43 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: oid 43 updating version 0 to 22 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: oid 43 updating version 22 to 40 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: oid 43 version 40 is already newer than 39 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 43 v 40 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:56: finishing write tid 3 to smithi16030472-56 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:57: finishing write tid 1 to smithi16030472-57 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:57: finishing write tid 2 to smithi16030472-57 2024-09-21T18:07:14.944 INFO:tasks.rados.rados.0.smithi160.stdout:57: finishing write tid 3 to smithi16030472-57 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: finishing write tid 4 to smithi16030472-57 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: finishing write tid 5 to smithi16030472-57 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: oid 57 updating version 0 to 26 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: oid 57 version 26 is already newer than 25 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: oid 57 version 26 is already newer than 24 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: oid 57 updating version 26 to 27 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 57 v 27 (ObjNum 56 snap 0 seq_num 56) dirty exists 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:57: left oid 57 (ObjNum 56 snap 0 seq_num 56) 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: finishing write tid 4 to smithi16030472-56 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: finishing write tid 5 to smithi16030472-56 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: oid 56 updating version 0 to 24 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: oid 56 version 24 is already newer than 23 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: oid 56 updating version 24 to 26 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:56: oid 56 version 26 is already newer than 25 2024-09-21T18:07:14.945 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 26 (ObjNum 55 snap 0 seq_num 55) dirty exists 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:56: left oid 56 (ObjNum 55 snap 0 seq_num 55) 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:58: finishing write tid 1 to smithi16030472-58 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:43: done (15 left) 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:56: done (14 left) 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:57: done (13 left) 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:68: write initial oid 68 2024-09-21T18:07:14.946 INFO:tasks.rados.rados.0.smithi160.stdout:68: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-21T18:07:14.955 INFO:tasks.rados.rados.0.smithi160.stdout:68: writing smithi16030472-68 from 726069 to 1498576 tid 1 2024-09-21T18:07:14.964 INFO:tasks.rados.rados.0.smithi160.stdout:68: writing smithi16030472-68 from 2071117 to 2786793 tid 2 2024-09-21T18:07:14.964 INFO:tasks.rados.rados.0.smithi160.stdout:69: write initial oid 69 2024-09-21T18:07:14.964 INFO:tasks.rados.rados.0.smithi160.stdout:69: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-21T18:07:14.972 INFO:tasks.rados.rados.0.smithi160.stdout:69: writing smithi16030472-69 from 742876 to 1190632 tid 1 2024-09-21T18:07:14.979 INFO:tasks.rados.rados.0.smithi160.stdout:69: writing smithi16030472-69 from 1729599 to 2305286 tid 2 2024-09-21T18:07:14.985 INFO:tasks.rados.rados.0.smithi160.stdout:69: writing smithi16030472-69 from 3101234 to 3142876 tid 3 2024-09-21T18:07:14.985 INFO:tasks.rados.rados.0.smithi160.stdout:70: write initial oid 70 2024-09-21T18:07:14.985 INFO:tasks.rados.rados.0.smithi160.stdout:70: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-21T18:07:14.995 INFO:tasks.rados.rados.0.smithi160.stdout:70: writing smithi16030472-70 from 759683 to 1399041 tid 1 2024-09-21T18:07:15.003 INFO:tasks.rados.rados.0.smithi160.stdout:70: writing smithi16030472-70 from 1904434 to 2623779 tid 2 2024-09-21T18:07:15.006 INFO:tasks.rados.rados.0.smithi160.stdout:70: writing smithi16030472-70 from 3128657 to 3159683 tid 3 2024-09-21T18:07:15.007 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.007 INFO:tasks.rados.rados.0.smithi160.stdout:65: finishing write tid 1 to smithi16030472-65 2024-09-21T18:07:15.023 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:15.019+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:15.031 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.032 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:15.047 INFO:tasks.rados.rados.0.smithi160.stdout:64: finishing write tid 1 to smithi16030472-64 2024-09-21T18:07:15.050 INFO:tasks.rados.rados.0.smithi160.stdout:64: finishing write tid 2 to smithi16030472-64 2024-09-21T18:07:15.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:15.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:15.089 INFO:tasks.rados.rados.0.smithi160.stdout:65: finishing write tid 2 to smithi16030472-65 2024-09-21T18:07:15.118 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:15.119 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:15.124 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:15.119+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: finishing write tid 2 to smithi16030472-58 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: finishing write tid 3 to smithi16030472-58 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: finishing write tid 4 to smithi16030472-58 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: finishing write tid 5 to smithi16030472-58 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: oid 58 updating version 0 to 31 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: oid 58 version 31 is already newer than 30 2024-09-21T18:07:15.147 INFO:tasks.rados.rados.0.smithi160.stdout:58: oid 58 version 31 is already newer than 28 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:58: oid 58 version 31 is already newer than 29 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 58 v 31 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:58: left oid 58 (ObjNum 57 snap 0 seq_num 57) 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:58: done (15 left) 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:71: write initial oid 71 2024-09-21T18:07:15.148 INFO:tasks.rados.rados.0.smithi160.stdout:71: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-21T18:07:15.157 INFO:tasks.rados.rados.0.smithi160.stdout:71: writing smithi16030472-71 from 776490 to 1491097 tid 1 2024-09-21T18:07:15.159 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:15.166 INFO:tasks.rados.rados.0.smithi160.stdout:71: writing smithi16030472-71 from 2246563 to 2825919 tid 2 2024-09-21T18:07:15.166 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.166 INFO:tasks.rados.rados.0.smithi160.stdout:54: finishing write tid 2 to smithi16030472-54 2024-09-21T18:07:15.166 INFO:tasks.rados.rados.0.smithi160.stdout:64: finishing write tid 3 to smithi16030472-64 2024-09-21T18:07:15.166 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.167 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:15.185 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:15.186 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:15.197 INFO:tasks.rados.rados.0.smithi160.stdout:69: finishing write tid 1 to smithi16030472-69 2024-09-21T18:07:15.197 INFO:tasks.rados.rados.0.smithi160.stdout:69: finishing write tid 2 to smithi16030472-69 2024-09-21T18:07:15.202 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:07:15.204 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:15.199+0000 7f728550b640 -1 osd.14 45 handle_osd_map injecting map crc failure 2024-09-21T18:07:15.204 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:15.199+0000 7f94db546640 -1 osd.15 45 handle_osd_map injecting map crc failure 2024-09-21T18:07:15.206 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.15 to 1 (10000) 2024-09-21T18:07:15.216 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:15.216 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:15.225 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:15.221+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:15.236 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-09-21T18:07:15.236 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-09-21T18:07:15.236 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T18:07:15.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:07:15.241 INFO:tasks.rados.rados.0.smithi160.stdout:54: finishing write tid 3 to smithi16030472-54 2024-09-21T18:07:15.241 INFO:tasks.rados.rados.0.smithi160.stdout:55: finishing write tid 1 to smithi16030472-55 2024-09-21T18:07:15.242 INFO:tasks.rados.rados.0.smithi160.stdout:54: finishing write tid 4 to smithi16030472-54 2024-09-21T18:07:15.242 INFO:tasks.rados.rados.0.smithi160.stdout:54: oid 54 updating version 0 to 12 2024-09-21T18:07:15.242 INFO:tasks.rados.rados.0.smithi160.stdout:54: oid 54 updating version 12 to 33 2024-09-21T18:07:15.242 INFO:tasks.rados.rados.0.smithi160.stdout:54: oid 54 updating version 33 to 34 2024-09-21T18:07:15.243 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 34 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-21T18:07:15.243 INFO:tasks.rados.rados.0.smithi160.stdout:54: left oid 54 (ObjNum 53 snap 0 seq_num 53) 2024-09-21T18:07:15.243 INFO:tasks.rados.rados.0.smithi160.stdout:54: done (15 left) 2024-09-21T18:07:15.243 INFO:tasks.rados.rados.0.smithi160.stdout:72: write initial oid 72 2024-09-21T18:07:15.243 INFO:tasks.rados.rados.0.smithi160.stdout:72: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-21T18:07:15.254 INFO:tasks.rados.rados.0.smithi160.stdout:72: writing smithi16030472-72 from 793297 to 1583153 tid 1 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:72: writing smithi16030472-72 from 2305045 to 3028059 tid 2 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: finishing write tid 4 to smithi16030472-64 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: finishing write tid 5 to smithi16030472-64 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: oid 64 updating version 0 to 29 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: oid 64 version 29 is already newer than 27 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: oid 64 version 29 is already newer than 28 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: oid 64 updating version 29 to 30 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 64 v 30 (ObjNum 63 snap 0 seq_num 63) dirty exists 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: left oid 64 (ObjNum 63 snap 0 seq_num 63) 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:64: done (15 left) 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:73: write initial oid 73 2024-09-21T18:07:15.267 INFO:tasks.rados.rados.0.smithi160.stdout:73: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-21T18:07:15.273 INFO:tasks.rados.rados.0.smithi160.stdout:73: writing smithi16030472-73 from 410104 to 875209 tid 1 2024-09-21T18:07:15.283 INFO:tasks.rados.rados.0.smithi160.stdout:73: writing smithi16030472-73 from 1563527 to 2146552 tid 2 2024-09-21T18:07:15.291 INFO:tasks.rados.rados.0.smithi160.stdout:73: writing smithi16030472-73 from 2810926 to 3210104 tid 3 2024-09-21T18:07:15.291 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.292 INFO:tasks.rados.rados.0.smithi160.stdout:61: finishing write tid 1 to smithi16030472-61 2024-09-21T18:07:15.292 INFO:tasks.rados.rados.0.smithi160.stdout:61: finishing write tid 2 to smithi16030472-61 2024-09-21T18:07:15.292 INFO:tasks.rados.rados.0.smithi160.stdout:61: finishing write tid 3 to smithi16030472-61 2024-09-21T18:07:15.318 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.318 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.326 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:15.323+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:15.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:15.350 INFO:tasks.rados.rados.0.smithi160.stdout:52: finishing write tid 2 to smithi16030472-52 2024-09-21T18:07:15.351 INFO:tasks.rados.rados.0.smithi160.stdout:52: finishing write tid 3 to smithi16030472-52 2024-09-21T18:07:15.351 INFO:tasks.rados.rados.0.smithi160.stdout:52: finishing write tid 4 to smithi16030472-52 2024-09-21T18:07:15.351 INFO:tasks.rados.rados.0.smithi160.stdout:52: oid 52 updating version 0 to 36 2024-09-21T18:07:15.351 INFO:tasks.rados.rados.0.smithi160.stdout:52: oid 52 version 36 is already newer than 35 2024-09-21T18:07:15.351 INFO:tasks.rados.rados.0.smithi160.stdout:52: oid 52 version 36 is already newer than 11 2024-09-21T18:07:15.352 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 52 v 36 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-21T18:07:15.352 INFO:tasks.rados.rados.0.smithi160.stdout:52: left oid 52 (ObjNum 51 snap 0 seq_num 51) 2024-09-21T18:07:15.352 INFO:tasks.rados.rados.0.smithi160.stdout:52: done (15 left) 2024-09-21T18:07:15.352 INFO:tasks.rados.rados.0.smithi160.stdout:74: write initial oid 74 2024-09-21T18:07:15.352 INFO:tasks.rados.rados.0.smithi160.stdout:74: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-21T18:07:15.360 INFO:tasks.rados.rados.0.smithi160.stdout:74: writing smithi16030472-74 from 426911 to 967265 tid 1 2024-09-21T18:07:15.371 INFO:tasks.rados.rados.0.smithi160.stdout:74: writing smithi16030472-74 from 1622009 to 2348692 tid 2 2024-09-21T18:07:15.378 INFO:tasks.rados.rados.0.smithi160.stdout:74: writing smithi16030472-74 from 3121996 to 3226911 tid 3 2024-09-21T18:07:15.378 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.403 INFO:tasks.rados.rados.0.smithi160.stdout:65: finishing write tid 3 to smithi16030472-65 2024-09-21T18:07:15.403 INFO:tasks.rados.rados.0.smithi160.stdout:65: finishing write tid 4 to smithi16030472-65 2024-09-21T18:07:15.403 INFO:tasks.rados.rados.0.smithi160.stdout:65: oid 65 updating version 0 to 22 2024-09-21T18:07:15.403 INFO:tasks.rados.rados.0.smithi160.stdout:65: oid 65 version 22 is already newer than 21 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:65: oid 65 updating version 22 to 42 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 65 v 42 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:65: left oid 65 (ObjNum 64 snap 0 seq_num 64) 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:65: done (15 left) 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:75: write initial oid 75 2024-09-21T18:07:15.404 INFO:tasks.rados.rados.0.smithi160.stdout:75: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-21T18:07:15.412 INFO:tasks.rados.rados.0.smithi160.stdout:75: writing smithi16030472-75 from 443718 to 1059321 tid 1 2024-09-21T18:07:15.420 INFO:tasks.rados.rados.0.smithi160.stdout:75: writing smithi16030472-75 from 1564138 to 2034479 tid 2 2024-09-21T18:07:15.427 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:15.425+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:15.431 INFO:tasks.rados.rados.0.smithi160.stdout:75: writing smithi16030472-75 from 2633066 to 3243718 tid 3 2024-09-21T18:07:15.431 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.431 INFO:tasks.rados.rados.0.smithi160.stdout:69: finishing write tid 3 to smithi16030472-69 2024-09-21T18:07:15.431 INFO:tasks.rados.rados.0.smithi160.stdout:69: finishing write tid 4 to smithi16030472-69 2024-09-21T18:07:15.479 INFO:tasks.rados.rados.0.smithi160.stdout:59: finishing write tid 1 to smithi16030472-59 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:60: finishing write tid 1 to smithi16030472-60 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:62: finishing write tid 1 to smithi16030472-62 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:67: finishing write tid 1 to smithi16030472-67 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: finishing write tid 2 to smithi16030472-55 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: finishing write tid 3 to smithi16030472-55 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: finishing write tid 4 to smithi16030472-55 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: finishing write tid 5 to smithi16030472-55 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: oid 55 updating version 0 to 39 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: oid 55 version 39 is already newer than 38 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: oid 55 version 39 is already newer than 13 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:55: oid 55 updating version 39 to 40 2024-09-21T18:07:15.480 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 55 v 40 (ObjNum 54 snap 0 seq_num 54) dirty exists 2024-09-21T18:07:15.481 INFO:tasks.rados.rados.0.smithi160.stdout:55: left oid 55 (ObjNum 54 snap 0 seq_num 54) 2024-09-21T18:07:15.481 INFO:tasks.rados.rados.0.smithi160.stdout:55: done (15 left) 2024-09-21T18:07:15.481 INFO:tasks.rados.rados.0.smithi160.stdout:76: write initial oid 76 2024-09-21T18:07:15.481 INFO:tasks.rados.rados.0.smithi160.stdout:76: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-21T18:07:15.489 INFO:tasks.rados.rados.0.smithi160.stdout:76: writing smithi16030472-76 from 460525 to 1151377 tid 1 2024-09-21T18:07:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.498 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.499 INFO:tasks.rados.rados.0.smithi160.stdout:76: writing smithi16030472-76 from 1622620 to 2352972 tid 2 2024-09-21T18:07:15.507 INFO:tasks.rados.rados.0.smithi160.stdout:76: writing smithi16030472-76 from 3060489 to 3260525 tid 3 2024-09-21T18:07:15.507 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.507 INFO:tasks.rados.rados.0.smithi160.stdout:73: finishing write tid 1 to smithi16030472-73 2024-09-21T18:07:15.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:15.523 INFO:tasks.rados.rados.0.smithi160.stdout:71: finishing write tid 1 to smithi16030472-71 2024-09-21T18:07:15.523 INFO:tasks.rados.rados.0.smithi160.stdout:71: finishing write tid 2 to smithi16030472-71 2024-09-21T18:07:15.523 INFO:tasks.rados.rados.0.smithi160.stdout:61: finishing write tid 4 to smithi16030472-61 2024-09-21T18:07:15.523 INFO:tasks.rados.rados.0.smithi160.stdout:61: finishing write tid 5 to smithi16030472-61 2024-09-21T18:07:15.523 INFO:tasks.rados.rados.0.smithi160.stdout:61: oid 61 updating version 0 to 33 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:61: oid 61 updating version 33 to 34 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:61: oid 61 version 34 is already newer than 32 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:61: oid 61 updating version 34 to 35 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 61 v 35 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:61: left oid 61 (ObjNum 60 snap 0 seq_num 60) 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:63: finishing write tid 1 to smithi16030472-63 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:61: done (15 left) 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:77: write initial oid 77 2024-09-21T18:07:15.524 INFO:tasks.rados.rados.0.smithi160.stdout:77: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-21T18:07:15.524 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:15.521+0000 7f4976023640 -1 osd.5 45 handle_osd_map injecting map crc failure 2024-09-21T18:07:15.528 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:15.523+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:15.534 INFO:tasks.rados.rados.0.smithi160.stdout:77: writing smithi16030472-77 from 477332 to 1243433 tid 1 2024-09-21T18:07:15.543 INFO:tasks.rados.rados.0.smithi160.stdout:77: writing smithi16030472-77 from 1681102 to 2155112 tid 2 2024-09-21T18:07:15.553 INFO:tasks.rados.rados.0.smithi160.stdout:77: writing smithi16030472-77 from 2687912 to 3277332 tid 3 2024-09-21T18:07:15.553 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.553 INFO:tasks.rados.rados.0.smithi160.stdout:68: finishing write tid 1 to smithi16030472-68 2024-09-21T18:07:15.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:07:15.590 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":46,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:07:15.195344+0000","last_up_change":"2024-09-21T18:06:41.831877+0000","last_in_change":"2024-09-21T18:06:32.496673+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-21T18:06:59.604621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-21T18:07:01.499947+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":3600,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":501927900}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":501927900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":501927900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":501927900}]},"public_addr":"172.21.15.160:6801/501927900","cluster_addr":"172.21.15.160:6802/501927900","heartbeat_back_addr":"172.21.15.160:6804/501927900","heartbeat_front_addr":"172.21.15.160:6803/501927900","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.888021+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.308804+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.971774+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.573107+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.734463+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.956194+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.774747+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.809566+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:38.936207+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.419469+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.578534+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.001270+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.397216+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.571531+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.219010+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:06:39.336388+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:07:15.596 INFO:tasks.rados.rados.0.smithi160.stdout:63: finishing write tid 2 to smithi16030472-63 2024-09-21T18:07:15.596 INFO:tasks.rados.rados.0.smithi160.stdout:63: finishing write tid 3 to smithi16030472-63 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: finishing write tid 4 to smithi16030472-63 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: oid 63 updating version 0 to 36 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: oid 63 updating version 36 to 37 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: oid 63 updating version 37 to 38 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 63 v 38 (ObjNum 62 snap 0 seq_num 62) dirty exists 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: left oid 63 (ObjNum 62 snap 0 seq_num 62) 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:63: done (15 left) 2024-09-21T18:07:15.597 INFO:tasks.rados.rados.0.smithi160.stdout:78: write initial oid 78 2024-09-21T18:07:15.598 INFO:tasks.rados.rados.0.smithi160.stdout:78: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-21T18:07:15.606 INFO:tasks.rados.rados.0.smithi160.stdout:78: writing smithi16030472-78 from 494139 to 1051842 tid 1 2024-09-21T18:07:15.612 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T18:07:15.615 INFO:tasks.rados.rados.0.smithi160.stdout:78: writing smithi16030472-78 from 1455937 to 2189958 tid 2 2024-09-21T18:07:15.625 INFO:tasks.rados.rados.0.smithi160.stdout:78: writing smithi16030472-78 from 2948041 to 3294139 tid 3 2024-09-21T18:07:15.625 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.628 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:15.623+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:15.683 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.683 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:15.729 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:15.723+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: finishing write tid 1 to smithi16030472-72 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: finishing write tid 2 to smithi16030472-72 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: finishing write tid 3 to smithi16030472-72 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: finishing write tid 4 to smithi16030472-72 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: oid 72 updating version 0 to 33 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: oid 72 updating version 33 to 34 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: oid 72 updating version 34 to 35 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 72 v 35 (ObjNum 71 snap 0 seq_num 71) dirty exists 2024-09-21T18:07:15.730 INFO:tasks.rados.rados.0.smithi160.stdout:72: left oid 72 (ObjNum 71 snap 0 seq_num 71) 2024-09-21T18:07:15.731 INFO:tasks.rados.rados.0.smithi160.stdout:72: done (15 left) 2024-09-21T18:07:15.731 INFO:tasks.rados.rados.0.smithi160.stdout:79: write initial oid 79 2024-09-21T18:07:15.731 INFO:tasks.rados.rados.0.smithi160.stdout:79: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-21T18:07:15.740 INFO:tasks.rados.rados.0.smithi160.stdout:79: writing smithi16030472-79 from 510946 to 1143898 tid 1 2024-09-21T18:07:15.750 INFO:tasks.rados.rados.0.smithi160.stdout:79: writing smithi16030472-79 from 1798066 to 2275745 tid 2 2024-09-21T18:07:15.759 INFO:tasks.rados.rados.0.smithi160.stdout:79: writing smithi16030472-79 from 2742758 to 3199975 tid 3 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:71: finishing write tid 3 to smithi16030472-71 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:71: finishing write tid 4 to smithi16030472-71 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:71: oid 71 updating version 0 to 42 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:71: oid 71 version 42 is already newer than 31 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:71: oid 71 version 42 is already newer than 32 2024-09-21T18:07:15.760 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 71 v 42 (ObjNum 70 snap 0 seq_num 70) dirty exists 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:71: left oid 71 (ObjNum 70 snap 0 seq_num 70) 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: finishing write tid 5 to smithi16030472-69 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: oid 69 updating version 0 to 45 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: oid 69 version 45 is already newer than 24 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: oid 69 version 45 is already newer than 23 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: oid 69 version 45 is already newer than 44 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 45 (ObjNum 68 snap 0 seq_num 68) dirty exists 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:69: left oid 69 (ObjNum 68 snap 0 seq_num 68) 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:77: finishing write tid 1 to smithi16030472-77 2024-09-21T18:07:15.761 INFO:tasks.rados.rados.0.smithi160.stdout:77: finishing write tid 2 to smithi16030472-77 2024-09-21T18:07:15.762 INFO:tasks.rados.rados.0.smithi160.stdout:69: done (15 left) 2024-09-21T18:07:15.762 INFO:tasks.rados.rados.0.smithi160.stdout:71: done (14 left) 2024-09-21T18:07:15.762 INFO:tasks.rados.rados.0.smithi160.stdout:80: write initial oid 80 2024-09-21T18:07:15.762 INFO:tasks.rados.rados.0.smithi160.stdout:80: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-21T18:07:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:80: writing smithi16030472-80 from 527753 to 1235954 tid 1 2024-09-21T18:07:15.782 INFO:tasks.rados.rados.0.smithi160.stdout:80: writing smithi16030472-80 from 1856548 to 2594238 tid 2 2024-09-21T18:07:15.788 INFO:tasks.rados.rados.0.smithi160.stdout:80: writing smithi16030472-80 from 3286534 to 3327753 tid 3 2024-09-21T18:07:15.788 INFO:tasks.rados.rados.0.smithi160.stdout:81: write initial oid 81 2024-09-21T18:07:15.789 INFO:tasks.rados.rados.0.smithi160.stdout:81: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-21T18:07:15.800 INFO:tasks.rados.rados.0.smithi160.stdout:81: writing smithi16030472-81 from 544560 to 1328010 tid 1 2024-09-21T18:07:15.807 INFO:tasks.rados.rados.0.smithi160.stdout:81: writing smithi16030472-81 from 1915030 to 2396378 tid 2 2024-09-21T18:07:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.813 INFO:tasks.rados.rados.0.smithi160.stdout:81: writing smithi16030472-81 from 2797604 to 3277365 tid 3 2024-09-21T18:07:15.813 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.813 INFO:tasks.rados.rados.0.smithi160.stdout:74: finishing write tid 1 to smithi16030472-74 2024-09-21T18:07:15.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: finishing write tid 3 to smithi16030472-77 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: finishing write tid 4 to smithi16030472-77 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: finishing write tid 5 to smithi16030472-77 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: oid 77 updating version 0 to 29 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: oid 77 version 29 is already newer than 28 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: oid 77 version 29 is already newer than 27 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: oid 77 version 29 is already newer than 26 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 77 v 29 (ObjNum 76 snap 0 seq_num 76) dirty exists 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: left oid 77 (ObjNum 76 snap 0 seq_num 76) 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:77: done (15 left) 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:82: write initial oid 82 2024-09-21T18:07:15.857 INFO:tasks.rados.rados.0.smithi160.stdout:82: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-21T18:07:15.863 INFO:tasks.rados.rados.0.smithi160.stdout:82: writing smithi16030472-82 from 561367 to 1020066 tid 1 2024-09-21T18:07:15.870 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:15.869+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:15.870 INFO:tasks.rados.rados.0.smithi160.stdout:82: writing smithi16030472-82 from 1573512 to 2314871 tid 2 2024-09-21T18:07:15.877 INFO:tasks.rados.rados.0.smithi160.stdout:82: writing smithi16030472-82 from 2941380 to 3361367 tid 3 2024-09-21T18:07:15.877 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:15.886 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:15.885+0000 7f8c3a0ca640 -1 osd.12 45 handle_osd_map injecting map crc failure 2024-09-21T18:07:15.915 INFO:tasks.rados.rados.0.smithi160.stdout:66: finishing write tid 1 to smithi16030472-66 2024-09-21T18:07:15.930 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:15.927+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:15.946 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:15.946 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:15.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:15.976 INFO:tasks.rados.rados.0.smithi160.stdout:78: finishing write tid 1 to smithi16030472-78 2024-09-21T18:07:16.016 INFO:tasks.rados.rados.0.smithi160.stdout:66: finishing write tid 2 to smithi16030472-66 2024-09-21T18:07:16.017 INFO:tasks.rados.rados.0.smithi160.stdout:66: finishing write tid 3 to smithi16030472-66 2024-09-21T18:07:16.032 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:16.031+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:16.045 INFO:tasks.rados.rados.0.smithi160.stdout:79: finishing write tid 1 to smithi16030472-79 2024-09-21T18:07:16.048 INFO:tasks.rados.rados.0.smithi160.stdout:79: finishing write tid 2 to smithi16030472-79 2024-09-21T18:07:16.118 INFO:tasks.rados.rados.0.smithi160.stdout:73: finishing write tid 2 to smithi16030472-73 2024-09-21T18:07:16.118 INFO:tasks.rados.rados.0.smithi160.stdout:66: finishing write tid 4 to smithi16030472-66 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:66: oid 66 updating version 0 to 41 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:66: oid 66 version 41 is already newer than 39 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:66: oid 66 version 41 is already newer than 40 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 66 v 41 (ObjNum 65 snap 0 seq_num 65) dirty exists 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:66: left oid 66 (ObjNum 65 snap 0 seq_num 65) 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:70: finishing write tid 1 to smithi16030472-70 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:66: done (15 left) 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:83: write initial oid 83 2024-09-21T18:07:16.119 INFO:tasks.rados.rados.0.smithi160.stdout:83: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-21T18:07:16.119 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.120 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.126 INFO:tasks.rados.rados.0.smithi160.stdout:83: writing smithi16030472-83 from 578174 to 1112122 tid 1 2024-09-21T18:07:16.132 INFO:tasks.rados.rados.0.smithi160.stdout:83: writing smithi16030472-83 from 1515641 to 2000658 tid 2 2024-09-21T18:07:16.132 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:16.127+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:16.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:16.140 INFO:tasks.rados.rados.0.smithi160.stdout:83: writing smithi16030472-83 from 2736097 to 3238402 tid 3 2024-09-21T18:07:16.140 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.140 INFO:tasks.rados.rados.0.smithi160.stdout:70: finishing write tid 2 to smithi16030472-70 2024-09-21T18:07:16.140 INFO:tasks.rados.rados.0.smithi160.stdout:73: finishing write tid 3 to smithi16030472-73 2024-09-21T18:07:16.151 INFO:tasks.rados.rados.0.smithi160.stdout:62: finishing write tid 2 to smithi16030472-62 2024-09-21T18:07:16.218 INFO:tasks.rados.rados.0.smithi160.stdout:73: finishing write tid 4 to smithi16030472-73 2024-09-21T18:07:16.219 INFO:tasks.rados.rados.0.smithi160.stdout:73: finishing write tid 5 to smithi16030472-73 2024-09-21T18:07:16.219 INFO:tasks.rados.rados.0.smithi160.stdout:73: oid 73 updating version 0 to 49 2024-09-21T18:07:16.219 INFO:tasks.rados.rados.0.smithi160.stdout:73: oid 73 version 49 is already newer than 47 2024-09-21T18:07:16.219 INFO:tasks.rados.rados.0.smithi160.stdout:73: oid 73 version 49 is already newer than 25 2024-09-21T18:07:16.219 INFO:tasks.rados.rados.0.smithi160.stdout:73: oid 73 version 49 is already newer than 48 2024-09-21T18:07:16.220 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 73 v 49 (ObjNum 72 snap 0 seq_num 72) dirty exists 2024-09-21T18:07:16.220 INFO:tasks.rados.rados.0.smithi160.stdout:73: left oid 73 (ObjNum 72 snap 0 seq_num 72) 2024-09-21T18:07:16.220 INFO:tasks.rados.rados.0.smithi160.stdout:73: done (15 left) 2024-09-21T18:07:16.220 INFO:tasks.rados.rados.0.smithi160.stdout:84: write initial oid 84 2024-09-21T18:07:16.220 INFO:tasks.rados.rados.0.smithi160.stdout:84: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-21T18:07:16.228 INFO:tasks.rados.rados.0.smithi160.stdout:84: writing smithi16030472-84 from 594981 to 1204178 tid 1 2024-09-21T18:07:16.233 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:16.229+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:16.240 INFO:tasks.rados.rados.0.smithi160.stdout:84: writing smithi16030472-84 from 1974123 to 2719151 tid 2 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:84: writing smithi16030472-84 from 3279873 to 3394981 tid 3 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:78: finishing write tid 2 to smithi16030472-78 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:78: finishing write tid 3 to smithi16030472-78 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:78: finishing write tid 4 to smithi16030472-78 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:70: finishing write tid 3 to smithi16030472-70 2024-09-21T18:07:16.246 INFO:tasks.rados.rados.0.smithi160.stdout:70: finishing write tid 4 to smithi16030472-70 2024-09-21T18:07:16.251 INFO:tasks.rados.rados.0.smithi160.stdout:62: finishing write tid 3 to smithi16030472-62 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: finishing write tid 4 to smithi16030472-62 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: oid 62 updating version 0 to 44 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: oid 62 version 44 is already newer than 16 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: oid 62 version 44 is already newer than 43 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 62 v 44 (ObjNum 61 snap 0 seq_num 61) dirty exists 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: left oid 62 (ObjNum 61 snap 0 seq_num 61) 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:62: done (15 left) 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:85: write initial oid 85 2024-09-21T18:07:16.252 INFO:tasks.rados.rados.0.smithi160.stdout:85: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-21T18:07:16.253 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.253 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.260 INFO:tasks.rados.rados.0.smithi160.stdout:85: writing smithi16030472-85 from 611788 to 1012587 tid 1 2024-09-21T18:07:16.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:16.270 INFO:tasks.rados.rados.0.smithi160.stdout:85: writing smithi16030472-85 from 1748958 to 2237644 tid 2 2024-09-21T18:07:16.280 INFO:tasks.rados.rados.0.smithi160.stdout:85: writing smithi16030472-85 from 2907296 to 3411788 tid 3 2024-09-21T18:07:16.280 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.280 INFO:tasks.rados.rados.0.smithi160.stdout:67: finishing write tid 2 to smithi16030472-67 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: finishing write tid 5 to smithi16030472-78 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: oid 78 updating version 0 to 53 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: oid 78 version 53 is already newer than 52 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: oid 78 version 53 is already newer than 51 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: oid 78 version 53 is already newer than 30 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 53 (ObjNum 77 snap 0 seq_num 77) dirty exists 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: left oid 78 (ObjNum 77 snap 0 seq_num 77) 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:78: done (15 left) 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:86: write initial oid 86 2024-09-21T18:07:16.321 INFO:tasks.rados.rados.0.smithi160.stdout:86: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-21T18:07:16.330 INFO:tasks.rados.rados.0.smithi160.stdout:86: writing smithi16030472-86 from 628595 to 1104643 tid 1 2024-09-21T18:07:16.334 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:16.329+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:16.342 INFO:tasks.rados.rados.0.smithi160.stdout:86: writing smithi16030472-86 from 1807440 to 2439784 tid 2 2024-09-21T18:07:16.350 INFO:tasks.rados.rados.0.smithi160.stdout:86: writing smithi16030472-86 from 2934719 to 3387193 tid 3 2024-09-21T18:07:16.350 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.350 INFO:tasks.rados.rados.0.smithi160.stdout:70: finishing write tid 5 to smithi16030472-70 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: oid 70 updating version 0 to 45 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: oid 70 version 45 is already newer than 44 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: oid 70 version 45 is already newer than 43 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: oid 70 version 45 is already newer than 42 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 45 (ObjNum 69 snap 0 seq_num 69) dirty exists 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: left oid 70 (ObjNum 69 snap 0 seq_num 69) 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:75: finishing write tid 1 to smithi16030472-75 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:75: finishing write tid 2 to smithi16030472-75 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:75: finishing write tid 3 to smithi16030472-75 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:75: finishing write tid 4 to smithi16030472-75 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:70: done (15 left) 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:87: write initial oid 87 2024-09-21T18:07:16.351 INFO:tasks.rados.rados.0.smithi160.stdout:87: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-21T18:07:16.361 INFO:tasks.rados.rados.0.smithi160.stdout:87: writing smithi16030472-87 from 645402 to 1196699 tid 1 2024-09-21T18:07:16.370 INFO:tasks.rados.rados.0.smithi160.stdout:87: writing smithi16030472-87 from 1749569 to 2241924 tid 2 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:87: writing smithi16030472-87 from 2845789 to 3445402 tid 3 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: finishing write tid 3 to smithi16030472-79 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: finishing write tid 4 to smithi16030472-79 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: finishing write tid 5 to smithi16030472-79 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: oid 79 updating version 0 to 37 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: oid 79 updating version 37 to 45 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: oid 79 version 45 is already newer than 36 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: oid 79 version 45 is already newer than 44 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 45 (ObjNum 78 snap 0 seq_num 78) dirty exists 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:79: left oid 79 (ObjNum 78 snap 0 seq_num 78) 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:67: finishing write tid 3 to smithi16030472-67 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:67: finishing write tid 4 to smithi16030472-67 2024-09-21T18:07:16.380 INFO:tasks.rados.rados.0.smithi160.stdout:67: finishing write tid 5 to smithi16030472-67 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:67: oid 67 updating version 0 to 47 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:67: oid 67 version 47 is already newer than 17 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:67: oid 67 version 47 is already newer than 45 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:67: oid 67 version 47 is already newer than 46 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 67 v 47 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:67: left oid 67 (ObjNum 66 snap 0 seq_num 66) 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:83: finishing write tid 1 to smithi16030472-83 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:83: finishing write tid 2 to smithi16030472-83 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: finishing write tid 2 to smithi16030472-59 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: finishing write tid 3 to smithi16030472-59 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: finishing write tid 4 to smithi16030472-59 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: oid 59 updating version 0 to 50 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: oid 59 updating version 50 to 51 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:59: oid 59 version 51 is already newer than 14 2024-09-21T18:07:16.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 51 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:59: left oid 59 (ObjNum 58 snap 0 seq_num 58) 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:60: finishing write tid 2 to smithi16030472-60 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:60: finishing write tid 3 to smithi16030472-60 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:60: finishing write tid 4 to smithi16030472-60 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:59: done (15 left) 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:67: done (14 left) 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:79: done (13 left) 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:88: write initial oid 88 2024-09-21T18:07:16.382 INFO:tasks.rados.rados.0.smithi160.stdout:88: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-21T18:07:16.388 INFO:tasks.rados.rados.0.smithi160.stdout:88: writing smithi16030472-88 from 662209 to 1288755 tid 1 2024-09-21T18:07:16.395 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.395 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.396 INFO:tasks.rados.rados.0.smithi160.stdout:88: writing smithi16030472-88 from 1808051 to 2444064 tid 2 2024-09-21T18:07:16.403 INFO:tasks.rados.rados.0.smithi160.stdout:88: writing smithi16030472-88 from 2873212 to 3462209 tid 3 2024-09-21T18:07:16.403 INFO:tasks.rados.rados.0.smithi160.stdout:89: write initial oid 89 2024-09-21T18:07:16.403 INFO:tasks.rados.rados.0.smithi160.stdout:89: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-21T18:07:16.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:16.412 INFO:tasks.rados.rados.0.smithi160.stdout:89: writing smithi16030472-89 from 679016 to 1380811 tid 1 2024-09-21T18:07:16.418 INFO:tasks.rados.rados.0.smithi160.stdout:89: writing smithi16030472-89 from 1866533 to 2362557 tid 2 2024-09-21T18:07:16.423 INFO:tasks.rados.rados.0.smithi160.stdout:89: writing smithi16030472-89 from 2900635 to 3303278 tid 3 2024-09-21T18:07:16.423 INFO:tasks.rados.rados.0.smithi160.stdout:90: write initial oid 90 2024-09-21T18:07:16.423 INFO:tasks.rados.rados.0.smithi160.stdout:90: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-21T18:07:16.431 INFO:tasks.rados.rados.0.smithi160.stdout:90: writing smithi16030472-90 from 695823 to 1472867 tid 1 2024-09-21T18:07:16.435 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:16.429+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:16.437 INFO:tasks.rados.rados.0.smithi160.stdout:90: writing smithi16030472-90 from 1925015 to 2564697 tid 2 2024-09-21T18:07:16.443 INFO:tasks.rados.rados.0.smithi160.stdout:90: writing smithi16030472-90 from 3328058 to 3495823 tid 3 2024-09-21T18:07:16.443 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.443 INFO:tasks.rados.rados.0.smithi160.stdout:84: finishing write tid 1 to smithi16030472-84 2024-09-21T18:07:16.453 INFO:tasks.rados.rados.0.smithi160.stdout:76: finishing write tid 1 to smithi16030472-76 2024-09-21T18:07:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:60: finishing write tid 5 to smithi16030472-60 2024-09-21T18:07:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:60: oid 60 updating version 0 to 54 2024-09-21T18:07:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:60: oid 60 version 54 is already newer than 53 2024-09-21T18:07:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:60: oid 60 version 54 is already newer than 15 2024-09-21T18:07:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:60: oid 60 version 54 is already newer than 52 2024-09-21T18:07:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 54 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-21T18:07:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:60: left oid 60 (ObjNum 59 snap 0 seq_num 59) 2024-09-21T18:07:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:60: done (15 left) 2024-09-21T18:07:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:91: write initial oid 91 2024-09-21T18:07:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:91: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-21T18:07:16.463 INFO:tasks.rados.rados.0.smithi160.stdout:91: writing smithi16030472-91 from 712630 to 1164923 tid 1 2024-09-21T18:07:16.468 INFO:tasks.rados.rados.0.smithi160.stdout:91: writing smithi16030472-91 from 1583497 to 2083190 tid 2 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:91: writing smithi16030472-91 from 2555481 to 2980668 tid 3 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: finishing write tid 2 to smithi16030472-68 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: finishing write tid 3 to smithi16030472-68 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: finishing write tid 4 to smithi16030472-68 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: oid 68 updating version 0 to 56 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: oid 68 version 56 is already newer than 18 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: oid 68 updating version 56 to 57 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 68 v 57 (ObjNum 67 snap 0 seq_num 67) dirty exists 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:68: left oid 68 (ObjNum 67 snap 0 seq_num 67) 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:74: finishing write tid 2 to smithi16030472-74 2024-09-21T18:07:16.474 INFO:tasks.rados.rados.0.smithi160.stdout:74: finishing write tid 3 to smithi16030472-74 2024-09-21T18:07:16.475 INFO:tasks.rados.rados.0.smithi160.stdout:74: finishing write tid 4 to smithi16030472-74 2024-09-21T18:07:16.475 INFO:tasks.rados.rados.0.smithi160.stdout:68: done (15 left) 2024-09-21T18:07:16.475 INFO:tasks.rados.rados.0.smithi160.stdout:92: write initial oid 92 2024-09-21T18:07:16.475 INFO:tasks.rados.rados.0.smithi160.stdout:92: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-21T18:07:16.481 INFO:tasks.rados.rados.0.smithi160.stdout:92: writing smithi16030472-92 from 729437 to 1256979 tid 1 2024-09-21T18:07:16.488 INFO:tasks.rados.rados.0.smithi160.stdout:92: writing smithi16030472-92 from 1925626 to 2568977 tid 2 2024-09-21T18:07:16.493 INFO:tasks.rados.rados.0.smithi160.stdout:92: writing smithi16030472-92 from 3266551 to 3529437 tid 3 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: finishing write tid 5 to smithi16030472-74 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: oid 74 updating version 0 to 61 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: oid 74 version 61 is already newer than 19 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: oid 74 version 61 is already newer than 60 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: oid 74 version 61 is already newer than 59 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 74 v 61 (ObjNum 73 snap 0 seq_num 73) dirty exists 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: left oid 74 (ObjNum 73 snap 0 seq_num 73) 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:74: done (15 left) 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:93: write initial oid 93 2024-09-21T18:07:16.494 INFO:tasks.rados.rados.0.smithi160.stdout:93: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-21T18:07:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:93: writing smithi16030472-93 from 746244 to 1465388 tid 1 2024-09-21T18:07:16.508 INFO:tasks.rados.rados.0.smithi160.stdout:93: writing smithi16030472-93 from 2100461 to 2603823 tid 2 2024-09-21T18:07:16.513 INFO:tasks.rados.rados.0.smithi160.stdout:93: writing smithi16030472-93 from 3126680 to 3546244 tid 3 2024-09-21T18:07:16.513 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.536 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:16.533+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:16.541 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.541 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.543 INFO:tasks.rados.rados.0.smithi160.stdout:86: finishing write tid 1 to smithi16030472-86 2024-09-21T18:07:16.554 INFO:tasks.rados.rados.0.smithi160.stdout:75: finishing write tid 5 to smithi16030472-75 2024-09-21T18:07:16.554 INFO:tasks.rados.rados.0.smithi160.stdout:75: oid 75 updating version 0 to 47 2024-09-21T18:07:16.554 INFO:tasks.rados.rados.0.smithi160.stdout:75: oid 75 updating version 47 to 48 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:75: oid 75 version 48 is already newer than 46 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:75: oid 75 updating version 48 to 49 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 75 v 49 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:75: left oid 75 (ObjNum 74 snap 0 seq_num 74) 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:75: done (15 left) 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:94: write initial oid 94 2024-09-21T18:07:16.555 INFO:tasks.rados.rados.0.smithi160.stdout:94: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-21T18:07:16.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:16.563 INFO:tasks.rados.rados.0.smithi160.stdout:94: writing smithi16030472-94 from 763051 to 1557444 tid 1 2024-09-21T18:07:16.570 INFO:tasks.rados.rados.0.smithi160.stdout:94: writing smithi16030472-94 from 2158943 to 2805963 tid 2 2024-09-21T18:07:16.574 INFO:tasks.rados.rados.0.smithi160.stdout:94: writing smithi16030472-94 from 3437750 to 3563051 tid 3 2024-09-21T18:07:16.575 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.575 INFO:tasks.rados.rados.0.smithi160.stdout:76: finishing write tid 2 to smithi16030472-76 2024-09-21T18:07:16.575 INFO:tasks.rados.rados.0.smithi160.stdout:76: finishing write tid 3 to smithi16030472-76 2024-09-21T18:07:16.636 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:16.633+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: finishing write tid 4 to smithi16030472-76 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: finishing write tid 5 to smithi16030472-76 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: oid 76 updating version 0 to 52 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: oid 76 version 52 is already newer than 50 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: oid 76 updating version 52 to 53 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: oid 76 version 53 is already newer than 51 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 53 (ObjNum 75 snap 0 seq_num 75) dirty exists 2024-09-21T18:07:16.665 INFO:tasks.rados.rados.0.smithi160.stdout:76: left oid 76 (ObjNum 75 snap 0 seq_num 75) 2024-09-21T18:07:16.666 INFO:tasks.rados.rados.0.smithi160.stdout:76: done (15 left) 2024-09-21T18:07:16.666 INFO:tasks.rados.rados.0.smithi160.stdout:95: write initial oid 95 2024-09-21T18:07:16.666 INFO:tasks.rados.rados.0.smithi160.stdout:95: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-21T18:07:16.672 INFO:tasks.rados.rados.0.smithi160.stdout:95: writing smithi16030472-95 from 779858 to 1249500 tid 1 2024-09-21T18:07:16.678 INFO:tasks.rados.rados.0.smithi160.stdout:95: writing smithi16030472-95 from 1817425 to 2324456 tid 2 2024-09-21T18:07:16.684 INFO:tasks.rados.rados.0.smithi160.stdout:95: writing smithi16030472-95 from 2781526 to 3368154 tid 3 2024-09-21T18:07:16.684 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.684 INFO:tasks.rados.rados.0.smithi160.stdout:83: finishing write tid 3 to smithi16030472-83 2024-09-21T18:07:16.684 INFO:tasks.rados.rados.0.smithi160.stdout:89: finishing write tid 1 to smithi16030472-89 2024-09-21T18:07:16.684 INFO:tasks.rados.rados.0.smithi160.stdout:89: finishing write tid 2 to smithi16030472-89 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: finishing write tid 3 to smithi16030472-89 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: finishing write tid 4 to smithi16030472-89 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: finishing write tid 5 to smithi16030472-89 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: oid 89 updating version 0 to 22 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: oid 89 updating version 22 to 24 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: oid 89 version 24 is already newer than 23 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: oid 89 updating version 24 to 25 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 89 v 25 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:89: left oid 89 (ObjNum 88 snap 0 seq_num 88) 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:83: finishing write tid 4 to smithi16030472-83 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:83: finishing write tid 5 to smithi16030472-83 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:83: oid 83 updating version 0 to 64 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:83: oid 83 version 64 is already newer than 20 2024-09-21T18:07:16.685 INFO:tasks.rados.rados.0.smithi160.stdout:83: oid 83 version 64 is already newer than 21 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:83: oid 83 version 64 is already newer than 63 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 83 v 64 (ObjNum 82 snap 0 seq_num 82) dirty exists 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:83: left oid 83 (ObjNum 82 snap 0 seq_num 82) 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:83: done (15 left) 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:89: done (14 left) 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:96: write initial oid 96 2024-09-21T18:07:16.686 INFO:tasks.rados.rados.0.smithi160.stdout:96: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-21T18:07:16.693 INFO:tasks.rados.rados.0.smithi160.stdout:96: writing smithi16030472-96 from 796665 to 1341556 tid 1 2024-09-21T18:07:16.699 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.699 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.700 INFO:tasks.rados.rados.0.smithi160.stdout:96: writing smithi16030472-96 from 1759554 to 2410243 tid 2 2024-09-21T18:07:16.709 INFO:tasks.rados.rados.0.smithi160.stdout:96: writing smithi16030472-96 from 2976243 to 3596665 tid 3 2024-09-21T18:07:16.709 INFO:tasks.rados.rados.0.smithi160.stdout:97: write initial oid 97 2024-09-21T18:07:16.709 INFO:tasks.rados.rados.0.smithi160.stdout:97: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-21T18:07:16.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:16.716 INFO:tasks.rados.rados.0.smithi160.stdout:97: writing smithi16030472-97 from 413472 to 1033612 tid 1 2024-09-21T18:07:16.724 INFO:tasks.rados.rados.0.smithi160.stdout:97: writing smithi16030472-97 from 1818036 to 2328736 tid 2 2024-09-21T18:07:16.732 INFO:tasks.rados.rados.0.smithi160.stdout:97: writing smithi16030472-97 from 3120019 to 3613472 tid 3 2024-09-21T18:07:16.732 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.732 INFO:tasks.rados.rados.0.smithi160.stdout:95: finishing write tid 1 to smithi16030472-95 2024-09-21T18:07:16.737 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:16.733+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:16.811 INFO:tasks.rados.rados.0.smithi160.stdout:80: finishing write tid 1 to smithi16030472-80 2024-09-21T18:07:16.811 INFO:tasks.rados.rados.0.smithi160.stdout:80: finishing write tid 2 to smithi16030472-80 2024-09-21T18:07:16.811 INFO:tasks.rados.rados.0.smithi160.stdout:86: finishing write tid 2 to smithi16030472-86 2024-09-21T18:07:16.811 INFO:tasks.rados.rados.0.smithi160.stdout:86: finishing write tid 3 to smithi16030472-86 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: finishing write tid 4 to smithi16030472-86 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: finishing write tid 5 to smithi16030472-86 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: oid 86 updating version 0 to 39 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: oid 86 updating version 39 to 41 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: oid 86 version 41 is already newer than 38 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: oid 86 version 41 is already newer than 40 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 86 v 41 (ObjNum 85 snap 0 seq_num 85) dirty exists 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: left oid 86 (ObjNum 85 snap 0 seq_num 85) 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:86: done (15 left) 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:98: write initial oid 98 2024-09-21T18:07:16.812 INFO:tasks.rados.rados.0.smithi160.stdout:98: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-21T18:07:16.817 INFO:tasks.rados.rados.0.smithi160.stdout:98: writing smithi16030472-98 from 430279 to 1125668 tid 1 2024-09-21T18:07:16.825 INFO:tasks.rados.rados.0.smithi160.stdout:98: writing smithi16030472-98 from 1876518 to 2530876 tid 2 2024-09-21T18:07:16.831 INFO:tasks.rados.rados.0.smithi160.stdout:98: writing smithi16030472-98 from 3031089 to 3567886 tid 3 2024-09-21T18:07:16.831 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:16.834 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:16.834 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:16.838 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:16.833+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:16.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:16.912 INFO:tasks.rados.rados.0.smithi160.stdout:80: finishing write tid 3 to smithi16030472-80 2024-09-21T18:07:16.917 INFO:tasks.rados.rados.0.smithi160.stdout:88: finishing write tid 1 to smithi16030472-88 2024-09-21T18:07:16.917 INFO:tasks.rados.rados.0.smithi160.stdout:88: finishing write tid 2 to smithi16030472-88 2024-09-21T18:07:16.917 INFO:tasks.rados.rados.0.smithi160.stdout:94: finishing write tid 1 to smithi16030472-94 2024-09-21T18:07:16.933 INFO:tasks.rados.rados.0.smithi160.stdout:95: finishing write tid 2 to smithi16030472-95 2024-09-21T18:07:16.939 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:16.937+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:17.021 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:17.021 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:97: finishing write tid 1 to smithi16030472-97 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:81: finishing write tid 1 to smithi16030472-81 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:81: finishing write tid 2 to smithi16030472-81 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:82: finishing write tid 1 to smithi16030472-82 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:82: finishing write tid 2 to smithi16030472-82 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: finishing write tid 4 to smithi16030472-80 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: finishing write tid 5 to smithi16030472-80 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: oid 80 updating version 0 to 55 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: oid 80 version 55 is already newer than 52 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: oid 80 version 55 is already newer than 54 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: oid 80 version 55 is already newer than 51 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 55 (ObjNum 79 snap 0 seq_num 79) dirty exists 2024-09-21T18:07:17.024 INFO:tasks.rados.rados.0.smithi160.stdout:80: left oid 80 (ObjNum 79 snap 0 seq_num 79) 2024-09-21T18:07:17.025 INFO:tasks.rados.rados.0.smithi160.stdout:96: finishing write tid 1 to smithi16030472-96 2024-09-21T18:07:17.025 INFO:tasks.rados.rados.0.smithi160.stdout:96: finishing write tid 2 to smithi16030472-96 2024-09-21T18:07:17.025 INFO:tasks.rados.rados.0.smithi160.stdout:80: done (15 left) 2024-09-21T18:07:17.025 INFO:tasks.rados.rados.0.smithi160.stdout:99: write initial oid 99 2024-09-21T18:07:17.025 INFO:tasks.rados.rados.0.smithi160.stdout:99: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-21T18:07:17.032 INFO:tasks.rados.rados.0.smithi160.stdout:99: writing smithi16030472-99 from 447086 to 1217724 tid 1 2024-09-21T18:07:17.038 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:17.041 INFO:tasks.rados.rados.0.smithi160.stdout:99: writing smithi16030472-99 from 1935000 to 2733016 tid 2 2024-09-21T18:07:17.047 INFO:tasks.rados.rados.0.smithi160.stdout:99: writing smithi16030472-99 from 3458512 to 3647086 tid 3 2024-09-21T18:07:17.047 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.047 INFO:tasks.rados.rados.0.smithi160.stdout:96: finishing write tid 3 to smithi16030472-96 2024-09-21T18:07:17.047 INFO:tasks.rados.rados.0.smithi160.stdout:98: finishing write tid 1 to smithi16030472-98 2024-09-21T18:07:17.047 INFO:tasks.rados.rados.0.smithi160.stdout:97: finishing write tid 2 to smithi16030472-97 2024-09-21T18:07:17.054 INFO:tasks.rados.rados.0.smithi160.stdout:84: finishing write tid 2 to smithi16030472-84 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: finishing write tid 1 to smithi16030472-87 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: finishing write tid 2 to smithi16030472-87 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: finishing write tid 3 to smithi16030472-87 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: finishing write tid 4 to smithi16030472-87 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: finishing write tid 5 to smithi16030472-87 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: oid 87 updating version 0 to 36 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: oid 87 version 36 is already newer than 34 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: oid 87 version 36 is already newer than 35 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: oid 87 version 36 is already newer than 33 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 87 v 36 (ObjNum 86 snap 0 seq_num 86) dirty exists 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:87: left oid 87 (ObjNum 86 snap 0 seq_num 86) 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:90: finishing write tid 1 to smithi16030472-90 2024-09-21T18:07:17.055 INFO:tasks.rados.rados.0.smithi160.stdout:90: finishing write tid 2 to smithi16030472-90 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: finishing write tid 3 to smithi16030472-84 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: finishing write tid 4 to smithi16030472-84 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: finishing write tid 5 to smithi16030472-84 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: oid 84 updating version 0 to 31 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: oid 84 updating version 31 to 32 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: oid 84 updating version 32 to 56 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: oid 84 version 56 is already newer than 55 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 84 v 56 (ObjNum 83 snap 0 seq_num 83) dirty exists 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: left oid 84 (ObjNum 83 snap 0 seq_num 83) 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:84: done (15 left) 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:87: done (14 left) 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:100: write initial oid 100 2024-09-21T18:07:17.056 INFO:tasks.rados.rados.0.smithi160.stdout:100: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-21T18:07:17.061 INFO:tasks.rados.rados.0.smithi160.stdout:100: writing smithi16030472-100 from 463893 to 1026133 tid 1 2024-09-21T18:07:17.069 INFO:tasks.rados.rados.0.smithi160.stdout:100: writing smithi16030472-100 from 1593482 to 2251509 tid 2 2024-09-21T18:07:17.075 INFO:tasks.rados.rados.0.smithi160.stdout:100: writing smithi16030472-100 from 2685935 to 3245276 tid 3 2024-09-21T18:07:17.075 INFO:tasks.rados.rados.0.smithi160.stdout:101: write initial oid 101 2024-09-21T18:07:17.075 INFO:tasks.rados.rados.0.smithi160.stdout:101: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-21T18:07:17.082 INFO:tasks.rados.rados.0.smithi160.stdout:101: writing smithi16030472-101 from 480700 to 1118189 tid 1 2024-09-21T18:07:17.083 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:17.081+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:17.088 INFO:tasks.rados.rados.0.smithi160.stdout:101: writing smithi16030472-101 from 1651964 to 2053649 tid 2 2024-09-21T18:07:17.097 INFO:tasks.rados.rados.0.smithi160.stdout:101: writing smithi16030472-101 from 2713358 to 3483971 tid 3 2024-09-21T18:07:17.097 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:82: finishing write tid 3 to smithi16030472-82 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:82: finishing write tid 4 to smithi16030472-82 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:85: finishing write tid 1 to smithi16030472-85 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:85: finishing write tid 2 to smithi16030472-85 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:85: finishing write tid 3 to smithi16030472-85 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:85: finishing write tid 4 to smithi16030472-85 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:94: finishing write tid 2 to smithi16030472-94 2024-09-21T18:07:17.135 INFO:tasks.rados.rados.0.smithi160.stdout:94: finishing write tid 3 to smithi16030472-94 2024-09-21T18:07:17.137 INFO:tasks.rados.rados.0.smithi160.stdout:94: finishing write tid 4 to smithi16030472-94 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: finishing write tid 5 to smithi16030472-94 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: oid 94 updating version 0 to 49 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: oid 94 version 49 is already newer than 48 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: oid 94 version 49 is already newer than 44 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: oid 94 updating version 49 to 50 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 50 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: left oid 94 (ObjNum 93 snap 0 seq_num 93) 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:94: done (15 left) 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:102: write initial oid 102 2024-09-21T18:07:17.138 INFO:tasks.rados.rados.0.smithi160.stdout:102: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-21T18:07:17.140 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:17.135+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:17.145 INFO:tasks.rados.rados.0.smithi160.stdout:102: writing smithi16030472-102 from 497507 to 1210245 tid 1 2024-09-21T18:07:17.153 INFO:tasks.rados.rados.0.smithi160.stdout:102: writing smithi16030472-102 from 1710446 to 2372142 tid 2 2024-09-21T18:07:17.161 INFO:tasks.rados.rados.0.smithi160.stdout:102: writing smithi16030472-102 from 3140781 to 3697507 tid 3 2024-09-21T18:07:17.161 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.161 INFO:tasks.rados.rados.0.smithi160.stdout:88: finishing write tid 3 to smithi16030472-88 2024-09-21T18:07:17.161 INFO:tasks.rados.rados.0.smithi160.stdout:88: finishing write tid 4 to smithi16030472-88 2024-09-21T18:07:17.204 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:17.204 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:17.219 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:17.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:07:17.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:17.327 INFO:tasks.rados.rados.0.smithi160.stdout:95: finishing write tid 3 to smithi16030472-95 2024-09-21T18:07:17.327 INFO:tasks.rados.rados.0.smithi160.stdout:98: finishing write tid 2 to smithi16030472-98 2024-09-21T18:07:17.327 INFO:tasks.rados.rados.0.smithi160.stdout:95: finishing write tid 4 to smithi16030472-95 2024-09-21T18:07:17.328 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:17.325+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:17.346 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:17.346 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:17.355 INFO:tasks.rados.rados.0.smithi160.stdout:88: finishing write tid 5 to smithi16030472-88 2024-09-21T18:07:17.355 INFO:tasks.rados.rados.0.smithi160.stdout:88: oid 88 updating version 0 to 51 2024-09-21T18:07:17.355 INFO:tasks.rados.rados.0.smithi160.stdout:88: oid 88 version 51 is already newer than 42 2024-09-21T18:07:17.355 INFO:tasks.rados.rados.0.smithi160.stdout:88: oid 88 version 51 is already newer than 43 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:88: oid 88 updating version 51 to 52 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 52 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:88: left oid 88 (ObjNum 87 snap 0 seq_num 87) 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: finishing write tid 3 to smithi16030472-97 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: finishing write tid 4 to smithi16030472-97 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: finishing write tid 5 to smithi16030472-97 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: oid 97 updating version 0 to 46 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: oid 97 updating version 46 to 47 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: oid 97 updating version 47 to 48 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: oid 97 version 48 is already newer than 45 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 97 v 48 (ObjNum 96 snap 0 seq_num 96) dirty exists 2024-09-21T18:07:17.356 INFO:tasks.rados.rados.0.smithi160.stdout:97: left oid 97 (ObjNum 96 snap 0 seq_num 96) 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:91: finishing write tid 1 to smithi16030472-91 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:91: finishing write tid 2 to smithi16030472-91 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:91: finishing write tid 3 to smithi16030472-91 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:91: finishing write tid 4 to smithi16030472-91 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:88: done (15 left) 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:97: done (14 left) 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:103: write initial oid 103 2024-09-21T18:07:17.357 INFO:tasks.rados.rados.0.smithi160.stdout:103: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-21T18:07:17.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:17.364 INFO:tasks.rados.rados.0.smithi160.stdout:103: writing smithi16030472-103 from 514314 to 1302301 tid 1 2024-09-21T18:07:17.370 INFO:tasks.rados.rados.0.smithi160.stdout:103: writing smithi16030472-103 from 1768928 to 2174282 tid 2 2024-09-21T18:07:17.377 INFO:tasks.rados.rados.0.smithi160.stdout:103: writing smithi16030472-103 from 2768204 to 3277714 tid 3 2024-09-21T18:07:17.377 INFO:tasks.rados.rados.0.smithi160.stdout:104: write initial oid 104 2024-09-21T18:07:17.377 INFO:tasks.rados.rados.0.smithi160.stdout:104: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-21T18:07:17.384 INFO:tasks.rados.rados.0.smithi160.stdout:104: writing smithi16030472-104 from 531121 to 994357 tid 1 2024-09-21T18:07:17.393 INFO:tasks.rados.rados.0.smithi160.stdout:104: writing smithi16030472-104 from 1711057 to 2376422 tid 2 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:104: writing smithi16030472-104 from 3079274 to 3731121 tid 3 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: finishing write tid 3 to smithi16030472-90 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: finishing write tid 4 to smithi16030472-90 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: finishing write tid 5 to smithi16030472-90 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: oid 90 updating version 0 to 37 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: oid 90 updating version 37 to 38 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: oid 90 updating version 38 to 39 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: oid 90 updating version 39 to 40 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 90 v 40 (ObjNum 89 snap 0 seq_num 89) dirty exists 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:90: left oid 90 (ObjNum 89 snap 0 seq_num 89) 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:92: finishing write tid 1 to smithi16030472-92 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:99: finishing write tid 1 to smithi16030472-99 2024-09-21T18:07:17.403 INFO:tasks.rados.rados.0.smithi160.stdout:91: finishing write tid 5 to smithi16030472-91 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: oid 91 updating version 0 to 64 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: oid 91 updating version 64 to 66 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: oid 91 version 66 is already newer than 63 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: oid 91 version 66 is already newer than 65 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 66 (ObjNum 90 snap 0 seq_num 90) dirty exists 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: left oid 91 (ObjNum 90 snap 0 seq_num 90) 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:92: finishing write tid 2 to smithi16030472-92 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:90: done (15 left) 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:91: done (14 left) 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:105: write initial oid 105 2024-09-21T18:07:17.404 INFO:tasks.rados.rados.0.smithi160.stdout:105: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-21T18:07:17.410 INFO:tasks.rados.rados.0.smithi160.stdout:105: writing smithi16030472-105 from 547928 to 1086413 tid 1 2024-09-21T18:07:17.418 INFO:tasks.rados.rados.0.smithi160.stdout:105: writing smithi16030472-105 from 1769539 to 2178562 tid 2 2024-09-21T18:07:17.425 INFO:tasks.rados.rados.0.smithi160.stdout:105: writing smithi16030472-105 from 2706697 to 3238751 tid 3 2024-09-21T18:07:17.426 INFO:tasks.rados.rados.0.smithi160.stdout:106: write initial oid 106 2024-09-21T18:07:17.426 INFO:tasks.rados.rados.0.smithi160.stdout:106: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-21T18:07:17.428 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:17.423+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:17.433 INFO:tasks.rados.rados.0.smithi160.stdout:106: writing smithi16030472-106 from 564735 to 1178469 tid 1 2024-09-21T18:07:17.440 INFO:tasks.rados.rados.0.smithi160.stdout:106: writing smithi16030472-106 from 1828021 to 2497055 tid 2 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:106: writing smithi16030472-106 from 3134120 to 3764735 tid 3 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: finishing write tid 5 to smithi16030472-85 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: oid 85 updating version 0 to 62 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: oid 85 version 62 is already newer than 60 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: oid 85 version 62 is already newer than 61 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: oid 85 version 62 is already newer than 59 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 62 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:85: left oid 85 (ObjNum 84 snap 0 seq_num 84) 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:82: finishing write tid 5 to smithi16030472-82 2024-09-21T18:07:17.448 INFO:tasks.rados.rados.0.smithi160.stdout:82: oid 82 updating version 0 to 58 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:82: oid 82 version 58 is already newer than 56 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:82: oid 82 version 58 is already newer than 57 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:82: oid 82 version 58 is already newer than 55 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 58 (ObjNum 81 snap 0 seq_num 81) dirty exists 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:82: left oid 82 (ObjNum 81 snap 0 seq_num 81) 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:81: finishing write tid 3 to smithi16030472-81 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:82: done (15 left) 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:85: done (14 left) 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:107: write initial oid 107 2024-09-21T18:07:17.449 INFO:tasks.rados.rados.0.smithi160.stdout:107: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-21T18:07:17.455 INFO:tasks.rados.rados.0.smithi160.stdout:107: writing smithi16030472-107 from 581542 to 1270525 tid 1 2024-09-21T18:07:17.460 INFO:tasks.rados.rados.0.smithi160.stdout:107: writing smithi16030472-107 from 1886503 to 2299195 tid 2 2024-09-21T18:07:17.467 INFO:tasks.rados.rados.0.smithi160.stdout:107: writing smithi16030472-107 from 2761543 to 3432494 tid 3 2024-09-21T18:07:17.467 INFO:tasks.rados.rados.0.smithi160.stdout:108: write initial oid 108 2024-09-21T18:07:17.467 INFO:tasks.rados.rados.0.smithi160.stdout:108: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-21T18:07:17.472 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:17.473 INFO:tasks.rados.rados.0.smithi160.stdout:108: writing smithi16030472-108 from 598349 to 1078934 tid 1 2024-09-21T18:07:17.479 INFO:tasks.rados.rados.0.smithi160.stdout:108: writing smithi16030472-108 from 1544985 to 2217688 tid 2 2024-09-21T18:07:17.486 INFO:tasks.rados.rados.0.smithi160.stdout:108: writing smithi16030472-108 from 2905319 to 3387542 tid 3 2024-09-21T18:07:17.486 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.491 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:17.491 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:17.492 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:17.494 INFO:tasks.rados.rados.0.smithi160.stdout:93: finishing write tid 1 to smithi16030472-93 2024-09-21T18:07:17.494 INFO:tasks.rados.rados.0.smithi160.stdout:93: finishing write tid 2 to smithi16030472-93 2024-09-21T18:07:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:100: finishing write tid 1 to smithi16030472-100 2024-09-21T18:07:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:100: finishing write tid 2 to smithi16030472-100 2024-09-21T18:07:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:100: finishing write tid 3 to smithi16030472-100 2024-09-21T18:07:17.500 INFO:tasks.rados.rados.0.smithi160.stdout:81: finishing write tid 4 to smithi16030472-81 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: finishing write tid 5 to smithi16030472-81 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: oid 81 updating version 0 to 57 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: oid 81 version 57 is already newer than 54 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: oid 81 version 57 is already newer than 53 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: oid 81 version 57 is already newer than 56 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 81 v 57 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: left oid 81 (ObjNum 80 snap 0 seq_num 80) 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:81: done (15 left) 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:109: write initial oid 109 2024-09-21T18:07:17.501 INFO:tasks.rados.rados.0.smithi160.stdout:109: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-21T18:07:17.507 INFO:tasks.rados.rados.0.smithi160.stdout:109: writing smithi16030472-109 from 615156 to 1170990 tid 1 2024-09-21T18:07:17.508 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:17.512 INFO:tasks.rados.rados.0.smithi160.stdout:109: writing smithi16030472-109 from 1603467 to 2019828 tid 2 2024-09-21T18:07:17.520 INFO:tasks.rados.rados.0.smithi160.stdout:109: writing smithi16030472-109 from 2816389 to 3509884 tid 3 2024-09-21T18:07:17.520 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.529 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:17.523+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:17.559 INFO:tasks.rados.rados.0.smithi160.stdout:95: finishing write tid 5 to smithi16030472-95 2024-09-21T18:07:17.559 INFO:tasks.rados.rados.0.smithi160.stdout:95: oid 95 updating version 0 to 27 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:95: oid 95 updating version 27 to 66 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:95: oid 95 version 66 is already newer than 26 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:95: oid 95 updating version 66 to 67 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 95 v 67 (ObjNum 94 snap 0 seq_num 94) dirty exists 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:95: left oid 95 (ObjNum 94 snap 0 seq_num 94) 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:95: done (15 left) 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:110: write initial oid 110 2024-09-21T18:07:17.560 INFO:tasks.rados.rados.0.smithi160.stdout:110: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-21T18:07:17.566 INFO:tasks.rados.rados.0.smithi160.stdout:110: writing smithi16030472-110 from 631963 to 1263046 tid 1 2024-09-21T18:07:17.569 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:17.569 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:17.574 INFO:tasks.rados.rados.0.smithi160.stdout:110: writing smithi16030472-110 from 2061949 to 2621968 tid 2 2024-09-21T18:07:17.580 INFO:tasks.rados.rados.0.smithi160.stdout:110: writing smithi16030472-110 from 3243812 to 3748579 tid 3 2024-09-21T18:07:17.580 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.595 INFO:tasks.rados.rados.0.smithi160.stdout:102: finishing write tid 1 to smithi16030472-102 2024-09-21T18:07:17.595 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:17.596 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:17.612 INFO:tasks.rados.rados.0.smithi160.stdout:100: finishing write tid 4 to smithi16030472-100 2024-09-21T18:07:17.627 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:17.629 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:17.625+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:17.632 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:17.632 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:17.647 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:17.651 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:17.652 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:17.675 INFO:tasks.rados.rados.0.smithi160.stdout:110: finishing write tid 1 to smithi16030472-110 2024-09-21T18:07:17.676 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:17.677 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:17.687 INFO:tasks.rados.rados.0.smithi160.stdout:96: finishing write tid 4 to smithi16030472-96 2024-09-21T18:07:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:96: finishing write tid 5 to smithi16030472-96 2024-09-21T18:07:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:96: oid 96 updating version 0 to 70 2024-09-21T18:07:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:96: oid 96 version 70 is already newer than 29 2024-09-21T18:07:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:96: oid 96 version 70 is already newer than 28 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:96: oid 96 version 70 is already newer than 30 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 96 v 70 (ObjNum 95 snap 0 seq_num 95) dirty exists 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:96: left oid 96 (ObjNum 95 snap 0 seq_num 95) 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:98: finishing write tid 3 to smithi16030472-98 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:96: done (15 left) 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:111: write initial oid 111 2024-09-21T18:07:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:111: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-21T18:07:17.696 INFO:tasks.rados.rados.0.smithi160.stdout:111: writing smithi16030472-111 from 648770 to 1355102 tid 1 2024-09-21T18:07:17.702 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:17.703 INFO:tasks.rados.rados.0.smithi160.stdout:111: writing smithi16030472-111 from 2120431 to 2540461 tid 2 2024-09-21T18:07:17.710 INFO:tasks.rados.rados.0.smithi160.stdout:111: writing smithi16030472-111 from 3271235 to 3703627 tid 3 2024-09-21T18:07:17.710 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:99: finishing write tid 2 to smithi16030472-99 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: finishing write tid 2 to smithi16030472-102 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: finishing write tid 3 to smithi16030472-102 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: finishing write tid 4 to smithi16030472-102 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: finishing write tid 5 to smithi16030472-102 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: oid 102 updating version 0 to 52 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: oid 102 version 52 is already newer than 51 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: oid 102 version 52 is already newer than 49 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: oid 102 version 52 is already newer than 50 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 102 v 52 (ObjNum 101 snap 0 seq_num 101) dirty exists 2024-09-21T18:07:17.711 INFO:tasks.rados.rados.0.smithi160.stdout:102: left oid 102 (ObjNum 101 snap 0 seq_num 101) 2024-09-21T18:07:17.712 INFO:tasks.rados.rados.0.smithi160.stdout:103: finishing write tid 1 to smithi16030472-103 2024-09-21T18:07:17.712 INFO:tasks.rados.rados.0.smithi160.stdout:103: finishing write tid 2 to smithi16030472-103 2024-09-21T18:07:17.712 INFO:tasks.rados.rados.0.smithi160.stdout:102: done (15 left) 2024-09-21T18:07:17.712 INFO:tasks.rados.rados.0.smithi160.stdout:112: write initial oid 112 2024-09-21T18:07:17.712 INFO:tasks.rados.rados.0.smithi160.stdout:112: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-21T18:07:17.718 INFO:tasks.rados.rados.0.smithi160.stdout:112: writing smithi16030472-112 from 665577 to 1447158 tid 1 2024-09-21T18:07:17.724 INFO:tasks.rados.rados.0.smithi160.stdout:112: writing smithi16030472-112 from 2062560 to 2626248 tid 2 2024-09-21T18:07:17.726 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:17.726 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:17.730 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:17.729+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:17.731 INFO:tasks.rados.rados.0.smithi160.stdout:112: writing smithi16030472-112 from 3182305 to 3825969 tid 3 2024-09-21T18:07:17.731 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: finishing write tid 3 to smithi16030472-103 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: finishing write tid 4 to smithi16030472-103 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: finishing write tid 5 to smithi16030472-103 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: oid 103 updating version 0 to 56 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: oid 103 version 56 is already newer than 55 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: oid 103 version 56 is already newer than 53 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: oid 103 version 56 is already newer than 54 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 56 (ObjNum 102 snap 0 seq_num 102) dirty exists 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: left oid 103 (ObjNum 102 snap 0 seq_num 102) 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:104: finishing write tid 1 to smithi16030472-104 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:103: done (15 left) 2024-09-21T18:07:17.732 INFO:tasks.rados.rados.0.smithi160.stdout:113: write initial oid 113 2024-09-21T18:07:17.733 INFO:tasks.rados.rados.0.smithi160.stdout:113: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-21T18:07:17.737 INFO:tasks.rados.rados.0.smithi160.stdout:113: writing smithi16030472-113 from 682384 to 1139214 tid 1 2024-09-21T18:07:17.743 INFO:tasks.rados.rados.0.smithi160.stdout:113: writing smithi16030472-113 from 1721042 to 2144741 tid 2 2024-09-21T18:07:17.749 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:17.749 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:17.749 INFO:tasks.rados.rados.0.smithi160.stdout:113: writing smithi16030472-113 from 2809728 to 3264664 tid 3 2024-09-21T18:07:17.753 INFO:tasks.rados.rados.0.smithi160.stdout:113: writing smithi16030472-113 from 3731357 to 3882384 tid 4 2024-09-21T18:07:17.753 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.764 INFO:tasks.rados.rados.0.smithi160.stdout:104: finishing write tid 2 to smithi16030472-104 2024-09-21T18:07:17.767 INFO:tasks.rados.rados.0.smithi160.stdout:104: finishing write tid 3 to smithi16030472-104 2024-09-21T18:07:17.782 INFO:tasks.rados.rados.0.smithi160.stdout:93: finishing write tid 3 to smithi16030472-93 2024-09-21T18:07:17.782 INFO:tasks.rados.rados.0.smithi160.stdout:93: finishing write tid 4 to smithi16030472-93 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:93: finishing write tid 5 to smithi16030472-93 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:93: oid 93 updating version 0 to 44 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:93: oid 93 version 44 is already newer than 43 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:93: oid 93 updating version 44 to 59 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:93: oid 93 version 59 is already newer than 58 2024-09-21T18:07:17.783 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 93 v 59 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-21T18:07:17.784 INFO:tasks.rados.rados.0.smithi160.stdout:93: left oid 93 (ObjNum 92 snap 0 seq_num 92) 2024-09-21T18:07:17.784 INFO:tasks.rados.rados.0.smithi160.stdout:93: done (15 left) 2024-09-21T18:07:17.784 INFO:tasks.rados.rados.0.smithi160.stdout:114: write initial oid 114 2024-09-21T18:07:17.784 INFO:tasks.rados.rados.0.smithi160.stdout:114: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-21T18:07:17.790 INFO:tasks.rados.rados.0.smithi160.stdout:114: writing smithi16030472-114 from 699191 to 1231270 tid 1 2024-09-21T18:07:17.796 INFO:tasks.rados.rados.0.smithi160.stdout:114: writing smithi16030472-114 from 1779524 to 2346881 tid 2 2024-09-21T18:07:17.804 INFO:tasks.rados.rados.0.smithi160.stdout:114: writing smithi16030472-114 from 2837151 to 3503359 tid 3 2024-09-21T18:07:17.804 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:17.810 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:17.810 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:17.811 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:17.825 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:17.832 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:17.827+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:17.891 INFO:tasks.rados.rados.0.smithi160.stdout:110: finishing write tid 2 to smithi16030472-110 2024-09-21T18:07:17.896 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:17.928 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:17.933 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:17.927+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:17.955 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:17.955 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:77 3.0 e42/31 3:317d9f73:::smithi16030472-20:head v 42'5, mlcod=42'3)", 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.132156+0000", 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.8211933670000002, 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054973799999999996, 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132156+0000", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132156+0000", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132161+0000", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1869999999999999e-06 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132579+0000", 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000418033 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132611+0000", 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.252e-05 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132627+0000", 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6262000000000001e-05 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132793+0000", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016589799999999999 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.132855+0000", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1991999999999999e-05 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.137572+0000", 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0047167060000000002 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.137654+0000", 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1781999999999997e-05 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:101 3.0 e43/31 3:308a1114:::smithi16030472-26:head v 43'6, mlcod=42'4)", 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.365759+0000", 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.5875912520000002, 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053446880000000002, 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.960 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.365759+0000", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.365759+0000", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.365761+0000", 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2299999999999998e-06 2024-09-21T18:07:17.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.366280+0000", 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051906499999999998 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.366308+0000", 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8365999999999999e-05 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.366330+0000", 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1403000000000002e-05 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.366519+0000", 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018953700000000001 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.366588+0000", 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8819000000000006e-05 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.370985+0000", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043972229999999996 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.371103+0000", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118045 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:105 3.0 e43/31 3:0e09a926:::smithi16030472-27:head v 43'7, mlcod=42'5)", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.486957+0000", 2024-09-21T18:07:17.964 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.4663932170000002, 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004144455, 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.486957+0000", 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.486957+0000", 2024-09-21T18:07:17.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.486961+0000", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0659999999999997e-06 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.487808+0000", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084749700000000001 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.487844+0000", 2024-09-21T18:07:17.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5843000000000002e-05 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.487862+0000", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7759000000000001e-05 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.488002+0000", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140278 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.488050+0000", 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7840000000000003e-05 2024-09-21T18:07:17.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.490969+0000", 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029192269999999999 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.491101+0000", 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013194500000000001 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.968 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:121 3.0 e43/31 3:1440a676:::smithi16030472-31:head v 43'8, mlcod=43'6)", 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0596766730000002, 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20492206299999999, 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893677+0000", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.653e-06 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.894421+0000", 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00074468899999999998 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.894451+0000", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9096000000000001e-05 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.994619+0000", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100168902 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.094872+0000", 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100252722 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.094932+0000", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0152000000000003e-05 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.098481+0000", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003548697 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.098595+0000", 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114152 2024-09-21T18:07:17.972 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:157 3.0 e44/31 3:1192aa23:::smithi16030472-40:head v 44'9, mlcod=43'7)", 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.506560+0000", 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.4467896800000002, 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.006133884, 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.506560+0000", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.506560+0000", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.506561+0000", 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.507393+0000", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00083261000000000001 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.507420+0000", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6863999999999999e-05 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.507436+0000", 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6135999999999999e-05 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.507608+0000", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017144400000000001 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.507651+0000", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3056000000000002e-05 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.512610+0000", 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049595780000000001 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.512694+0000", 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3576000000000001e-05 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:165 3.0 e44/31 3:01b9dd46:::smithi16030472-42:head v 44'10, mlcod=43'7)", 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.612216+0000", 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3411338300000004, 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003553924, 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.977 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612216+0000", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612216+0000", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612220+0000", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.873e-06 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612804+0000", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058454799999999995 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612834+0000", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9496e-05 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.612851+0000", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6886000000000001e-05 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.613002+0000", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015106200000000001 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.613039+0000", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7018999999999999e-05 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615655+0000", 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026166119999999999 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615770+0000", 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114428 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:166 3.0 e44/31 3:01b9dd46:::smithi16030472-42:head v 44'11, mlcod=43'7)", 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.615130+0000", 2024-09-21T18:07:17.981 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3382194959999998, 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057786030000000002, 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615130+0000", 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615130+0000", 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615133+0000", 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.19e-06 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616234+0000", 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00110052 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616262+0000", 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7976000000000001e-05 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616284+0000", 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2459e-05 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616419+0000", 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013439800000000001 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616456+0000", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6776000000000002e-05 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.620798+0000", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043428690000000001 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.620909+0000", 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011041500000000001 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.985 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:173 3.0 e44/31 3:364cb6d5:::smithi16030472-44:head v 44'12, mlcod=44'10)", 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.626679+0000", 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3266705539999997, 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0067316240000000003, 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.626679+0000", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.626679+0000", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.626682+0000", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0369999999999999e-06 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.627071+0000", 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038884400000000001 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.627099+0000", 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8096e-05 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.627116+0000", 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6623e-05 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.627260+0000", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144048 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.627300+0000", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0009000000000003e-05 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.633214+0000", 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0059139500000000003 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.633411+0000", 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019701699999999999 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:174 3.0 e44/31 3:364cb6d5:::smithi16030472-44:head v 44'13, mlcod=44'10)", 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.629658+0000", 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3236916369999996, 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048267170000000003, 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.990 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.629658+0000", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.629658+0000", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.629661+0000", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8059999999999999e-06 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.630369+0000", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00070757299999999997 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.630418+0000", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9858999999999998e-05 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.630440+0000", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2033000000000001e-05 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.630581+0000", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000140457 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.630616+0000", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4900000000000001e-05 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634410+0000", 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037945800000000001 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634485+0000", 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4509000000000001e-05 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:175 3.0 e44/31 3:364cb6d5:::smithi16030472-44:head v 44'14, mlcod=44'10)", 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:17.994 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.3193089860000002, 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10156053399999999, 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634044+0000", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4759999999999998e-06 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634062+0000", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7717e-05 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634096+0000", 2024-09-21T18:07:17.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3679000000000001e-05 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734214+0000", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100118467 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734422+0000", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020803400000000001 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734466+0000", 2024-09-21T18:07:17.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3639e-05 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735528+0000", 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00106251 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735601+0000", 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3011999999999998e-05 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:17.998 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:177 3.0 e44/31 3:16d9fc1d:::smithi16030472-45:head v 44'15, mlcod=44'11)", 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.735300+0000", 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2180493439999998, 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0048740629999999997, 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735300+0000", 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:17.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735300+0000", 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735303+0000", 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0029999999999999e-06 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735892+0000", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058829100000000001 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735926+0000", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4409000000000002e-05 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735950+0000", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3974000000000001e-05 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.736078+0000", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012814400000000001 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.736112+0000", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4229e-05 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.740097+0000", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039842280000000002 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.740175+0000", 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7785000000000001e-05 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:181 3.0 e44/31 3:15fed93e:::smithi16030472-46:head v 44'16, mlcod=44'13)", 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.938757+0000", 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.0145928690000003, 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039549049999999999, 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.003 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.938757+0000", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.938757+0000", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.938761+0000", 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7730000000000001e-06 2024-09-21T18:07:18.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.939324+0000", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00056359800000000001 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.939364+0000", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9946e-05 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.939390+0000", 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6063000000000001e-05 2024-09-21T18:07:18.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.939594+0000", 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0002039 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.939636+0000", 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1996000000000002e-05 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.942622+0000", 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0029861929999999998 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.942712+0000", 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9436000000000002e-05 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:182 3.0 e44/31 3:15fed93e:::smithi16030472-46:head v 44'17, mlcod=44'14)", 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.944644+0000", 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.0087053179999996, 2024-09-21T18:07:18.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043009989999999998, 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944644+0000", 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944644+0000", 2024-09-21T18:07:18.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944648+0000", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8029999999999998e-06 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.945572+0000", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00092383899999999998 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.945612+0000", 2024-09-21T18:07:18.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9629000000000003e-05 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.945636+0000", 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4556999999999999e-05 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.945780+0000", 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014337400000000001 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.945814+0000", 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4140000000000002e-05 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.948860+0000", 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030457549999999998 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.948945+0000", 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5902000000000005e-05 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:189 3.0 e44/31 3:1f69a690:::smithi16030472-48:head v 44'18, mlcod=44'15)", 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:14.041858+0000", 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.911491813, 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062292349999999996, 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041858+0000", 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041858+0000", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041861+0000", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2629999999999999e-06 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.042323+0000", 2024-09-21T18:07:18.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046195699999999998 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.042351+0000", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8180000000000001e-05 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.042367+0000", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.042546+0000", 2024-09-21T18:07:18.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017883499999999999 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.042611+0000", 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5204999999999997e-05 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.047994+0000", 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0053825510000000002 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.048087+0000", 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3362000000000005e-05 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:190 3.0 e44/31 3:1f69a690:::smithi16030472-48:head v 44'19, mlcod=44'15)", 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:14.044032+0000", 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.9093182419999999, 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0066226130000000003, 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:18.016 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044032+0000", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044032+0000", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044036+0000", 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0400000000000003e-06 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044724+0000", 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00068805400000000001 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044752+0000", 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8428999999999998e-05 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.044773+0000", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.141e-05 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.045497+0000", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072345600000000003 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.045563+0000", 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6166000000000003e-05 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.050557+0000", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049937929999999998 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.050654+0000", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7264999999999998e-05 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:197 3.0 e44/31 3:11edfedf:::smithi16030472-50:head v 44'20, mlcod=44'17)", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:18.020 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 3.8997138590000002, 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104761221, 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:18.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053639+0000", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2729999999999998e-06 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054586+0000", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00094677300000000004 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054621+0000", 2024-09-21T18:07:18.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5095999999999997e-05 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054646+0000", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4530000000000001e-05 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.154924+0000", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10027860700000001 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.154961+0000", 2024-09-21T18:07:18.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6636e-05 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158262+0000", 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033014049999999999 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158397+0000", 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013490099999999999 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:18.024 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:18.025 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:18.025 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:18.026 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:18.027 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:18.027 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:18.027 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:92: finishing write tid 3 to smithi16030472-92 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: finishing write tid 3 to smithi16030472-110 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: finishing write tid 4 to smithi16030472-110 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: finishing write tid 5 to smithi16030472-110 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: oid 110 updating version 0 to 68 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: oid 110 version 68 is already newer than 67 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: oid 110 updating version 68 to 70 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: oid 110 version 70 is already newer than 69 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 110 v 70 (ObjNum 109 snap 0 seq_num 109) dirty exists 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: left oid 110 (ObjNum 109 snap 0 seq_num 109) 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:110: done (15 left) 2024-09-21T18:07:18.029 INFO:tasks.rados.rados.0.smithi160.stdout:115: write initial oid 115 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:115: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:115: writing smithi16030472-115 from 715998 to 1323326 tid 1 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:115: writing smithi16030472-115 from 1838006 to 2265374 tid 2 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:115: writing smithi16030472-115 from 2864574 to 3342054 tid 3 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:115: writing smithi16030472-115 from 3863835 to 3915998 tid 4 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: finishing write tid 4 to smithi16030472-92 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: finishing write tid 5 to smithi16030472-92 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: oid 92 updating version 0 to 42 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: oid 92 updating version 42 to 61 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: oid 92 updating version 61 to 63 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: oid 92 version 63 is already newer than 41 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 92 v 63 (ObjNum 91 snap 0 seq_num 91) dirty exists 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: left oid 92 (ObjNum 91 snap 0 seq_num 91) 2024-09-21T18:07:18.030 INFO:tasks.rados.rados.0.smithi160.stdout:92: done (15 left) 2024-09-21T18:07:18.031 INFO:tasks.rados.rados.0.smithi160.stdout:116: write initial oid 116 2024-09-21T18:07:18.031 INFO:tasks.rados.rados.0.smithi160.stdout:116: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-21T18:07:18.031 INFO:tasks.rados.rados.0.smithi160.stdout:116: writing smithi16030472-116 from 732805 to 1531735 tid 1 2024-09-21T18:07:18.031 INFO:tasks.rados.rados.0.smithi160.stdout:116: writing smithi16030472-116 from 2296488 to 2867514 tid 2 2024-09-21T18:07:18.033 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:18.031+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:18.035 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:18.035 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:18.036 INFO:tasks.rados.rados.0.smithi160.stdout:116: writing smithi16030472-116 from 3291997 to 3697102 tid 3 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: finishing write tid 3 to smithi16030472-99 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: finishing write tid 4 to smithi16030472-99 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: finishing write tid 5 to smithi16030472-99 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: oid 99 updating version 0 to 75 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: oid 99 version 75 is already newer than 73 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: oid 99 version 75 is already newer than 74 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: oid 99 version 75 is already newer than 33 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 75 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:99: left oid 99 (ObjNum 98 snap 0 seq_num 98) 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:98: finishing write tid 4 to smithi16030472-98 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:98: finishing write tid 5 to smithi16030472-98 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:98: oid 98 updating version 0 to 31 2024-09-21T18:07:18.037 INFO:tasks.rados.rados.0.smithi160.stdout:98: oid 98 updating version 31 to 32 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:98: oid 98 updating version 32 to 71 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:98: oid 98 updating version 71 to 76 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 98 v 76 (ObjNum 97 snap 0 seq_num 97) dirty exists 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:98: left oid 98 (ObjNum 97 snap 0 seq_num 97) 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:98: done (15 left) 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:99: done (14 left) 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:117: write initial oid 117 2024-09-21T18:07:18.038 INFO:tasks.rados.rados.0.smithi160.stdout:117: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-21T18:07:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:117: writing smithi16030472-117 from 749612 to 1223791 tid 1 2024-09-21T18:07:18.052 INFO:tasks.rados.rados.0.smithi160.stdout:117: writing smithi16030472-117 from 1954970 to 2386007 tid 2 2024-09-21T18:07:18.059 INFO:tasks.rados.rados.0.smithi160.stdout:117: writing smithi16030472-117 from 2919420 to 3535797 tid 3 2024-09-21T18:07:18.060 INFO:tasks.rados.rados.0.smithi160.stdout:118: write initial oid 118 2024-09-21T18:07:18.060 INFO:tasks.rados.rados.0.smithi160.stdout:118: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-21T18:07:18.065 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:18.067 INFO:tasks.rados.rados.0.smithi160.stdout:118: writing smithi16030472-118 from 766419 to 1315847 tid 1 2024-09-21T18:07:18.074 INFO:tasks.rados.rados.0.smithi160.stdout:118: writing smithi16030472-118 from 2013452 to 2588147 tid 2 2024-09-21T18:07:18.081 INFO:tasks.rados.rados.0.smithi160.stdout:118: writing smithi16030472-118 from 3346843 to 3774492 tid 3 2024-09-21T18:07:18.081 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: finishing write tid 5 to smithi16030472-100 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: oid 100 updating version 0 to 45 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: oid 100 updating version 45 to 48 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: oid 100 version 48 is already newer than 46 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: oid 100 version 48 is already newer than 47 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 100 v 48 (ObjNum 99 snap 0 seq_num 99) dirty exists 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: left oid 100 (ObjNum 99 snap 0 seq_num 99) 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:100: done (15 left) 2024-09-21T18:07:18.100 INFO:tasks.rados.rados.0.smithi160.stdout:119: write initial oid 119 2024-09-21T18:07:18.101 INFO:tasks.rados.rados.0.smithi160.stdout:119: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-21T18:07:18.107 INFO:tasks.rados.rados.0.smithi160.stdout:119: writing smithi16030472-119 from 783226 to 1407903 tid 1 2024-09-21T18:07:18.114 INFO:tasks.rados.rados.0.smithi160.stdout:119: writing smithi16030472-119 from 2071934 to 2506640 tid 2 2024-09-21T18:07:18.120 INFO:tasks.rados.rados.0.smithi160.stdout:119: writing smithi16030472-119 from 2974266 to 3613187 tid 3 2024-09-21T18:07:18.120 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.134 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:18.131+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:18.160 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:18.160 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:18.160 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:18.160 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:18.175 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:18.194 INFO:tasks.rados.rados.0.smithi160.stdout:101: finishing write tid 1 to smithi16030472-101 2024-09-21T18:07:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:07:18.235 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:18.233+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:18.295 INFO:tasks.rados.rados.0.smithi160.stdout:101: finishing write tid 2 to smithi16030472-101 2024-09-21T18:07:18.295 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:18.296 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:18.296 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:18.296 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:18.296 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:18.296 INFO:tasks.rados.rados.0.smithi160.stdout:101: finishing write tid 3 to smithi16030472-101 2024-09-21T18:07:18.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 1 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 2 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 3 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 4 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 5 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: finishing write tid 6 to smithi16030472-115 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: oid 115 updating version 0 to 71 2024-09-21T18:07:18.318 INFO:tasks.rados.rados.0.smithi160.stdout:115: oid 115 updating version 71 to 75 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:115: oid 115 version 75 is already newer than 72 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:115: oid 115 version 75 is already newer than 73 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:115: oid 115 version 75 is already newer than 74 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 115 v 75 (ObjNum 114 snap 0 seq_num 114) dirty exists 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:115: left oid 115 (ObjNum 114 snap 0 seq_num 114) 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:104: finishing write tid 4 to smithi16030472-104 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:105: finishing write tid 1 to smithi16030472-105 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:105: finishing write tid 2 to smithi16030472-105 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:115: done (15 left) 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:120: write initial oid 120 2024-09-21T18:07:18.319 INFO:tasks.rados.rados.0.smithi160.stdout:120: seq_num 119 ranges {400033=699926,1614063=385970} 2024-09-21T18:07:18.328 INFO:tasks.rados.rados.0.smithi160.stdout:120: writing smithi16030472-120 from 400033 to 1099959 tid 1 2024-09-21T18:07:18.336 INFO:tasks.rados.rados.0.smithi160.stdout:120: writing smithi16030472-120 from 1614063 to 2000033 tid 2 2024-09-21T18:07:18.336 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.336 INFO:tasks.rados.rados.0.smithi160.stdout:105: finishing write tid 3 to smithi16030472-105 2024-09-21T18:07:18.336 INFO:tasks.rados.rados.0.smithi160.stdout:116: finishing write tid 1 to smithi16030472-116 2024-09-21T18:07:18.337 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:18.331+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:18.432 INFO:tasks.rados.rados.0.smithi160.stdout:101: finishing write tid 4 to smithi16030472-101 2024-09-21T18:07:18.432 INFO:tasks.rados.rados.0.smithi160.stdout:106: finishing write tid 1 to smithi16030472-106 2024-09-21T18:07:18.432 INFO:tasks.rados.rados.0.smithi160.stdout:106: finishing write tid 2 to smithi16030472-106 2024-09-21T18:07:18.437 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:18.437 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:18.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:18.437 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:18.437 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:18.437 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:18.431+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:18.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: finishing write tid 5 to smithi16030472-101 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: oid 101 updating version 0 to 37 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: oid 101 version 37 is already newer than 34 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: oid 101 version 37 is already newer than 35 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: oid 101 version 37 is already newer than 36 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 101 v 37 (ObjNum 100 snap 0 seq_num 100) dirty exists 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: left oid 101 (ObjNum 100 snap 0 seq_num 100) 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:101: done (15 left) 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:121: write initial oid 121 2024-09-21T18:07:18.508 INFO:tasks.rados.rados.0.smithi160.stdout:121: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-21T18:07:18.519 INFO:tasks.rados.rados.0.smithi160.stdout:121: writing smithi16030472-121 from 416840 to 1192015 tid 1 2024-09-21T18:07:18.526 INFO:tasks.rados.rados.0.smithi160.stdout:121: writing smithi16030472-121 from 1672545 to 2016840 tid 2 2024-09-21T18:07:18.526 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.538 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:18.533+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:18.583 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:525 3.3 e46/31)", 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.15148250899999999, 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15150032999999999, 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432100+0000", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1060000000000002e-06 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432578+0000", 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00047806600000000003 2024-09-21T18:07:18.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432610+0000", 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1511999999999999e-05 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432625+0000", 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5426000000000001e-05 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:526 3.3 e46/31)", 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.14433215599999999, 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14437575899999999, 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439250+0000", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0199999999999999e-06 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439262+0000", 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2193e-05 2024-09-21T18:07:18.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439288+0000", 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6126e-05 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439302+0000", 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3686000000000001e-05 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.25265990199999999, 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25272905699999998, 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:07:18.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:07:18.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 3 2024-09-21T18:07:18.593 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:18.598 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: finishing write tid 4 to smithi16030472-105 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: finishing write tid 5 to smithi16030472-105 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: oid 105 updating version 0 to 61 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: oid 105 updating version 61 to 63 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: oid 105 version 63 is already newer than 62 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: oid 105 updating version 63 to 64 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 105 v 64 (ObjNum 104 snap 0 seq_num 104) dirty exists 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: left oid 105 (ObjNum 104 snap 0 seq_num 104) 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:109: finishing write tid 1 to smithi16030472-109 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:105: done (15 left) 2024-09-21T18:07:18.629 INFO:tasks.rados.rados.0.smithi160.stdout:122: write initial oid 122 2024-09-21T18:07:18.630 INFO:tasks.rados.rados.0.smithi160.stdout:122: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-21T18:07:18.639 INFO:tasks.rados.rados.0.smithi160.stdout:122: writing smithi16030472-122 from 433647 to 884071 tid 1 2024-09-21T18:07:18.639 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:18.635+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:122: writing smithi16030472-122 from 1331027 to 1913060 tid 2 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:104: finishing write tid 5 to smithi16030472-104 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:104: oid 104 updating version 0 to 58 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:104: oid 104 updating version 58 to 60 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:104: oid 104 version 60 is already newer than 57 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:104: oid 104 version 60 is already newer than 59 2024-09-21T18:07:18.650 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 104 v 60 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:104: left oid 104 (ObjNum 103 snap 0 seq_num 103) 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: finishing write tid 2 to smithi16030472-109 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: finishing write tid 3 to smithi16030472-109 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: finishing write tid 4 to smithi16030472-109 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: finishing write tid 5 to smithi16030472-109 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: oid 109 updating version 0 to 65 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: oid 109 updating version 65 to 67 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: oid 109 updating version 67 to 68 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: oid 109 version 68 is already newer than 66 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 68 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:109: left oid 109 (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:114: finishing write tid 1 to smithi16030472-114 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:114: finishing write tid 2 to smithi16030472-114 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:114: finishing write tid 3 to smithi16030472-114 2024-09-21T18:07:18.651 INFO:tasks.rados.rados.0.smithi160.stdout:114: finishing write tid 4 to smithi16030472-114 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: finishing write tid 5 to smithi16030472-114 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: oid 114 updating version 0 to 71 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: oid 114 version 71 is already newer than 69 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: oid 114 updating version 71 to 72 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: oid 114 version 72 is already newer than 70 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 72 (ObjNum 113 snap 0 seq_num 113) dirty exists 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:114: left oid 114 (ObjNum 113 snap 0 seq_num 113) 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:117: finishing write tid 1 to smithi16030472-117 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: finishing write tid 2 to smithi16030472-116 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: finishing write tid 3 to smithi16030472-116 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: finishing write tid 4 to smithi16030472-116 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: finishing write tid 5 to smithi16030472-116 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: oid 116 updating version 0 to 78 2024-09-21T18:07:18.652 INFO:tasks.rados.rados.0.smithi160.stdout:116: oid 116 version 78 is already newer than 77 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:116: oid 116 version 78 is already newer than 76 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:116: oid 116 updating version 78 to 79 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 79 (ObjNum 115 snap 0 seq_num 115) dirty exists 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:116: left oid 116 (ObjNum 115 snap 0 seq_num 115) 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:118: finishing write tid 1 to smithi16030472-118 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:118: finishing write tid 2 to smithi16030472-118 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:104: done (15 left) 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:109: done (14 left) 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:114: done (13 left) 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:116: done (12 left) 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:123: write initial oid 123 2024-09-21T18:07:18.653 INFO:tasks.rados.rados.0.smithi160.stdout:123: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-21T18:07:18.661 INFO:tasks.rados.rados.0.smithi160.stdout:123: writing smithi16030472-123 from 450454 to 1092480 tid 1 2024-09-21T18:07:18.672 INFO:tasks.rados.rados.0.smithi160.stdout:123: writing smithi16030472-123 from 1505862 to 2050454 tid 2 2024-09-21T18:07:18.672 INFO:tasks.rados.rados.0.smithi160.stdout:124: write initial oid 124 2024-09-21T18:07:18.672 INFO:tasks.rados.rados.0.smithi160.stdout:124: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-21T18:07:18.684 INFO:tasks.rados.rados.0.smithi160.stdout:124: writing smithi16030472-124 from 467261 to 1184536 tid 1 2024-09-21T18:07:18.692 INFO:tasks.rados.rados.0.smithi160.stdout:124: writing smithi16030472-124 from 1847991 to 2067261 tid 2 2024-09-21T18:07:18.692 INFO:tasks.rados.rados.0.smithi160.stdout:125: write initial oid 125 2024-09-21T18:07:18.692 INFO:tasks.rados.rados.0.smithi160.stdout:125: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-21T18:07:18.702 INFO:tasks.rados.rados.0.smithi160.stdout:125: writing smithi16030472-125 from 484068 to 1276592 tid 1 2024-09-21T18:07:18.709 INFO:tasks.rados.rados.0.smithi160.stdout:125: writing smithi16030472-125 from 1906473 to 2084068 tid 2 2024-09-21T18:07:18.709 INFO:tasks.rados.rados.0.smithi160.stdout:126: write initial oid 126 2024-09-21T18:07:18.709 INFO:tasks.rados.rados.0.smithi160.stdout:126: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-21T18:07:18.715 INFO:tasks.rados.rados.0.smithi160.stdout:126: writing smithi16030472-126 from 500875 to 968648 tid 1 2024-09-21T18:07:18.722 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:18.722 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:18.722 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:18.722 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:18.722 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:126: writing smithi16030472-126 from 1564955 to 2100875 tid 2 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: finishing write tid 3 to smithi16030472-118 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: finishing write tid 4 to smithi16030472-118 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: finishing write tid 5 to smithi16030472-118 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: oid 118 updating version 0 to 80 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: oid 118 updating version 80 to 82 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: oid 118 updating version 82 to 83 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: oid 118 version 83 is already newer than 81 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 118 v 83 (ObjNum 117 snap 0 seq_num 117) dirty exists 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: left oid 118 (ObjNum 117 snap 0 seq_num 117) 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:118: done (15 left) 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:127: write initial oid 127 2024-09-21T18:07:18.723 INFO:tasks.rados.rados.0.smithi160.stdout:127: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-21T18:07:18.729 INFO:tasks.rados.rados.0.smithi160.stdout:127: writing smithi16030472-127 from 517682 to 1060704 tid 1 2024-09-21T18:07:18.735 INFO:tasks.rados.rados.0.smithi160.stdout:127: writing smithi16030472-127 from 1623437 to 2117682 tid 2 2024-09-21T18:07:18.735 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:18.739 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:18.735+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:18.839 INFO:tasks.rados.rados.0.smithi160.stdout:121: finishing write tid 1 to smithi16030472-121 2024-09-21T18:07:18.839 INFO:tasks.rados.rados.0.smithi160.stdout:121: finishing write tid 2 to smithi16030472-121 2024-09-21T18:07:18.839 INFO:tasks.rados.rados.0.smithi160.stdout:121: finishing write tid 3 to smithi16030472-121 2024-09-21T18:07:18.840 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:18.835+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:18.855 INFO:tasks.rados.rados.0.smithi160.stdout:122: finishing write tid 1 to smithi16030472-122 2024-09-21T18:07:18.856 INFO:tasks.rados.rados.0.smithi160.stdout:122: finishing write tid 2 to smithi16030472-122 2024-09-21T18:07:18.862 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 32.970584031000001, 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10982987, 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.863 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 5 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889121+0000", 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889126+0000", 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6820000000000002e-06 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889143+0000", 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7246000000000001e-05 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889177+0000", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3584999999999998e-05 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889196+0000", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9658999999999999e-05 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889338+0000", 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000141996 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op must wait for map", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.889348+0000", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0261999999999999e-05 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.895716+0000", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0063680359999999997 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.895936+0000", 2024-09-21T18:07:18.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00021955799999999999 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 12", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.896153+0000", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000216481 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.897514+0000", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013610479999999999 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998806+0000", 2024-09-21T18:07:18.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10129218500000001 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998885+0000", 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9565000000000001e-05 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:45.998951+0000", 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5567000000000004e-05 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_op(client.4347.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.979673626, 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20442987400000001, 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client_info": { 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client": "client.4347", 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "client_addr": "v1:172.21.15.29:0/2681628713", 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "tid": 7 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880031+0000", 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880035+0000", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0629999999999999e-06 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880049+0000", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3253e-05 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880082+0000", 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3408000000000003e-05 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.880103+0000", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0628999999999998e-05 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.888592+0000", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0084889909999999996 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op must wait for map", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.888603+0000", 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1369000000000001e-05 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981159+0000", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.092556303000000006 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981307+0000", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000147319 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "waiting for subops from 12", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.981546+0000", 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000239368 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "op_commit", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:46.983148+0000", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0016015700000000001 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084293+0000", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101145496 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084384+0000", 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1259000000000002e-05 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:06:47.084461+0000", 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6846000000000003e-05 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:55 4.1 e42/33 4:b9fb878b:::smithi16030472-14:head v 42'7, mlcod=42'1)", 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.035375+0000", 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.8243301360000004, 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20219168200000001, 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.876 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035375+0000", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035375+0000", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035378+0000", 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6029999999999999e-06 2024-09-21T18:07:18.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035393+0000", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4983e-05 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035411+0000", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8213e-05 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035430+0000", 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9296e-05 2024-09-21T18:07:18.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.235559+0000", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20012854299999999 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.235588+0000", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9158000000000001e-05 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.237460+0000", 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0018726369999999999 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.237567+0000", 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010624900000000001 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:30 4.1 e42/33 4:92089e4d:::smithi16030472-8:head v 42'8, mlcod=42'1)", 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.035447+0000", 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.8242580879999997, 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20218657000000001, 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.880 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035447+0000", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035447+0000", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035448+0000", 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.063e-06 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035635+0000", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018661099999999999 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.035648+0000", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3324e-05 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.135728+0000", 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100079637 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.235849+0000", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100121583 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.235871+0000", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2249e-05 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.237572+0000", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0017009589999999999 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.237634+0000", 2024-09-21T18:07:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1143999999999996e-05 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:130 3.3 e43/31 3:f5c0ad04:::smithi16030472-33:head v 43'15, mlcod=43'10)", 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.953333+0000", 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.9063724940000002, 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.202359394, 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.953333+0000", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.953333+0000", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.953335+0000", 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.953988+0000", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065201899999999999 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.954019+0000", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1015e-05 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.954038+0000", 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9763e-05 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.053270+0000", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099231352999999994 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.053315+0000", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5442999999999998e-05 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.155600+0000", 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102285297 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.155692+0000", 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1574000000000001e-05 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:131 3.3 e43/31 3:f5c0ad04:::smithi16030472-33:head v 43'16, mlcod=43'10)", 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.955489+0000", 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.9042160040000002, 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200837457, 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.888 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955489+0000", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955489+0000", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955491+0000", 2024-09-21T18:07:18.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3470000000000001e-06 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955502+0000", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0852999999999999e-05 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955521+0000", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9176000000000001e-05 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.955533+0000", 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1486e-05 2024-09-21T18:07:18.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.155254+0000", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19972158100000001 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.155294+0000", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9048000000000003e-05 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.156244+0000", 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00095084800000000001 2024-09-21T18:07:18.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.156327+0000", 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2118000000000002e-05 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:138 3.3 e43/31 3:cd35ea38:::smithi16030472-35:head v 43'18, mlcod=43'15)", 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.701279+0000", 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.1584255829999996, 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108543985, 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.892 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.701279+0000", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.701279+0000", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.701280+0000", 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9599999999999999e-07 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702269+0000", 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098844699999999994 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702295+0000", 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6503e-05 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702309+0000", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4149e-05 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.802667+0000", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100357352 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.802708+0000", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1306000000000001e-05 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.809752+0000", 2024-09-21T18:07:18.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007043891 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.809823+0000", 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1741000000000003e-05 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:141 3.3 e44/31 3:d903dcdd:::smithi16030472-36:head v 44'19, mlcod=43'16)", 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.702340+0000", 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.1573648189999997, 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10753149400000001, 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.896 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702340+0000", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702340+0000", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702342+0000", 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.33e-06 2024-09-21T18:07:18.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702791+0000", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00044989500000000001 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.702815+0000", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3799999999999999e-05 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.802902+0000", 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100086551 2024-09-21T18:07:18.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.804488+0000", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001585772 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.804548+0000", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0108000000000001e-05 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.809833+0000", 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052853850000000001 2024-09-21T18:07:18.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.809872+0000", 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8652999999999997e-05 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:459 3.3 e46/31 3:d13927db:::smithi16030472-104:head v 46'60, mlcod=46'57)", 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.013593+0000", 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.84611175800000005, 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108496378, 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.900 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.013593+0000", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.013593+0000", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.013597+0000", 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5769999999999998e-06 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.013689+0000", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2151000000000006e-05 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.014633+0000", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00094373299999999999 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.014652+0000", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9032999999999999e-05 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.114878+0000", 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100226259 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.114918+0000", 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0198999999999999e-05 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121987+0000", 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007068675 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122090+0000", 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010275100000000001 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:461 3.3 e46/31 3:c168376b:::smithi16030472-105:head v 46'61, mlcod=46'57)", 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.014692+0000", 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.84501290799999995, 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10990237899999999, 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.014692+0000", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.014692+0000", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.014694+0000", 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8500000000000001e-06 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.015068+0000", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00037425400000000003 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.015087+0000", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8785999999999999e-05 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.115186+0000", 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100099199 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.115372+0000", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018559199999999999 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.115433+0000", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1321999999999993e-05 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124539+0000", 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091056809999999992 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124595+0000", 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5695000000000001e-05 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:482 3.3 e46/31 3:ccecab68:::smithi16030472-109:head v 46'66, mlcod=46'58)", 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.74107683099999999, 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212039585, 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.908 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118630+0000", 2024-09-21T18:07:18.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8640000000000001e-06 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119297+0000", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066705300000000003 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119316+0000", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9245999999999998e-05 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119327+0000", 2024-09-21T18:07:18.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0842999999999999e-05 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124681+0000", 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0053538709999999996 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124708+0000", 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6931999999999999e-05 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330606+0000", 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20589761400000001 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330668+0000", 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2162000000000006e-05 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:483 3.3 e46/31 3:ccecab68:::smithi16030472-109:head v 46'67, mlcod=46'58)", 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.119346+0000", 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.74035920200000005, 2024-09-21T18:07:18.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21136380199999999, 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119346+0000", 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119346+0000", 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119347+0000", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3370000000000001e-06 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120278+0000", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093083700000000001 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120295+0000", 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7342000000000001e-05 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120309+0000", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3390000000000001e-05 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.126266+0000", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059567630000000003 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.126293+0000", 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7219000000000001e-05 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330676+0000", 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20438357700000001 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330710+0000", 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3337000000000003e-05 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:484 3.3 e46/31 3:ccecab68:::smithi16030472-109:head v 46'68, mlcod=46'58)", 2024-09-21T18:07:18.916 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.120331+0000", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.73937410000000003, 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21041252499999999, 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120331+0000", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120331+0000", 2024-09-21T18:07:18.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120332+0000", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4959999999999999e-06 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120343+0000", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0859999999999999e-05 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120357+0000", 2024-09-21T18:07:18.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3477e-05 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120376+0000", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9296e-05 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.222795+0000", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102419263 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.222861+0000", 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6049000000000004e-05 2024-09-21T18:07:18.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330713+0000", 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10785172799999999 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330743+0000", 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0355999999999999e-05 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.920 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:507 3.3 e46/31 3:f63a4439:::smithi16030472-114:head v 46'69, mlcod=46'58)", 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.120397+0000", 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.739308458, 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210378019, 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120397+0000", 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120397+0000", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.120398+0000", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1200000000000001e-06 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121236+0000", 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083842200000000002 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121252+0000", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5393e-05 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121261+0000", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.6660000000000003e-06 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.223447+0000", 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10218582700000001 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.223499+0000", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2292e-05 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330745+0000", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10724618 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330775+0000", 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9119e-05 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:508 3.3 e46/31 3:f63a4439:::smithi16030472-114:head v 46'70, mlcod=46'58)", 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.121279+0000", 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.73842649999999999, 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209533783, 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121279+0000", 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121279+0000", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.121280+0000", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122018+0000", 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073817200000000005 2024-09-21T18:07:18.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122033+0000", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5619000000000001e-05 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122041+0000", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.464e-06 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.325779+0000", 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20373851900000001 2024-09-21T18:07:18.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.325814+0000", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4873000000000003e-05 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330777+0000", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0049625809999999998 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330812+0000", 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5432000000000002e-05 2024-09-21T18:07:18.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:509 3.3 e46/31 3:f63a4439:::smithi16030472-114:head v 46'71, mlcod=46'58)", 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.122054+0000", 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.737651379, 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20930765900000001, 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122054+0000", 2024-09-21T18:07:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122054+0000", 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.122055+0000", 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.113e-06 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123236+0000", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001181267 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123267+0000", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0456000000000001e-05 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123284+0000", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7399999999999999e-05 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.327870+0000", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204586513 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.327901+0000", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1065000000000002e-05 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331273+0000", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033712210000000002 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331361+0000", 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8623999999999997e-05 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:510 3.3 e46/31 3:f63a4439:::smithi16030472-114:head v 46'72, mlcod=46'58)", 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.123312+0000", 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.73639306400000004, 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21003671199999999, 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123312+0000", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123312+0000", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123314+0000", 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9300000000000002e-06 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123352+0000", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7839000000000001e-05 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.123364+0000", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2403e-05 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.223460+0000", 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100096031 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330264+0000", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106803462 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330296+0000", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2459000000000002e-05 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.333285+0000", 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029884260000000002 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.333349+0000", 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4162e-05 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.52878629300000002, 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.937 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:07:18.938 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:07:18.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:525 3.3 e46/31)", 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.42760890000000001, 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.201668293, 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.941 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432096+0000", 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432100+0000", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1060000000000002e-06 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432578+0000", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00047806600000000003 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432610+0000", 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1511999999999999e-05 2024-09-21T18:07:18.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.432625+0000", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5426000000000001e-05 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.631947+0000", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.199321797 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.631970+0000", 2024-09-21T18:07:18.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2671999999999999e-05 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633726+0000", 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001755936 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633764+0000", 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8778e-05 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:526 3.3 e46/31)", 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.42045854700000002, 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.194719961, 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439247+0000", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439250+0000", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0199999999999999e-06 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439262+0000", 2024-09-21T18:07:18.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2193e-05 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439288+0000", 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6126e-05 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.439302+0000", 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3686000000000001e-05 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:18.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633390+0000", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19408810700000001 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633407+0000", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7708000000000001e-05 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633939+0000", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053153599999999999 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:18.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633966+0000", 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7585000000000001e-05 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:18.950 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:18.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:18.953 INFO:tasks.rados.rados.0.smithi160.stdout:121: finishing write tid 4 to smithi16030472-121 2024-09-21T18:07:18.953 INFO:tasks.rados.rados.0.smithi160.stdout:121: oid 121 updating version 0 to 51 2024-09-21T18:07:18.953 INFO:tasks.rados.rados.0.smithi160.stdout:121: oid 121 version 51 is already newer than 49 2024-09-21T18:07:18.953 INFO:tasks.rados.rados.0.smithi160.stdout:121: oid 121 version 51 is already newer than 50 2024-09-21T18:07:18.953 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 121 v 51 (ObjNum 120 snap 0 seq_num 120) dirty exists 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:121: left oid 121 (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:121: done (15 left) 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:128: write initial oid 128 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:128: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:128: writing smithi16030472-128 from 534489 to 1152760 tid 1 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:128: writing smithi16030472-128 from 1565566 to 2134489 tid 2 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:117: finishing write tid 2 to smithi16030472-117 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:117: finishing write tid 3 to smithi16030472-117 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:117: finishing write tid 4 to smithi16030472-117 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:120: finishing write tid 1 to smithi16030472-120 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:120: finishing write tid 2 to smithi16030472-120 2024-09-21T18:07:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:120: finishing write tid 3 to smithi16030472-120 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: finishing write tid 4 to smithi16030472-120 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: oid 120 updating version 0 to 77 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: oid 120 updating version 77 to 78 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: oid 120 updating version 78 to 79 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 120 v 79 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: left oid 120 (ObjNum 119 snap 0 seq_num 119) 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:120: done (15 left) 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:129: write initial oid 129 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:129: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:129: writing smithi16030472-129 from 551296 to 1244816 tid 1 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout:129: writing smithi16030472-129 from 2024048 to 2151296 tid 2 2024-09-21T18:07:18.955 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:18.956 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:18.949+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:19.053 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:19.049+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:19.081 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.081 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:19.081 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:19.081 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.095 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:119: finishing write tid 1 to smithi16030472-119 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:119: finishing write tid 2 to smithi16030472-119 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: finishing write tid 5 to smithi16030472-117 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: oid 117 updating version 0 to 75 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: oid 117 version 75 is already newer than 74 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: oid 117 updating version 75 to 76 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: oid 117 version 76 is already newer than 73 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 117 v 76 (ObjNum 116 snap 0 seq_num 116) dirty exists 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: left oid 117 (ObjNum 116 snap 0 seq_num 116) 2024-09-21T18:07:19.131 INFO:tasks.rados.rados.0.smithi160.stdout:117: done (15 left) 2024-09-21T18:07:19.132 INFO:tasks.rados.rados.0.smithi160.stdout:130: write initial oid 130 2024-09-21T18:07:19.132 INFO:tasks.rados.rados.0.smithi160.stdout:130: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-21T18:07:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:130: writing smithi16030472-130 from 568103 to 1336872 tid 1 2024-09-21T18:07:19.148 INFO:tasks.rados.rados.0.smithi160.stdout:130: writing smithi16030472-130 from 2082530 to 2168103 tid 2 2024-09-21T18:07:19.148 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.148 INFO:tasks.rados.rados.0.smithi160.stdout:106: finishing write tid 3 to smithi16030472-106 2024-09-21T18:07:19.148 INFO:tasks.rados.rados.0.smithi160.stdout:106: finishing write tid 4 to smithi16030472-106 2024-09-21T18:07:19.154 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:19.149+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:19.218 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.218 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:19.218 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:19.218 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:19.218 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:19.235 INFO:tasks.rados.rados.0.smithi160.stdout:122: finishing write tid 3 to smithi16030472-122 2024-09-21T18:07:19.235 INFO:tasks.rados.rados.0.smithi160.stdout:122: finishing write tid 4 to smithi16030472-122 2024-09-21T18:07:19.235 INFO:tasks.rados.rados.0.smithi160.stdout:122: oid 122 updating version 0 to 52 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:122: oid 122 updating version 52 to 53 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:122: oid 122 updating version 53 to 54 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 122 v 54 (ObjNum 121 snap 0 seq_num 121) dirty exists 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:122: left oid 122 (ObjNum 121 snap 0 seq_num 121) 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:122: done (15 left) 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:131: write initial oid 131 2024-09-21T18:07:19.236 INFO:tasks.rados.rados.0.smithi160.stdout:131: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-21T18:07:19.246 INFO:tasks.rados.rados.0.smithi160.stdout:131: writing smithi16030472-131 from 584910 to 1145281 tid 1 2024-09-21T18:07:19.255 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:19.253+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:19.256 INFO:tasks.rados.rados.0.smithi160.stdout:131: writing smithi16030472-131 from 1857365 to 2184910 tid 2 2024-09-21T18:07:19.256 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.256 INFO:tasks.rados.rados.0.smithi160.stdout:128: finishing write tid 1 to smithi16030472-128 2024-09-21T18:07:19.256 INFO:tasks.rados.rados.0.smithi160.stdout:128: finishing write tid 2 to smithi16030472-128 2024-09-21T18:07:19.339 INFO:tasks.rados.rados.0.smithi160.stdout:119: finishing write tid 3 to smithi16030472-119 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: finishing write tid 4 to smithi16030472-119 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: finishing write tid 5 to smithi16030472-119 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: oid 119 updating version 0 to 86 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: oid 119 version 86 is already newer than 85 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: oid 119 updating version 86 to 87 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: oid 119 version 87 is already newer than 84 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 119 v 87 (ObjNum 118 snap 0 seq_num 118) dirty exists 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:119: left oid 119 (ObjNum 118 snap 0 seq_num 118) 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:125: finishing write tid 1 to smithi16030472-125 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:125: finishing write tid 2 to smithi16030472-125 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:125: finishing write tid 3 to smithi16030472-125 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:125: finishing write tid 4 to smithi16030472-125 2024-09-21T18:07:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:125: oid 125 updating version 0 to 88 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:125: oid 125 updating version 88 to 90 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:125: oid 125 version 90 is already newer than 89 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 125 v 90 (ObjNum 124 snap 0 seq_num 124) dirty exists 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:125: left oid 125 (ObjNum 124 snap 0 seq_num 124) 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:126: finishing write tid 1 to smithi16030472-126 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:126: finishing write tid 2 to smithi16030472-126 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:126: finishing write tid 3 to smithi16030472-126 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:119: done (15 left) 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:125: done (14 left) 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:132: write initial oid 132 2024-09-21T18:07:19.341 INFO:tasks.rados.rados.0.smithi160.stdout:132: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-21T18:07:19.353 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.353 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:19.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:19.353 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:07:19.353 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.357 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:19.351+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:19.357 INFO:tasks.rados.rados.0.smithi160.stdout:132: writing smithi16030472-132 from 601717 to 1237337 tid 1 2024-09-21T18:07:19.367 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:19.370 INFO:tasks.rados.rados.0.smithi160.stdout:132: writing smithi16030472-132 from 1799494 to 2201717 tid 2 2024-09-21T18:07:19.370 INFO:tasks.rados.rados.0.smithi160.stdout:133: write initial oid 133 2024-09-21T18:07:19.370 INFO:tasks.rados.rados.0.smithi160.stdout:133: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-21T18:07:19.388 INFO:tasks.rados.rados.0.smithi160.stdout:133: writing smithi16030472-133 from 618524 to 1329393 tid 1 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:133: writing smithi16030472-133 from 1857976 to 2218524 tid 2 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: finishing write tid 1 to smithi16030472-107 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: finishing write tid 2 to smithi16030472-107 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: finishing write tid 3 to smithi16030472-107 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: finishing write tid 4 to smithi16030472-107 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: finishing write tid 5 to smithi16030472-107 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: oid 107 updating version 0 to 42 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: oid 107 updating version 42 to 45 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: oid 107 version 45 is already newer than 44 2024-09-21T18:07:19.399 INFO:tasks.rados.rados.0.smithi160.stdout:107: oid 107 version 45 is already newer than 43 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 107 v 45 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:107: left oid 107 (ObjNum 106 snap 0 seq_num 106) 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:108: finishing write tid 1 to smithi16030472-108 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:108: finishing write tid 2 to smithi16030472-108 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: finishing write tid 1 to smithi16030472-111 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: finishing write tid 2 to smithi16030472-111 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: finishing write tid 3 to smithi16030472-111 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: finishing write tid 4 to smithi16030472-111 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: finishing write tid 5 to smithi16030472-111 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: oid 111 updating version 0 to 49 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: oid 111 version 49 is already newer than 48 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: oid 111 updating version 49 to 50 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: oid 111 updating version 50 to 51 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 111 v 51 (ObjNum 110 snap 0 seq_num 110) dirty exists 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: left oid 111 (ObjNum 110 snap 0 seq_num 110) 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:107: done (15 left) 2024-09-21T18:07:19.400 INFO:tasks.rados.rados.0.smithi160.stdout:111: done (14 left) 2024-09-21T18:07:19.401 INFO:tasks.rados.rados.0.smithi160.stdout:134: write initial oid 134 2024-09-21T18:07:19.401 INFO:tasks.rados.rados.0.smithi160.stdout:134: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-21T18:07:19.414 INFO:tasks.rados.rados.0.smithi160.stdout:134: writing smithi16030472-134 from 635331 to 1421449 tid 1 2024-09-21T18:07:19.423 INFO:tasks.rados.rados.0.smithi160.stdout:134: writing smithi16030472-134 from 1916458 to 2235331 tid 2 2024-09-21T18:07:19.424 INFO:tasks.rados.rados.0.smithi160.stdout:135: write initial oid 135 2024-09-21T18:07:19.424 INFO:tasks.rados.rados.0.smithi160.stdout:135: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-21T18:07:19.435 INFO:tasks.rados.rados.0.smithi160.stdout:135: writing smithi16030472-135 from 652138 to 1113505 tid 1 2024-09-21T18:07:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:135: writing smithi16030472-135 from 1574940 to 2252138 tid 2 2024-09-21T18:07:19.445 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: finishing write tid 1 to smithi16030472-112 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: finishing write tid 2 to smithi16030472-112 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: finishing write tid 3 to smithi16030472-112 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: finishing write tid 4 to smithi16030472-112 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: finishing write tid 5 to smithi16030472-112 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: oid 112 updating version 0 to 53 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: oid 112 updating version 53 to 55 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: oid 112 version 55 is already newer than 52 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: oid 112 version 55 is already newer than 54 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 112 v 55 (ObjNum 111 snap 0 seq_num 111) dirty exists 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: left oid 112 (ObjNum 111 snap 0 seq_num 111) 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:112: done (15 left) 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:136: write initial oid 136 2024-09-21T18:07:19.446 INFO:tasks.rados.rados.0.smithi160.stdout:136: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-21T18:07:19.456 INFO:tasks.rados.rados.0.smithi160.stdout:136: writing smithi16030472-136 from 668945 to 1205561 tid 1 2024-09-21T18:07:19.457 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:19.453+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:19.463 INFO:tasks.rados.rados.0.smithi160.stdout:136: writing smithi16030472-136 from 1633422 to 2124785 tid 2 2024-09-21T18:07:19.463 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.463 INFO:tasks.rados.rados.0.smithi160.stdout:126: finishing write tid 4 to smithi16030472-126 2024-09-21T18:07:19.463 INFO:tasks.rados.rados.0.smithi160.stdout:126: oid 126 updating version 0 to 93 2024-09-21T18:07:19.463 INFO:tasks.rados.rados.0.smithi160.stdout:126: oid 126 version 93 is already newer than 91 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:126: oid 126 version 93 is already newer than 92 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 126 v 93 (ObjNum 125 snap 0 seq_num 125) dirty exists 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:126: left oid 126 (ObjNum 125 snap 0 seq_num 125) 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:126: done (15 left) 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:137: write initial oid 137 2024-09-21T18:07:19.464 INFO:tasks.rados.rados.0.smithi160.stdout:137: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-21T18:07:19.473 INFO:tasks.rados.rados.0.smithi160.stdout:137: writing smithi16030472-137 from 685752 to 1297617 tid 1 2024-09-21T18:07:19.481 INFO:tasks.rados.rados.0.smithi160.stdout:137: writing smithi16030472-137 from 1975551 to 2285752 tid 2 2024-09-21T18:07:19.481 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.482 INFO:tasks.rados.rados.0.smithi160.stdout:127: finishing write tid 1 to smithi16030472-127 2024-09-21T18:07:19.482 INFO:tasks.rados.rados.0.smithi160.stdout:127: finishing write tid 2 to smithi16030472-127 2024-09-21T18:07:19.482 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 1 to smithi16030472-113 2024-09-21T18:07:19.482 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 2 to smithi16030472-113 2024-09-21T18:07:19.482 INFO:tasks.rados.rados.0.smithi160.stdout:123: finishing write tid 1 to smithi16030472-123 2024-09-21T18:07:19.539 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.539 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:19.539 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:19.539 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:19.553 INFO:tasks.rados.rados.0.smithi160.stdout:127: finishing write tid 3 to smithi16030472-127 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: finishing write tid 4 to smithi16030472-127 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: oid 127 updating version 0 to 94 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: oid 127 updating version 94 to 95 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: oid 127 updating version 95 to 96 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 127 v 96 (ObjNum 126 snap 0 seq_num 126) dirty exists 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: left oid 127 (ObjNum 126 snap 0 seq_num 126) 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:127: done (15 left) 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:138: write initial oid 138 2024-09-21T18:07:19.554 INFO:tasks.rados.rados.0.smithi160.stdout:138: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-21T18:07:19.558 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:19.557+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:19.561 INFO:tasks.rados.rados.0.smithi160.stdout:138: writing smithi16030472-138 from 702559 to 1106026 tid 1 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:138: writing smithi16030472-138 from 1750386 to 2245418 tid 2 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:128: finishing write tid 3 to smithi16030472-128 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:128: finishing write tid 4 to smithi16030472-128 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:128: oid 128 updating version 0 to 56 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:128: oid 128 version 56 is already newer than 55 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:128: oid 128 updating version 56 to 57 2024-09-21T18:07:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 128 v 57 (ObjNum 127 snap 0 seq_num 127) dirty exists 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:128: left oid 128 (ObjNum 127 snap 0 seq_num 127) 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: finishing write tid 1 to smithi16030472-129 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: finishing write tid 2 to smithi16030472-129 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: finishing write tid 3 to smithi16030472-129 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: finishing write tid 4 to smithi16030472-129 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: oid 129 updating version 0 to 58 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: oid 129 updating version 58 to 60 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: oid 129 version 60 is already newer than 59 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 60 (ObjNum 128 snap 0 seq_num 128) dirty exists 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: left oid 129 (ObjNum 128 snap 0 seq_num 128) 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:128: done (15 left) 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:129: done (14 left) 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:139: write initial oid 139 2024-09-21T18:07:19.572 INFO:tasks.rados.rados.0.smithi160.stdout:139: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-21T18:07:19.580 INFO:tasks.rados.rados.0.smithi160.stdout:139: writing smithi16030472-139 from 719366 to 1198082 tid 1 2024-09-21T18:07:19.589 INFO:tasks.rados.rados.0.smithi160.stdout:139: writing smithi16030472-139 from 1808868 to 2319366 tid 2 2024-09-21T18:07:19.589 INFO:tasks.rados.rados.0.smithi160.stdout:140: write initial oid 140 2024-09-21T18:07:19.589 INFO:tasks.rados.rados.0.smithi160.stdout:140: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-21T18:07:19.598 INFO:tasks.rados.rados.0.smithi160.stdout:140: writing smithi16030472-140 from 736173 to 1290138 tid 1 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:140: writing smithi16030472-140 from 1867350 to 2336173 tid 2 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: finishing write tid 1 to smithi16030472-130 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: finishing write tid 2 to smithi16030472-130 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: finishing write tid 3 to smithi16030472-130 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: finishing write tid 4 to smithi16030472-130 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: oid 130 updating version 0 to 61 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: oid 130 updating version 61 to 62 2024-09-21T18:07:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:130: oid 130 updating version 62 to 63 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 130 v 63 (ObjNum 129 snap 0 seq_num 129) dirty exists 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:130: left oid 130 (ObjNum 129 snap 0 seq_num 129) 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:134: finishing write tid 1 to smithi16030472-134 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:134: finishing write tid 2 to smithi16030472-134 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:134: finishing write tid 3 to smithi16030472-134 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:130: done (15 left) 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:141: write initial oid 141 2024-09-21T18:07:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:141: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-21T18:07:19.615 INFO:tasks.rados.rados.0.smithi160.stdout:141: writing smithi16030472-141 from 752980 to 1382194 tid 1 2024-09-21T18:07:19.621 INFO:tasks.rados.rados.0.smithi160.stdout:141: writing smithi16030472-141 from 1809479 to 2352980 tid 2 2024-09-21T18:07:19.622 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.622 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 3 to smithi16030472-113 2024-09-21T18:07:19.622 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 4 to smithi16030472-113 2024-09-21T18:07:19.658 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:19.657+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:19.681 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:19.681 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:19.681 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.696 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:19.724 INFO:tasks.rados.rados.0.smithi160.stdout:106: finishing write tid 5 to smithi16030472-106 2024-09-21T18:07:19.724 INFO:tasks.rados.rados.0.smithi160.stdout:106: oid 106 updating version 0 to 39 2024-09-21T18:07:19.724 INFO:tasks.rados.rados.0.smithi160.stdout:106: oid 106 updating version 39 to 41 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:106: oid 106 version 41 is already newer than 38 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:106: oid 106 version 41 is already newer than 40 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 106 v 41 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:106: left oid 106 (ObjNum 105 snap 0 seq_num 105) 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:106: done (15 left) 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:142: write initial oid 142 2024-09-21T18:07:19.725 INFO:tasks.rados.rados.0.smithi160.stdout:142: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-21T18:07:19.734 INFO:tasks.rados.rados.0.smithi160.stdout:142: writing smithi16030472-142 from 769787 to 1474250 tid 1 2024-09-21T18:07:19.740 INFO:tasks.rados.rados.0.smithi160.stdout:142: writing smithi16030472-142 from 2267961 to 2369787 tid 2 2024-09-21T18:07:19.740 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.755 INFO:tasks.rados.rados.0.smithi160.stdout:123: finishing write tid 2 to smithi16030472-123 2024-09-21T18:07:19.759 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:19.757+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: finishing write tid 4 to smithi16030472-134 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: oid 134 updating version 0 to 66 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: oid 134 version 66 is already newer than 65 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: oid 134 version 66 is already newer than 64 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 134 v 66 (ObjNum 133 snap 0 seq_num 133) dirty exists 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: left oid 134 (ObjNum 133 snap 0 seq_num 133) 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:134: done (15 left) 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:143: write initial oid 143 2024-09-21T18:07:19.774 INFO:tasks.rados.rados.0.smithi160.stdout:143: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-21T18:07:19.786 INFO:tasks.rados.rados.0.smithi160.stdout:143: writing smithi16030472-143 from 786594 to 1566306 tid 1 2024-09-21T18:07:19.792 INFO:tasks.rados.rados.0.smithi160.stdout:143: writing smithi16030472-143 from 2326443 to 2386594 tid 2 2024-09-21T18:07:19.792 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.824 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:19.824 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:19.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:22 4.2 e43/33 4:55c9f125:::smithi16030472-23:head v 43'13, mlcod=42'11)", 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.714492+0000", 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.1083330440000001, 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102691984, 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.714492+0000", 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.714492+0000", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.714496+0000", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8260000000000003e-06 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.716443+0000", 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019468899999999999 2024-09-21T18:07:19.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.716473+0000", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0273e-05 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.716489+0000", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5705999999999998e-05 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.718421+0000", 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001932783 2024-09-21T18:07:19.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.718486+0000", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4348000000000001e-05 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.817113+0000", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.098627413999999997 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.817184+0000", 2024-09-21T18:07:19.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0743999999999999e-05 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:29 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'19, mlcod=43'17)", 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.9771071820000001, 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20988762999999999, 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845721+0000", 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0570000000000001e-06 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847734+0000", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020132850000000001 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847771+0000", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6415000000000003e-05 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847794+0000", 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3730000000000001e-05 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948080+0000", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100285443 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948115+0000", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5098999999999997e-05 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055500+0000", 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107384654 2024-09-21T18:07:19.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055605+0000", 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010594699999999999 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:106 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'20, mlcod=43'17)", 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.9749727510000001, 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20783109999999999, 2024-09-21T18:07:19.833 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847855+0000", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4569999999999999e-06 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849293+0000", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014381109999999999 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849325+0000", 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2361999999999999e-05 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949454+0000", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128488 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950524+0000", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010704860000000001 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950583+0000", 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8643000000000001e-05 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055613+0000", 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10503064500000001 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055683+0000", 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9907999999999995e-05 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:107 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'21, mlcod=43'17)", 2024-09-21T18:07:19.837 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.949495+0000", 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.8733302780000001, 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106729466, 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949495+0000", 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949495+0000", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949497+0000", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3939999999999999e-06 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949510+0000", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2418999999999999e-05 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949530+0000", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0089999999999999e-05 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949540+0000", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0586e-05 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055006+0000", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105465636 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055068+0000", 2024-09-21T18:07:19.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1758000000000001e-05 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.056145+0000", 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001077375 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.056224+0000", 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9208000000000005e-05 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:25 4.2 e43/33 4:5d6444ba:::smithi16030472-32:head v 43'16, mlcod=42'11)", 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.949620+0000", 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.8732046579999997, 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102849027, 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949620+0000", 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949620+0000", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949622+0000", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9999999999999999e-06 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.951642+0000", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020193640000000001 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.951658+0000", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6473e-05 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.951670+0000", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1387e-05 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.952471+0000", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00080121399999999996 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.952503+0000", 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2558999999999998e-05 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.052376+0000", 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099872354999999996 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.052469+0000", 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3675e-05 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:150 4.2 e44/33 4:4dd833f5:::smithi16030472-38:head v 44'20, mlcod=43'17)", 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.710178+0000", 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.1126473050000003, 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104256265, 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.710178+0000", 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.710178+0000", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.710181+0000", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.954e-06 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.711513+0000", 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0013322410000000001 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.711535+0000", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2501999999999999e-05 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811635+0000", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100099303 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811745+0000", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010978 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811779+0000", 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4521999999999999e-05 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.814367+0000", 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025875249999999998 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.814434+0000", 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7438e-05 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:151 4.2 e44/33 4:4dd833f5:::smithi16030472-38:head v 44'21, mlcod=43'17)", 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.811712+0000", 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.0111126639999997, 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102730712, 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811712+0000", 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811712+0000", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811715+0000", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3499999999999999e-06 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811728+0000", 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2775999999999999e-05 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.811746+0000", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8085999999999999e-05 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.911836+0000", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100089916 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912011+0000", 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000175479 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912051+0000", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0309000000000003e-05 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.914377+0000", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0023259840000000001 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.914443+0000", 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5811999999999999e-05 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.853 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:154 4.2 e44/33 4:6abed166:::smithi16030472-39:head v 44'23, mlcod=43'17)", 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.9100241650000003, 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20421822000000001, 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:19.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912802+0000", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3030000000000001e-06 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913279+0000", 2024-09-21T18:07:19.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000476928 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913294+0000", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5209e-05 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013383+0000", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10008903600000001 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013498+0000", 2024-09-21T18:07:19.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011462 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013532+0000", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4069000000000002e-05 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.116937+0000", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103404914 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117019+0000", 2024-09-21T18:07:19.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2140999999999998e-05 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:155 4.2 e44/33 4:6abed166:::smithi16030472-39:head v 44'24, mlcod=43'17)", 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.013419+0000", 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.8094062639999997, 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104240712, 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013419+0000", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013419+0000", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013421+0000", 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.21e-06 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013433+0000", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2123000000000001e-05 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013450+0000", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.645e-05 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013463+0000", 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3716e-05 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.116462+0000", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102999118 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.116522+0000", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9366e-05 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117590+0000", 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010686770000000001 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117659+0000", 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9052000000000002e-05 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:159 4.0 e44/33 4:1192aa23:::smithi16030472-40:head v 44'27, mlcod=43'23)", 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.017445+0000", 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.8053804470000001, 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101584102, 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.862 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.017445+0000", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.017445+0000", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.017447+0000", 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1399999999999998e-06 2024-09-21T18:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.017458+0000", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1406000000000001e-05 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.017556+0000", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7464000000000001e-05 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117637+0000", 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10008170199999999 2024-09-21T18:07:19.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117785+0000", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014805299999999999 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117844+0000", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9064999999999998e-05 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.118958+0000", 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0011130809999999999 2024-09-21T18:07:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.119029+0000", 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1191000000000003e-05 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:44 4.0 e44/33 4:01b9dd46:::smithi16030472-42:head v 44'28, mlcod=43'23)", 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.7051549020000003, 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.117588128, 2024-09-21T18:07:19.866 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117673+0000", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4899999999999999e-06 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225500+0000", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107826882 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225515+0000", 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5379999999999998e-05 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225524+0000", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4029999999999996e-06 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225651+0000", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012647299999999999 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225722+0000", 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1401000000000003e-05 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.235172+0000", 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0094498140000000008 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.235258+0000", 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6285000000000005e-05 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:186 4.2 e44/33 4:638b5b54:::smithi16030472-47:head v 44'26, mlcod=44'21)", 2024-09-21T18:07:19.870 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.120958+0000", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.7018667040000004, 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10793040399999999, 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.120958+0000", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.120958+0000", 2024-09-21T18:07:19.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.120960+0000", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2300000000000001e-06 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.121656+0000", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000696888 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.121674+0000", 2024-09-21T18:07:19.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7139999999999999e-05 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.221813+0000", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139174 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.221934+0000", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012150399999999999 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.221975+0000", 2024-09-21T18:07:19.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0748999999999999e-05 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.228718+0000", 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0067428330000000002 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.228889+0000", 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000170886 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:179 4.0 e44/33 4:16d9fc1d:::smithi16030472-45:head v 44'33, mlcod=43'23)", 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.234004+0000", 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.5888208769999999, 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10446142899999999, 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234004+0000", 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234004+0000", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234005+0000", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2359999999999999e-06 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234012+0000", 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6270000000000002e-06 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234021+0000", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5560000000000008e-06 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234027+0000", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8730000000000001e-06 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.337348+0000", 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10332088 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.337384+0000", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6152999999999998e-05 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338394+0000", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001009839 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338466+0000", 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1265000000000003e-05 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.878 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:52 4.0 e44/33 4:15fed93e:::smithi16030472-46:head v 44'34, mlcod=43'23)", 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.234037+0000", 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.5887883049999996, 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108950599, 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234037+0000", 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234037+0000", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.234037+0000", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5000000000000002e-07 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.238589+0000", 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0045516619999999997 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.238609+0000", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9511999999999999e-05 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338697+0000", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10008866700000001 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338853+0000", 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015604900000000001 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338885+0000", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1992999999999997e-05 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.342924+0000", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040390249999999999 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.342987+0000", 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3040999999999995e-05 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.882 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:54 4.0 e44/33 4:1f69a690:::smithi16030472-48:head v 44'36, mlcod=43'23)", 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.483961656, 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.118230935, 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338865+0000", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.15e-06 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.347997+0000", 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091320800000000008 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348022+0000", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5681999999999999e-05 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348036+0000", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3570000000000001e-05 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348151+0000", 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011503 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348186+0000", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5046000000000003e-05 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457034+0000", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10884822199999999 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457094+0000", 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0155000000000003e-05 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.886 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:191 4.0 e44/33 4:1f69a690:::smithi16030472-48:head v 44'37, mlcod=43'23)", 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.348069+0000", 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.4747564679999998, 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10907241600000001, 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348069+0000", 2024-09-21T18:07:19.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348069+0000", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348071+0000", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0260000000000002e-06 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348078+0000", 2024-09-21T18:07:19.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3540000000000002e-06 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348090+0000", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2139e-05 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348098+0000", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8359999999999992e-06 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.352706+0000", 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046081669999999998 2024-09-21T18:07:19.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.352742+0000", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6099000000000001e-05 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457101+0000", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104358889 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457141+0000", 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9906e-05 2024-09-21T18:07:19.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:59 4.0 e44/33 4:11edfedf:::smithi16030472-50:head v 44'38, mlcod=43'23)", 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.348112+0000", 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.4747132289999998, 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10908454200000001, 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348112+0000", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348112+0000", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348112+0000", 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7599999999999997e-07 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349318+0000", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012052409999999999 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349332+0000", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.415e-05 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349347+0000", 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4636e-05 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.353089+0000", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00374296 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.353134+0000", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4585000000000001e-05 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457147+0000", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10401279500000001 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457196+0000", 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9499000000000002e-05 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:198 4.0 e44/33 4:11edfedf:::smithi16030472-50:head v 44'39, mlcod=43'23)", 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.349363+0000", 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.47346194, 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107933687, 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349363+0000", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349363+0000", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.349364+0000", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350607+0000", 2024-09-21T18:07:19.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001243154 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350626+0000", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8816000000000001e-05 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350636+0000", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5359999999999998e-06 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.356574+0000", 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059377120000000004 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.356610+0000", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6795999999999998e-05 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457200+0000", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10058940199999999 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457297+0000", 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7060999999999998e-05 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:199 4.0 e44/33 4:11edfedf:::smithi16030472-50:head v 44'40, mlcod=43'23)", 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.350666+0000", 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.4721594209999997, 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107435236, 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350666+0000", 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350666+0000", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350667+0000", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350673+0000", 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.896e-06 2024-09-21T18:07:19.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350680+0000", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8669999999999994e-06 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.350687+0000", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2829999999999998e-06 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.456696+0000", 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106009302 2024-09-21T18:07:19.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.456731+0000", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4542000000000002e-05 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.458021+0000", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012909620000000001 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.458101+0000", 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9400999999999994e-05 2024-09-21T18:07:19.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:151 4.2 e46/33 4:6a817314:::smithi16030472-101:head v 46'77, mlcod=46'75)", 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.398226+0000", 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.424599049, 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10862298500000001, 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.398226+0000", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.398226+0000", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.398230+0000", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4999999999999999e-06 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:19.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.402866+0000", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046364819999999999 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.402896+0000", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9519e-05 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.402914+0000", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8125999999999999e-05 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.503173+0000", 2024-09-21T18:07:19.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100258981 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.503198+0000", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5585999999999999e-05 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.506789+0000", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035911979999999999 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.506849+0000", 2024-09-21T18:07:19.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9592999999999997e-05 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:19.907 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:19.909 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:19.910 INFO:tasks.rados.rados.0.smithi160.stdout:132: finishing write tid 1 to smithi16030472-132 2024-09-21T18:07:19.910 INFO:tasks.rados.rados.0.smithi160.stdout:132: finishing write tid 2 to smithi16030472-132 2024-09-21T18:07:19.910 INFO:tasks.rados.rados.0.smithi160.stdout:133: finishing write tid 1 to smithi16030472-133 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: finishing write tid 3 to smithi16030472-123 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: finishing write tid 4 to smithi16030472-123 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: oid 123 updating version 0 to 58 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: oid 123 updating version 58 to 59 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: oid 123 updating version 59 to 60 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 60 (ObjNum 122 snap 0 seq_num 122) dirty exists 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:123: left oid 123 (ObjNum 122 snap 0 seq_num 122) 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:124: finishing write tid 1 to smithi16030472-124 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:124: finishing write tid 2 to smithi16030472-124 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:124: finishing write tid 3 to smithi16030472-124 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:131: finishing write tid 1 to smithi16030472-131 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:131: finishing write tid 2 to smithi16030472-131 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:131: finishing write tid 3 to smithi16030472-131 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:131: finishing write tid 4 to smithi16030472-131 2024-09-21T18:07:19.911 INFO:tasks.rados.rados.0.smithi160.stdout:131: oid 131 updating version 0 to 64 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:131: oid 131 updating version 64 to 66 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:131: oid 131 version 66 is already newer than 65 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 131 v 66 (ObjNum 130 snap 0 seq_num 130) dirty exists 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:131: left oid 131 (ObjNum 130 snap 0 seq_num 130) 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:123: done (15 left) 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:131: done (14 left) 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:144: write initial oid 144 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:144: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:144: writing smithi16030472-144 from 403401 to 858362 tid 1 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:144: writing smithi16030472-144 from 1584925 to 2090964 tid 2 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:145: write initial oid 145 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:145: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:145: writing smithi16030472-145 from 420208 to 950418 tid 1 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout:145: writing smithi16030472-145 from 1527054 to 2293104 tid 2 2024-09-21T18:07:19.912 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: finishing write tid 3 to smithi16030472-108 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: finishing write tid 4 to smithi16030472-108 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: finishing write tid 5 to smithi16030472-108 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: oid 108 updating version 0 to 47 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: oid 108 updating version 47 to 86 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: oid 108 version 86 is already newer than 85 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: oid 108 version 86 is already newer than 46 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 108 v 86 (ObjNum 107 snap 0 seq_num 107) dirty exists 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: left oid 108 (ObjNum 107 snap 0 seq_num 107) 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:108: done (15 left) 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:146: write initial oid 146 2024-09-21T18:07:19.913 INFO:tasks.rados.rados.0.smithi160.stdout:146: seq_num 145 ranges {437015=721812,1701889=509708} 2024-09-21T18:07:19.914 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:19.905+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:19.918 INFO:tasks.rados.rados.0.smithi160.stdout:146: writing smithi16030472-146 from 437015 to 1158827 tid 1 2024-09-21T18:07:19.928 INFO:tasks.rados.rados.0.smithi160.stdout:146: writing smithi16030472-146 from 1701889 to 2211597 tid 2 2024-09-21T18:07:19.928 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.963 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 5 to smithi16030472-113 2024-09-21T18:07:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:113: finishing write tid 6 to smithi16030472-113 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: oid 113 updating version 0 to 81 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: oid 113 version 81 is already newer than 57 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: oid 113 version 81 is already newer than 56 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: oid 113 version 81 is already newer than 80 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: oid 113 updating version 81 to 87 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 113 v 87 (ObjNum 112 snap 0 seq_num 112) dirty exists 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: left oid 113 (ObjNum 112 snap 0 seq_num 112) 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:113: done (15 left) 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:147: write initial oid 147 2024-09-21T18:07:19.965 INFO:tasks.rados.rados.0.smithi160.stdout:147: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-21T18:07:19.976 INFO:tasks.rados.rados.0.smithi160.stdout:147: writing smithi16030472-147 from 453822 to 1250883 tid 1 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:147: writing smithi16030472-147 from 1760371 to 2413737 tid 2 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: finishing write tid 4 to smithi16030472-124 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: oid 124 updating version 0 to 62 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: oid 124 updating version 62 to 63 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: oid 124 version 63 is already newer than 61 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 124 v 63 (ObjNum 123 snap 0 seq_num 123) dirty exists 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: left oid 124 (ObjNum 123 snap 0 seq_num 123) 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:124: done (15 left) 2024-09-21T18:07:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:148: write initial oid 148 2024-09-21T18:07:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:148: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-21T18:07:19.996 INFO:tasks.rados.rados.0.smithi160.stdout:148: writing smithi16030472-148 from 470629 to 942939 tid 1 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:148: writing smithi16030472-148 from 1418853 to 1932230 tid 2 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:132: finishing write tid 3 to smithi16030472-132 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:137: finishing write tid 1 to smithi16030472-137 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:137: finishing write tid 2 to smithi16030472-137 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:141: finishing write tid 1 to smithi16030472-141 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:141: finishing write tid 2 to smithi16030472-141 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:136: finishing write tid 1 to smithi16030472-136 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:136: finishing write tid 2 to smithi16030472-136 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:138: finishing write tid 1 to smithi16030472-138 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:138: finishing write tid 2 to smithi16030472-138 2024-09-21T18:07:20.006 INFO:tasks.rados.rados.0.smithi160.stdout:138: finishing write tid 3 to smithi16030472-138 2024-09-21T18:07:20.009 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:20.005+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:20.042 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:20.043 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:20.043 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:20.043 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:20.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:20.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: finishing write tid 1 to smithi16030472-139 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: finishing write tid 2 to smithi16030472-139 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: finishing write tid 3 to smithi16030472-139 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: finishing write tid 4 to smithi16030472-139 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: oid 139 updating version 0 to 103 2024-09-21T18:07:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:139: oid 139 updating version 103 to 104 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:139: oid 139 version 104 is already newer than 102 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 104 (ObjNum 138 snap 0 seq_num 138) dirty exists 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:139: left oid 139 (ObjNum 138 snap 0 seq_num 138) 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:139: done (15 left) 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:149: write initial oid 149 2024-09-21T18:07:20.108 INFO:tasks.rados.rados.0.smithi160.stdout:149: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-21T18:07:20.109 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:20.105+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:20.116 INFO:tasks.rados.rados.0.smithi160.stdout:149: writing smithi16030472-149 from 487436 to 1034995 tid 1 2024-09-21T18:07:20.130 INFO:tasks.rados.rados.0.smithi160.stdout:149: writing smithi16030472-149 from 1760982 to 2418017 tid 2 2024-09-21T18:07:20.130 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.130 INFO:tasks.rados.rados.0.smithi160.stdout:132: finishing write tid 4 to smithi16030472-132 2024-09-21T18:07:20.130 INFO:tasks.rados.rados.0.smithi160.stdout:132: oid 132 updating version 0 to 99 2024-09-21T18:07:20.130 INFO:tasks.rados.rados.0.smithi160.stdout:132: oid 132 version 99 is already newer than 98 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:132: oid 132 version 99 is already newer than 97 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 99 (ObjNum 131 snap 0 seq_num 131) dirty exists 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:132: left oid 132 (ObjNum 131 snap 0 seq_num 131) 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: finishing write tid 3 to smithi16030472-137 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: finishing write tid 4 to smithi16030472-137 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: oid 137 updating version 0 to 101 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: oid 137 version 101 is already newer than 100 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: oid 137 updating version 101 to 102 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 137 v 102 (ObjNum 136 snap 0 seq_num 136) dirty exists 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:137: left oid 137 (ObjNum 136 snap 0 seq_num 136) 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:141: finishing write tid 3 to smithi16030472-141 2024-09-21T18:07:20.131 INFO:tasks.rados.rados.0.smithi160.stdout:132: done (15 left) 2024-09-21T18:07:20.132 INFO:tasks.rados.rados.0.smithi160.stdout:137: done (14 left) 2024-09-21T18:07:20.132 INFO:tasks.rados.rados.0.smithi160.stdout:150: write initial oid 150 2024-09-21T18:07:20.132 INFO:tasks.rados.rados.0.smithi160.stdout:150: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-21T18:07:20.142 INFO:tasks.rados.rados.0.smithi160.stdout:150: writing smithi16030472-150 from 504243 to 1127051 tid 1 2024-09-21T18:07:20.155 INFO:tasks.rados.rados.0.smithi160.stdout:150: writing smithi16030472-150 from 1819464 to 2336510 tid 2 2024-09-21T18:07:20.156 INFO:tasks.rados.rados.0.smithi160.stdout:151: write initial oid 151 2024-09-21T18:07:20.156 INFO:tasks.rados.rados.0.smithi160.stdout:151: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-21T18:07:20.167 INFO:tasks.rados.rados.0.smithi160.stdout:151: writing smithi16030472-151 from 521050 to 1219107 tid 1 2024-09-21T18:07:20.178 INFO:tasks.rados.rados.0.smithi160.stdout:151: writing smithi16030472-151 from 1877946 to 2521050 tid 2 2024-09-21T18:07:20.178 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.179 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:20.179 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:20.179 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:20.179 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:20.179 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:20.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:20.210 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:20.209+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:20.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:07:20.255 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:145: finishing write tid 1 to smithi16030472-145 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:145: finishing write tid 2 to smithi16030472-145 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:145: finishing write tid 3 to smithi16030472-145 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:133: finishing write tid 2 to smithi16030472-133 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:141: finishing write tid 4 to smithi16030472-141 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:141: oid 141 updating version 0 to 68 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:141: oid 141 version 68 is already newer than 67 2024-09-21T18:07:20.259 INFO:tasks.rados.rados.0.smithi160.stdout:141: oid 141 updating version 68 to 69 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 141 v 69 (ObjNum 140 snap 0 seq_num 140) dirty exists 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:141: left oid 141 (ObjNum 140 snap 0 seq_num 140) 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:133: finishing write tid 3 to smithi16030472-133 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: finishing write tid 4 to smithi16030472-145 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: oid 145 updating version 0 to 106 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: oid 145 updating version 106 to 107 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: oid 145 version 107 is already newer than 105 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 145 v 107 (ObjNum 144 snap 0 seq_num 144) dirty exists 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: left oid 145 (ObjNum 144 snap 0 seq_num 144) 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:142: finishing write tid 1 to smithi16030472-142 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:141: done (15 left) 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:145: done (14 left) 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:152: write initial oid 152 2024-09-21T18:07:20.260 INFO:tasks.rados.rados.0.smithi160.stdout:152: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-21T18:07:20.268 INFO:tasks.rados.rados.0.smithi160.stdout:152: writing smithi16030472-152 from 537857 to 1311163 tid 1 2024-09-21T18:07:20.277 INFO:tasks.rados.rados.0.smithi160.stdout:152: writing smithi16030472-152 from 1936428 to 2457143 tid 2 2024-09-21T18:07:20.277 INFO:tasks.rados.rados.0.smithi160.stdout:153: write initial oid 153 2024-09-21T18:07:20.277 INFO:tasks.rados.rados.0.smithi160.stdout:153: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-21T18:07:20.284 INFO:tasks.rados.rados.0.smithi160.stdout:153: writing smithi16030472-153 from 554664 to 1003219 tid 1 2024-09-21T18:07:20.292 INFO:tasks.rados.rados.0.smithi160.stdout:153: writing smithi16030472-153 from 1478557 to 2142930 tid 2 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:148: finishing write tid 1 to smithi16030472-148 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:142: finishing write tid 2 to smithi16030472-142 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: finishing write tid 4 to smithi16030472-133 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: oid 133 updating version 0 to 81 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: oid 133 version 81 is already newer than 80 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: oid 133 updating version 81 to 82 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 133 v 82 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: left oid 133 (ObjNum 132 snap 0 seq_num 132) 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:135: finishing write tid 1 to smithi16030472-135 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:135: finishing write tid 2 to smithi16030472-135 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:133: done (15 left) 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:154: write initial oid 154 2024-09-21T18:07:20.293 INFO:tasks.rados.rados.0.smithi160.stdout:154: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-21T18:07:20.301 INFO:tasks.rados.rados.0.smithi160.stdout:154: writing smithi16030472-154 from 571471 to 1211628 tid 1 2024-09-21T18:07:20.308 INFO:tasks.rados.rados.0.smithi160.stdout:154: writing smithi16030472-154 from 1653392 to 2177776 tid 2 2024-09-21T18:07:20.308 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: finishing write tid 3 to smithi16030472-135 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: finishing write tid 2 to smithi16030472-148 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: finishing write tid 3 to smithi16030472-148 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: finishing write tid 4 to smithi16030472-148 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: oid 148 updating version 0 to 108 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: oid 148 updating version 108 to 110 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: oid 148 version 110 is already newer than 109 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 148 v 110 (ObjNum 147 snap 0 seq_num 147) dirty exists 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:148: left oid 148 (ObjNum 147 snap 0 seq_num 147) 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: finishing write tid 4 to smithi16030472-135 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: oid 135 updating version 0 to 85 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: oid 135 version 85 is already newer than 84 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: oid 135 version 85 is already newer than 83 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 135 v 85 (ObjNum 134 snap 0 seq_num 134) dirty exists 2024-09-21T18:07:20.309 INFO:tasks.rados.rados.0.smithi160.stdout:135: left oid 135 (ObjNum 134 snap 0 seq_num 134) 2024-09-21T18:07:20.310 INFO:tasks.rados.rados.0.smithi160.stdout:135: done (15 left) 2024-09-21T18:07:20.310 INFO:tasks.rados.rados.0.smithi160.stdout:148: done (14 left) 2024-09-21T18:07:20.310 INFO:tasks.rados.rados.0.smithi160.stdout:155: write initial oid 155 2024-09-21T18:07:20.310 INFO:tasks.rados.rados.0.smithi160.stdout:155: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-21T18:07:20.310 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:20.311 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:20.311 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:20.309+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:20.318 INFO:tasks.rados.rados.0.smithi160.stdout:155: writing smithi16030472-155 from 588278 to 1303684 tid 1 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:98 3.1 e43/31 3:af0b33ab:::smithi16030472-25:head v 43'6, mlcod=42'3)", 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.745855+0000", 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.5717223950000001, 2024-09-21T18:07:20.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11062973600000001, 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.745855+0000", 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.745855+0000", 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.745867+0000", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2058999999999999e-05 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.746858+0000", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00099046300000000002 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.746887+0000", 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9748999999999999e-05 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847000+0000", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100112971 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847164+0000", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016371199999999999 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847220+0000", 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5761999999999999e-05 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.856410+0000", 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091901120000000003 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.856485+0000", 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4907999999999995e-05 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:134 3.1 e43/31 3:b53d2f27:::smithi16030472-34:head v 43'15, mlcod=43'12)", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.166036+0000", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.1515418090000002, 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103486446, 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.166036+0000", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.166036+0000", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.166038+0000", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.43e-06 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.166436+0000", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00039816299999999999 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.166458+0000", 2024-09-21T18:07:20.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2048999999999998e-05 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.266641+0000", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100182913 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.266821+0000", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017996599999999999 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.266861+0000", 2024-09-21T18:07:20.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0244999999999998e-05 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.269430+0000", 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025682449999999998 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.269522+0000", 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2435000000000005e-05 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:201 3.1 e44/31 3:928f936c:::smithi16030472-51:head v 44'18, mlcod=43'16)", 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.800660+0000", 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.5169175480000003, 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104245448, 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.800660+0000", 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.800660+0000", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.800663+0000", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.596e-06 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.801057+0000", 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00039345 2024-09-21T18:07:20.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.801087+0000", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9745999999999999e-05 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.801103+0000", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6016e-05 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.901372+0000", 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100269281 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.901407+0000", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5318999999999998e-05 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.904818+0000", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0034105199999999998 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.904905+0000", 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7520000000000002e-05 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.331 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:210 3.1 e44/31 3:980dfd36:::smithi16030472-53:head v 44'20, mlcod=44'18)", 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.136259+0000", 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.1813180079999999, 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107317812, 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.136259+0000", 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.136259+0000", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.136264+0000", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.3760000000000001e-06 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.136955+0000", 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00069149499999999998 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.136990+0000", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4638999999999998e-05 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.137012+0000", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2299000000000001e-05 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.237343+0000", 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100331089 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.237407+0000", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3972000000000003e-05 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.243405+0000", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0059980570000000002 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.243577+0000", 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017188500000000001 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.335 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:232 3.1 e45/31 3:b3326bec:::smithi16030472-57:head v 45'27, mlcod=45'24)", 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.680310+0000", 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.6372676149999998, 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104038491, 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680310+0000", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680310+0000", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680313+0000", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0699999999999998e-06 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680325+0000", 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2269999999999999e-05 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680351+0000", 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6248999999999999e-05 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.680366+0000", 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5109999999999999e-05 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.685371+0000", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0050044060000000003 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.685406+0000", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5002999999999997e-05 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.784194+0000", 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.098787651000000004 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.784348+0000", 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015473200000000001 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:234 3.1 e45/31 3:96516620:::smithi16030472-58:head v 45'28, mlcod=45'24)", 2024-09-21T18:07:20.340 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.681152+0000", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.6364256700000004, 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103242951, 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.681152+0000", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.681152+0000", 2024-09-21T18:07:20.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.681155+0000", 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4429999999999999e-06 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.682358+0000", 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012026280000000001 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.682384+0000", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5953e-05 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.682399+0000", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5515999999999998e-05 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.685748+0000", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033489269999999998 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.685775+0000", 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6846e-05 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.784354+0000", 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.098578679000000002 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.784395+0000", 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0958999999999999e-05 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:236 3.1 e45/31 3:96516620:::smithi16030472-58:head v 45'30, mlcod=45'24)", 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.6332270590000002, 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.202914602, 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684354+0000", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2959999999999999e-06 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684626+0000", 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027191299999999998 2024-09-21T18:07:20.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684653+0000", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7195999999999998e-05 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684667+0000", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3767e-05 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787798+0000", 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103131111 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787831+0000", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3229000000000003e-05 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887202+0000", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099371371 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887265+0000", 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2718999999999994e-05 2024-09-21T18:07:20.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:520 3.1 e47/31 3:967b012e:::smithi16030472-116:head v 47'78, mlcod=46'75)", 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.317666+0000", 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.999911867, 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105019637, 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.317666+0000", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.317666+0000", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.317669+0000", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1860000000000001e-06 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.318253+0000", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00058442599999999998 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.318280+0000", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7076e-05 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.318294+0000", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3803e-05 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.418438+0000", 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014350800000001 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.418474+0000", 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109000000000003e-05 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.422565+0000", 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0040913950000000003 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.422685+0000", 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120134 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:521 3.1 e47/31 3:967b012e:::smithi16030472-116:head v 47'79, mlcod=46'75)", 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.319638+0000", 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.9979391660000001, 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103092398, 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.353 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.319638+0000", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.319638+0000", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.319641+0000", 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8959999999999999e-06 2024-09-21T18:07:20.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.319653+0000", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.319684+0000", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0076000000000001e-05 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.419826+0000", 2024-09-21T18:07:20.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142521 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.420045+0000", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000219058 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.420081+0000", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109000000000003e-05 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.422696+0000", 2024-09-21T18:07:20.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002615062 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.422731+0000", 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4348999999999999e-05 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.479032948, 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:558 3.1 e47/31 3:b2cc57d2:::smithi16030472-125:head v 47'88, mlcod=47'83)", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.938711+0000", 2024-09-21T18:07:20.361 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.3788663130000001, 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10806102099999999, 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938711+0000", 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938711+0000", 2024-09-21T18:07:20.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938713+0000", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.708e-06 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939190+0000", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00047767799999999999 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939208+0000", 2024-09-21T18:07:20.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.791e-05 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939220+0000", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1609e-05 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939279+0000", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9038e-05 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939302+0000", 2024-09-21T18:07:20.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3215999999999999e-05 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046669+0000", 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107367168 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046772+0000", 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000102694 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.365 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:559 3.1 e47/31 3:b2cc57d2:::smithi16030472-125:head v 47'89, mlcod=47'83)", 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.939238+0000", 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.3783390149999999, 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10758599200000001, 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939238+0000", 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939238+0000", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939240+0000", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2869999999999999e-06 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939513+0000", 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027361600000000002 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939523+0000", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9599999999999995e-06 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939531+0000", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4170000000000003e-06 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.940975+0000", 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001444287 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.941034+0000", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9388999999999997e-05 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046779+0000", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105745064 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046824+0000", 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4972000000000002e-05 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.369 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:560 3.1 e47/31 3:b2cc57d2:::smithi16030472-125:head v 47'90, mlcod=47'83)", 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.939546+0000", 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.378031056, 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10731270599999999, 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939546+0000", 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939546+0000", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939547+0000", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7400000000000002e-07 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939553+0000", 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7459999999999997e-06 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939560+0000", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0330000000000003e-06 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939568+0000", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0569999999999993e-06 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042439+0000", 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10287138699999999 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042500+0000", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0195000000000002e-05 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046830+0000", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0043303179999999997 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046859+0000", 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9196e-05 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:562 3.1 e47/31 3:9332d87c:::smithi16030472-126:head v 47'91, mlcod=47'83)", 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.939576+0000", 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.3780012830000001, 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10731157199999999, 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939576+0000", 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939576+0000", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939577+0000", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.4700000000000002e-07 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939873+0000", 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00029597199999999998 2024-09-21T18:07:20.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939882+0000", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5370000000000003e-06 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939889+0000", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9600000000000003e-06 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043187+0000", 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10329787 2024-09-21T18:07:20.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043241+0000", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4395999999999999e-05 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046861+0000", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0036193140000000002 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.046888+0000", 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7076e-05 2024-09-21T18:07:20.377 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:563 3.1 e47/31 3:9332d87c:::smithi16030472-126:head v 47'92, mlcod=47'84)", 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.939903+0000", 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.3776745909999999, 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107385904, 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939903+0000", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939903+0000", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.939903+0000", 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9299999999999998e-07 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.940580+0000", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00067673699999999995 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.940593+0000", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3006e-05 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040711+0000", 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10011832399999999 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.044974+0000", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0042627389999999998 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.045015+0000", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1208999999999998e-05 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.047246+0000", 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00223105 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.047289+0000", 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2246000000000002e-05 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:564 3.1 e47/31 3:9332d87c:::smithi16030472-126:head v 47'93, mlcod=47'84)", 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:19.040740+0000", 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.2768369550000001, 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103561837, 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.382 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040740+0000", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040740+0000", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040743+0000", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0930000000000001e-06 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040757+0000", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4368e-05 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040767+0000", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0274e-05 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040774+0000", 2024-09-21T18:07:20.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.72e-06 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.141619+0000", 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100845213 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.141654+0000", 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4428000000000003e-05 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.144225+0000", 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025715099999999999 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.144302+0000", 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7231000000000006e-05 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:566 3.1 e47/31 3:b5eec191:::smithi16030472-127:head v 47'94, mlcod=47'84)", 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:19.040779+0000", 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.276797956, 2024-09-21T18:07:20.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103628096, 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040779+0000", 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040779+0000", 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.040780+0000", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.53e-07 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.041393+0000", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00061314599999999996 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.041408+0000", 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5556000000000001e-05 2024-09-21T18:07:20.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.141525+0000", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10011650900000001 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.141990+0000", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00046488900000000001 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.142010+0000", 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0551e-05 2024-09-21T18:07:20.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.144307+0000", 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0022967270000000001 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.144408+0000", 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010036500000000001 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:606 3.1 e47/31 3:b048eba1:::smithi16030472-137:head v 47'100, mlcod=47'95)", 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:19.773236+0000", 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.54434163800000002, 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105003865, 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.773236+0000", 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.773236+0000", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.773248+0000", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2353e-05 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774110+0000", 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00086208300000000003 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774144+0000", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3794000000000002e-05 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774171+0000", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6778e-05 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774259+0000", 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8728999999999997e-05 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774284+0000", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4272000000000001e-05 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878142+0000", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103858595 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878240+0000", 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7261000000000003e-05 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.394 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:607 3.1 e47/31 3:b048eba1:::smithi16030472-137:head v 47'101, mlcod=47'95)", 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:19.774212+0000", 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.54336500700000001, 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104074347, 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774212+0000", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774212+0000", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774213+0000", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.02e-07 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774459+0000", 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000245954 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774467+0000", 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8609999999999998e-06 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774472+0000", 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.9280000000000001e-06 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.775451+0000", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009789620000000001 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.775479+0000", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7900999999999999e-05 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878245+0000", 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102765943 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878287+0000", 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1896e-05 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:614 3.1 e47/31 3:adbb2c09:::smithi16030472-139:head v 47'102, mlcod=47'95)", 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:19.774485+0000", 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.543092405, 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104299819, 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:20.399 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774485+0000", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774485+0000", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774486+0000", 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0299999999999998e-07 2024-09-21T18:07:20.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774894+0000", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040846300000000002 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774905+0000", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1184e-05 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.774911+0000", 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5990000000000001e-06 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.876383+0000", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101471831 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.876420+0000", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7611999999999998e-05 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878739+0000", 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002318414 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:20.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.878785+0000", 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6013000000000001e-05 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:20.403 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:20.404 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:20.405 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:20.406 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:20.407 INFO:tasks.rados.rados.0.smithi160.stdout:155: writing smithi16030472-155 from 1711874 to 2379916 tid 2 2024-09-21T18:07:20.407 INFO:tasks.rados.rados.0.smithi160.stdout:156: write initial oid 156 2024-09-21T18:07:20.407 INFO:tasks.rados.rados.0.smithi160.stdout:156: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-21T18:07:20.407 INFO:tasks.rados.rados.0.smithi160.stdout:156: writing smithi16030472-156 from 605085 to 1395740 tid 1 2024-09-21T18:07:20.407 INFO:tasks.rados.rados.0.smithi160.stdout:156: writing smithi16030472-156 from 2170356 to 2605085 tid 2 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:138: finishing write tid 4 to smithi16030472-138 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:138: oid 138 updating version 0 to 70 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:138: oid 138 version 70 is already newer than 69 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:138: oid 138 updating version 70 to 71 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 138 v 71 (ObjNum 137 snap 0 seq_num 137) dirty exists 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:138: left oid 138 (ObjNum 137 snap 0 seq_num 137) 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: finishing write tid 1 to smithi16030472-140 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: finishing write tid 2 to smithi16030472-140 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: finishing write tid 3 to smithi16030472-140 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: finishing write tid 4 to smithi16030472-140 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: oid 140 updating version 0 to 73 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: oid 140 updating version 73 to 74 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: oid 140 version 74 is already newer than 72 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 140 v 74 (ObjNum 139 snap 0 seq_num 139) dirty exists 2024-09-21T18:07:20.408 INFO:tasks.rados.rados.0.smithi160.stdout:140: left oid 140 (ObjNum 139 snap 0 seq_num 139) 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:144: finishing write tid 1 to smithi16030472-144 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:138: done (15 left) 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:140: done (14 left) 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:157: write initial oid 157 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:157: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:157: writing smithi16030472-157 from 621892 to 1087796 tid 1 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:157: writing smithi16030472-157 from 1712485 to 2384196 tid 2 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:158: write initial oid 158 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:158: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:158: writing smithi16030472-158 from 638699 to 1179852 tid 1 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:158: writing smithi16030472-158 from 1770967 to 2302689 tid 2 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:144: finishing write tid 2 to smithi16030472-144 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:144: finishing write tid 3 to smithi16030472-144 2024-09-21T18:07:20.409 INFO:tasks.rados.rados.0.smithi160.stdout:144: finishing write tid 4 to smithi16030472-144 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:144: oid 144 updating version 0 to 76 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:144: oid 144 version 76 is already newer than 75 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:144: oid 144 updating version 76 to 77 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 144 v 77 (ObjNum 143 snap 0 seq_num 143) dirty exists 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:144: left oid 144 (ObjNum 143 snap 0 seq_num 143) 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:146: finishing write tid 1 to smithi16030472-146 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:146: finishing write tid 2 to smithi16030472-146 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:144: done (15 left) 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:159: write initial oid 159 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:159: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-21T18:07:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:159: writing smithi16030472-159 from 655506 to 1271908 tid 1 2024-09-21T18:07:20.411 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:20.409+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:20.413 INFO:tasks.rados.rados.0.smithi160.stdout:159: writing smithi16030472-159 from 1829449 to 2504829 tid 2 2024-09-21T18:07:20.413 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: finishing write tid 1 to smithi16030472-143 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: finishing write tid 2 to smithi16030472-143 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:151: finishing write tid 1 to smithi16030472-151 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:151: finishing write tid 2 to smithi16030472-151 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: finishing write tid 3 to smithi16030472-143 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: finishing write tid 4 to smithi16030472-143 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: oid 143 updating version 0 to 87 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: oid 143 updating version 87 to 88 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: oid 143 version 88 is already newer than 86 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 143 v 88 (ObjNum 142 snap 0 seq_num 142) dirty exists 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:143: left oid 143 (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:07:20.466 INFO:tasks.rados.rados.0.smithi160.stdout:151: finishing write tid 3 to smithi16030472-151 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: finishing write tid 4 to smithi16030472-151 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: oid 151 updating version 0 to 113 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: oid 151 version 113 is already newer than 111 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: oid 151 version 113 is already newer than 112 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 151 v 113 (ObjNum 150 snap 0 seq_num 150) dirty exists 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: left oid 151 (ObjNum 150 snap 0 seq_num 150) 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:143: done (15 left) 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:151: done (14 left) 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:160: write initial oid 160 2024-09-21T18:07:20.467 INFO:tasks.rados.rados.0.smithi160.stdout:160: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-21T18:07:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:160: writing smithi16030472-160 from 672313 to 1363964 tid 1 2024-09-21T18:07:20.479 INFO:tasks.rados.rados.0.smithi160.stdout:160: writing smithi16030472-160 from 1887931 to 2306969 tid 2 2024-09-21T18:07:20.479 INFO:tasks.rados.rados.0.smithi160.stdout:161: write initial oid 161 2024-09-21T18:07:20.479 INFO:tasks.rados.rados.0.smithi160.stdout:161: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-21T18:07:20.486 INFO:tasks.rados.rados.0.smithi160.stdout:161: writing smithi16030472-161 from 689120 to 1172373 tid 1 2024-09-21T18:07:20.495 INFO:tasks.rados.rados.0.smithi160.stdout:161: writing smithi16030472-161 from 1946413 to 2625462 tid 2 2024-09-21T18:07:20.495 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.512 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:20.509+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:20.534 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:20.534 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:20.548 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:20.567 INFO:tasks.rados.rados.0.smithi160.stdout:152: finishing write tid 1 to smithi16030472-152 2024-09-21T18:07:20.567 INFO:tasks.rados.rados.0.smithi160.stdout:152: finishing write tid 2 to smithi16030472-152 2024-09-21T18:07:20.613 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:20.607+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: finishing write tid 3 to smithi16030472-152 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: finishing write tid 4 to smithi16030472-152 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: oid 152 updating version 0 to 89 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: oid 152 updating version 89 to 91 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: oid 152 version 91 is already newer than 90 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 152 v 91 (ObjNum 151 snap 0 seq_num 151) dirty exists 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: left oid 152 (ObjNum 151 snap 0 seq_num 151) 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:152: done (15 left) 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:162: write initial oid 162 2024-09-21T18:07:20.645 INFO:tasks.rados.rados.0.smithi160.stdout:162: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-21T18:07:20.653 INFO:tasks.rados.rados.0.smithi160.stdout:162: writing smithi16030472-162 from 705927 to 1264429 tid 1 2024-09-21T18:07:20.661 INFO:tasks.rados.rados.0.smithi160.stdout:162: writing smithi16030472-162 from 2004895 to 2427602 tid 2 2024-09-21T18:07:20.661 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: finishing write tid 3 to smithi16030472-136 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: finishing write tid 4 to smithi16030472-136 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: oid 136 updating version 0 to 69 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: oid 136 version 69 is already newer than 68 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: oid 136 version 69 is already newer than 67 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 136 v 69 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:136: left oid 136 (ObjNum 135 snap 0 seq_num 135) 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:146: finishing write tid 3 to smithi16030472-146 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:146: finishing write tid 4 to smithi16030472-146 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:146: oid 146 updating version 0 to 79 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:146: oid 146 updating version 79 to 80 2024-09-21T18:07:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:146: oid 146 version 80 is already newer than 78 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 80 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:146: left oid 146 (ObjNum 145 snap 0 seq_num 145) 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:149: finishing write tid 1 to smithi16030472-149 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:149: finishing write tid 2 to smithi16030472-149 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:136: done (15 left) 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:146: done (14 left) 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:163: write initial oid 163 2024-09-21T18:07:20.689 INFO:tasks.rados.rados.0.smithi160.stdout:163: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-21T18:07:20.689 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:20.690 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:20.692 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:20.692 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:20.699 INFO:tasks.rados.rados.0.smithi160.stdout:163: writing smithi16030472-163 from 722734 to 1356485 tid 1 2024-09-21T18:07:20.707 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:20.712 INFO:tasks.rados.rados.0.smithi160.stdout:163: writing smithi16030472-163 from 2063377 to 2722734 tid 2 2024-09-21T18:07:20.712 INFO:tasks.rados.rados.0.smithi160.stdout:164: write initial oid 164 2024-09-21T18:07:20.712 INFO:tasks.rados.rados.0.smithi160.stdout:164: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-21T18:07:20.714 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:20.709+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:20.724 INFO:tasks.rados.rados.0.smithi160.stdout:164: writing smithi16030472-164 from 739541 to 1448541 tid 1 2024-09-21T18:07:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:20.734 INFO:tasks.rados.rados.0.smithi160.stdout:164: writing smithi16030472-164 from 2121859 to 2548235 tid 2 2024-09-21T18:07:20.734 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.773 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:20.815 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:20.809+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:20.816 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:20.816 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:20.829 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:20.829 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:20.843 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:20.850 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:20.877 INFO:tasks.rados.rados.0.smithi160.stdout:153: finishing write tid 1 to smithi16030472-153 2024-09-21T18:07:20.877 INFO:tasks.rados.rados.0.smithi160.stdout:153: finishing write tid 2 to smithi16030472-153 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: finishing write tid 1 to smithi16030472-156 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: finishing write tid 2 to smithi16030472-156 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: finishing write tid 3 to smithi16030472-156 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: finishing write tid 4 to smithi16030472-156 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: oid 156 updating version 0 to 116 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: oid 156 version 116 is already newer than 114 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: oid 156 version 116 is already newer than 115 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 156 v 116 (ObjNum 155 snap 0 seq_num 155) dirty exists 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:156: left oid 156 (ObjNum 155 snap 0 seq_num 155) 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:159: finishing write tid 1 to smithi16030472-159 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:159: finishing write tid 2 to smithi16030472-159 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:159: finishing write tid 3 to smithi16030472-159 2024-09-21T18:07:20.878 INFO:tasks.rados.rados.0.smithi160.stdout:159: finishing write tid 4 to smithi16030472-159 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:159: oid 159 updating version 0 to 119 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:159: oid 159 version 119 is already newer than 118 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:159: oid 159 version 119 is already newer than 117 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 159 v 119 (ObjNum 158 snap 0 seq_num 158) dirty exists 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:159: left oid 159 (ObjNum 158 snap 0 seq_num 158) 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: finishing write tid 3 to smithi16030472-153 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: finishing write tid 4 to smithi16030472-153 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: oid 153 updating version 0 to 94 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: oid 153 version 94 is already newer than 92 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: oid 153 version 94 is already newer than 93 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 153 v 94 (ObjNum 152 snap 0 seq_num 152) dirty exists 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: left oid 153 (ObjNum 152 snap 0 seq_num 152) 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:153: done (15 left) 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:156: done (14 left) 2024-09-21T18:07:20.879 INFO:tasks.rados.rados.0.smithi160.stdout:159: done (13 left) 2024-09-21T18:07:20.880 INFO:tasks.rados.rados.0.smithi160.stdout:165: write initial oid 165 2024-09-21T18:07:20.880 INFO:tasks.rados.rados.0.smithi160.stdout:165: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-21T18:07:20.886 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:20.886 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:20.889 INFO:tasks.rados.rados.0.smithi160.stdout:165: writing smithi16030472-165 from 756348 to 1540597 tid 1 2024-09-21T18:07:20.898 INFO:tasks.rados.rados.0.smithi160.stdout:165: writing smithi16030472-165 from 2063988 to 2750375 tid 2 2024-09-21T18:07:20.898 INFO:tasks.rados.rados.0.smithi160.stdout:166: write initial oid 166 2024-09-21T18:07:20.898 INFO:tasks.rados.rados.0.smithi160.stdout:166: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-21T18:07:20.908 INFO:tasks.rados.rados.0.smithi160.stdout:166: writing smithi16030472-166 from 773155 to 1232653 tid 1 2024-09-21T18:07:20.915 INFO:tasks.rados.rados.0.smithi160.stdout:166: writing smithi16030472-166 from 1722470 to 2152515 tid 2 2024-09-21T18:07:20.915 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:20.913+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:20.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:20.917 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:20.920 INFO:tasks.rados.rados.0.smithi160.stdout:166: writing smithi16030472-166 from 2648676 to 2773155 tid 3 2024-09-21T18:07:20.920 INFO:tasks.rados.rados.0.smithi160.stdout:167: write initial oid 167 2024-09-21T18:07:20.920 INFO:tasks.rados.rados.0.smithi160.stdout:167: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-21T18:07:20.930 INFO:tasks.rados.rados.0.smithi160.stdout:167: writing smithi16030472-167 from 789962 to 1324709 tid 1 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:167: writing smithi16030472-167 from 1780952 to 2471008 tid 2 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:142: finishing write tid 3 to smithi16030472-142 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:142: finishing write tid 4 to smithi16030472-142 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:142: oid 142 updating version 0 to 71 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:142: oid 142 version 71 is already newer than 70 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:142: oid 142 updating version 71 to 72 2024-09-21T18:07:20.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 72 (ObjNum 141 snap 0 seq_num 141) dirty exists 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:142: left oid 142 (ObjNum 141 snap 0 seq_num 141) 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: finishing write tid 1 to smithi16030472-147 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: finishing write tid 2 to smithi16030472-147 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: finishing write tid 3 to smithi16030472-147 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: finishing write tid 4 to smithi16030472-147 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: oid 147 updating version 0 to 74 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: oid 147 version 74 is already newer than 73 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: oid 147 updating version 74 to 75 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 75 (ObjNum 146 snap 0 seq_num 146) dirty exists 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: left oid 147 (ObjNum 146 snap 0 seq_num 146) 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:142: done (15 left) 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:147: done (14 left) 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:168: write initial oid 168 2024-09-21T18:07:20.940 INFO:tasks.rados.rados.0.smithi160.stdout:168: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-21T18:07:20.946 INFO:tasks.rados.rados.0.smithi160.stdout:168: writing smithi16030472-168 from 406769 to 1016765 tid 1 2024-09-21T18:07:20.952 INFO:tasks.rados.rados.0.smithi160.stdout:168: writing smithi16030472-168 from 1439434 to 1873148 tid 2 2024-09-21T18:07:20.959 INFO:tasks.rados.rados.0.smithi160.stdout:168: writing smithi16030472-168 from 2303522 to 2806769 tid 3 2024-09-21T18:07:20.959 INFO:tasks.rados.rados.0.smithi160.stdout:169: write initial oid 169 2024-09-21T18:07:20.959 INFO:tasks.rados.rados.0.smithi160.stdout:169: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-21T18:07:20.964 INFO:tasks.rados.rados.0.smithi160.stdout:169: writing smithi16030472-169 from 423576 to 825174 tid 1 2024-09-21T18:07:20.974 INFO:tasks.rados.rados.0.smithi160.stdout:169: writing smithi16030472-169 from 1497916 to 2191641 tid 2 2024-09-21T18:07:20.974 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:20.995 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:20.995 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:21.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:21.016 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:21.013+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:21.116 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:21.111+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:21.141 INFO:tasks.rados.rados.0.smithi160.stdout:149: finishing write tid 3 to smithi16030472-149 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: finishing write tid 4 to smithi16030472-149 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: oid 149 updating version 0 to 82 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: oid 149 version 82 is already newer than 81 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: oid 149 updating version 82 to 83 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 83 (ObjNum 148 snap 0 seq_num 148) dirty exists 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: left oid 149 (ObjNum 148 snap 0 seq_num 148) 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:157: finishing write tid 1 to smithi16030472-157 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:149: done (15 left) 2024-09-21T18:07:21.142 INFO:tasks.rados.rados.0.smithi160.stdout:170: write initial oid 170 2024-09-21T18:07:21.143 INFO:tasks.rados.rados.0.smithi160.stdout:170: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-21T18:07:21.147 INFO:tasks.rados.rados.0.smithi160.stdout:170: writing smithi16030472-170 from 440383 to 917230 tid 1 2024-09-21T18:07:21.154 INFO:tasks.rados.rados.0.smithi160.stdout:170: writing smithi16030472-170 from 1556398 to 1993781 tid 2 2024-09-21T18:07:21.159 INFO:tasks.rados.rados.0.smithi160.stdout:170: writing smithi16030472-170 from 2474721 to 2840383 tid 3 2024-09-21T18:07:21.159 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.179 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:21.180 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:21.182 INFO:tasks.rados.rados.0.smithi160.stdout:163: finishing write tid 1 to smithi16030472-163 2024-09-21T18:07:21.184 INFO:tasks.rados.rados.0.smithi160.stdout:150: finishing write tid 1 to smithi16030472-150 2024-09-21T18:07:21.184 INFO:tasks.rados.rados.0.smithi160.stdout:150: finishing write tid 2 to smithi16030472-150 2024-09-21T18:07:21.184 INFO:tasks.rados.rados.0.smithi160.stdout:150: finishing write tid 3 to smithi16030472-150 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:150: finishing write tid 4 to smithi16030472-150 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:150: oid 150 updating version 0 to 76 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:150: oid 150 updating version 76 to 77 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:150: oid 150 updating version 77 to 78 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 150 v 78 (ObjNum 149 snap 0 seq_num 149) dirty exists 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:150: left oid 150 (ObjNum 149 snap 0 seq_num 149) 2024-09-21T18:07:21.185 INFO:tasks.rados.rados.0.smithi160.stdout:154: finishing write tid 1 to smithi16030472-154 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: finishing write tid 2 to smithi16030472-154 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: finishing write tid 3 to smithi16030472-154 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: finishing write tid 4 to smithi16030472-154 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: oid 154 updating version 0 to 81 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: oid 154 version 81 is already newer than 79 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: oid 154 version 81 is already newer than 80 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 154 v 81 (ObjNum 153 snap 0 seq_num 153) dirty exists 2024-09-21T18:07:21.186 INFO:tasks.rados.rados.0.smithi160.stdout:154: left oid 154 (ObjNum 153 snap 0 seq_num 153) 2024-09-21T18:07:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:155: finishing write tid 1 to smithi16030472-155 2024-09-21T18:07:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:150: done (15 left) 2024-09-21T18:07:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:154: done (14 left) 2024-09-21T18:07:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:171: write initial oid 171 2024-09-21T18:07:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:171: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-21T18:07:21.191 INFO:tasks.rados.rados.0.smithi160.stdout:171: writing smithi16030472-171 from 457190 to 1009286 tid 1 2024-09-21T18:07:21.196 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:21.199 INFO:tasks.rados.rados.0.smithi160.stdout:171: writing smithi16030472-171 from 1614880 to 2195921 tid 2 2024-09-21T18:07:21.205 INFO:tasks.rados.rados.0.smithi160.stdout:171: writing smithi16030472-171 from 2785791 to 2857190 tid 3 2024-09-21T18:07:21.205 INFO:tasks.rados.rados.0.smithi160.stdout:172: write initial oid 172 2024-09-21T18:07:21.205 INFO:tasks.rados.rados.0.smithi160.stdout:172: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-21T18:07:21.213 INFO:tasks.rados.rados.0.smithi160.stdout:172: writing smithi16030472-172 from 473997 to 1101342 tid 1 2024-09-21T18:07:21.217 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:21.213+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:21.220 INFO:tasks.rados.rados.0.smithi160.stdout:172: writing smithi16030472-172 from 1673362 to 2114414 tid 2 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout:172: writing smithi16030472-172 from 2529567 to 2873997 tid 3 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout:155: finishing write tid 2 to smithi16030472-155 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout:155: finishing write tid 3 to smithi16030472-155 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout:155: finishing write tid 4 to smithi16030472-155 2024-09-21T18:07:21.226 INFO:tasks.rados.rados.0.smithi160.stdout:155: oid 155 updating version 0 to 83 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:155: oid 155 version 83 is already newer than 82 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:155: oid 155 updating version 83 to 84 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 155 v 84 (ObjNum 154 snap 0 seq_num 154) dirty exists 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:155: left oid 155 (ObjNum 154 snap 0 seq_num 154) 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: finishing write tid 2 to smithi16030472-157 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: finishing write tid 3 to smithi16030472-157 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: finishing write tid 4 to smithi16030472-157 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: oid 157 updating version 0 to 86 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: oid 157 version 86 is already newer than 84 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: oid 157 version 86 is already newer than 85 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 86 (ObjNum 156 snap 0 seq_num 156) dirty exists 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:157: left oid 157 (ObjNum 156 snap 0 seq_num 156) 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:158: finishing write tid 1 to smithi16030472-158 2024-09-21T18:07:21.227 INFO:tasks.rados.rados.0.smithi160.stdout:155: done (15 left) 2024-09-21T18:07:21.228 INFO:tasks.rados.rados.0.smithi160.stdout:157: done (14 left) 2024-09-21T18:07:21.228 INFO:tasks.rados.rados.0.smithi160.stdout:173: write initial oid 173 2024-09-21T18:07:21.228 INFO:tasks.rados.rados.0.smithi160.stdout:173: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-21T18:07:21.234 INFO:tasks.rados.rados.0.smithi160.stdout:173: writing smithi16030472-173 from 490804 to 1193398 tid 1 2024-09-21T18:07:21.242 INFO:tasks.rados.rados.0.smithi160.stdout:173: writing smithi16030472-173 from 1615491 to 2200201 tid 2 2024-09-21T18:07:21.247 INFO:tasks.rados.rados.0.smithi160.stdout:173: writing smithi16030472-173 from 2724284 to 2890804 tid 3 2024-09-21T18:07:21.247 INFO:tasks.rados.rados.0.smithi160.stdout:174: write initial oid 174 2024-09-21T18:07:21.247 INFO:tasks.rados.rados.0.smithi160.stdout:174: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-21T18:07:21.255 INFO:tasks.rados.rados.0.smithi160.stdout:174: writing smithi16030472-174 from 507611 to 1285454 tid 1 2024-09-21T18:07:21.263 INFO:tasks.rados.rados.0.smithi160.stdout:174: writing smithi16030472-174 from 2073973 to 2518694 tid 2 2024-09-21T18:07:21.263 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.263 INFO:tasks.rados.rados.0.smithi160.stdout:161: finishing write tid 1 to smithi16030472-161 2024-09-21T18:07:21.299 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:07:21.306 INFO:tasks.rados.rados.0.smithi160.stdout:161: finishing write tid 2 to smithi16030472-161 2024-09-21T18:07:21.307 INFO:tasks.rados.rados.0.smithi160.stdout:161: finishing write tid 3 to smithi16030472-161 2024-09-21T18:07:21.317 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:21.313+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:21.326 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:21.326 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:21.341 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:21.400 INFO:tasks.rados.rados.0.smithi160.stdout:161: finishing write tid 4 to smithi16030472-161 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:161: oid 161 updating version 0 to 90 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:161: oid 161 version 90 is already newer than 88 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:161: oid 161 version 90 is already newer than 89 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 161 v 90 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:161: left oid 161 (ObjNum 160 snap 0 seq_num 160) 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:161: done (15 left) 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:175: write initial oid 175 2024-09-21T18:07:21.401 INFO:tasks.rados.rados.0.smithi160.stdout:175: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-21T18:07:21.406 INFO:tasks.rados.rados.0.smithi160.stdout:175: writing smithi16030472-175 from 524418 to 977510 tid 1 2024-09-21T18:07:21.414 INFO:tasks.rados.rados.0.smithi160.stdout:175: writing smithi16030472-175 from 1732455 to 2320834 tid 2 2024-09-21T18:07:21.418 INFO:tasks.rados.rados.0.smithi160.stdout:175: writing smithi16030472-175 from 2779130 to 2924418 tid 3 2024-09-21T18:07:21.418 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.418 INFO:tasks.rados.rados.0.smithi160.stdout:162: finishing write tid 1 to smithi16030472-162 2024-09-21T18:07:21.418 INFO:tasks.rados.rados.0.smithi160.stdout:162: finishing write tid 2 to smithi16030472-162 2024-09-21T18:07:21.418 INFO:tasks.rados.rados.0.smithi160.stdout:162: finishing write tid 3 to smithi16030472-162 2024-09-21T18:07:21.419 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:21.417+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: finishing write tid 2 to smithi16030472-163 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: finishing write tid 3 to smithi16030472-163 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: finishing write tid 4 to smithi16030472-163 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: oid 163 updating version 0 to 122 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: oid 163 version 122 is already newer than 120 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: oid 163 version 122 is already newer than 121 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 163 v 122 (ObjNum 162 snap 0 seq_num 162) dirty exists 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:163: left oid 163 (ObjNum 162 snap 0 seq_num 162) 2024-09-21T18:07:21.489 INFO:tasks.rados.rados.0.smithi160.stdout:164: finishing write tid 1 to smithi16030472-164 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: finishing write tid 2 to smithi16030472-164 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: finishing write tid 3 to smithi16030472-164 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: finishing write tid 4 to smithi16030472-164 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: oid 164 updating version 0 to 95 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: oid 164 updating version 95 to 97 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: oid 164 version 97 is already newer than 96 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 97 (ObjNum 163 snap 0 seq_num 163) dirty exists 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: left oid 164 (ObjNum 163 snap 0 seq_num 163) 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:163: done (15 left) 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:164: done (14 left) 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:176: write initial oid 176 2024-09-21T18:07:21.490 INFO:tasks.rados.rados.0.smithi160.stdout:176: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-21T18:07:21.497 INFO:tasks.rados.rados.0.smithi160.stdout:176: writing smithi16030472-176 from 541225 to 1185919 tid 1 2024-09-21T18:07:21.505 INFO:tasks.rados.rados.0.smithi160.stdout:176: writing smithi16030472-176 from 1907290 to 2355680 tid 2 2024-09-21T18:07:21.505 INFO:tasks.rados.rados.0.smithi160.stdout:177: write initial oid 177 2024-09-21T18:07:21.505 INFO:tasks.rados.rados.0.smithi160.stdout:177: seq_num 176 ranges {558032=719943,1849419=592048} 2024-09-21T18:07:21.511 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:756 4.3 4:c92382c6:::smithi16030472-173:head [write 1615491~584710 in=584710b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.385654+0000", 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121429037, 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121477132, 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.512 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 756 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385654+0000", 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385654+0000", 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385655+0000", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6499999999999999e-07 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386010+0000", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035496900000000001 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386018+0000", 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386022+0000", 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5600000000000004e-06 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386288+0000", 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000266082 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:757 4.3 4:c92382c6:::smithi16030472-173:head [write 2724284~166520 in=166520b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.515 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386034+0000", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12104916, 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12120268100000001, 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 757 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386034+0000", 2024-09-21T18:07:21.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386034+0000", 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386034+0000", 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9500000000000002e-07 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386143+0000", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010812800000000001 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386148+0000", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.643e-06 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386155+0000", 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7100000000000001e-06 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386380+0000", 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022520600000000001 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:758 4.3 4:c92382c6:::smithi16030472-173:head [setxattr _header (57) in=64b,truncate 2890804] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386162+0000", 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12092143399999999, 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12116547499999999, 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.519 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 758 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386162+0000", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386162+0000", 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386162+0000", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0800000000000004e-07 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386165+0000", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0180000000000002e-06 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386170+0000", 2024-09-21T18:07:21.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4889999999999997e-06 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386175+0000", 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2000000000000002e-06 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386462+0000", 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028728800000000002 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:759 4.3 4:c92382c6:::smithi16030472-173:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386180+0000", 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12090303199999999, 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121230174, 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 759 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386180+0000", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386180+0000", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386180+0000", 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2e-07 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386183+0000", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1440000000000001e-06 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386187+0000", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3760000000000001e-06 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386191+0000", 2024-09-21T18:07:21.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.968e-06 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.486854+0000", 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10066320300000001 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:718 4.3 4:c3d868ee:::smithi16030472-165:head [write 756348~784249 in=784249b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:20.977674+0000", 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.529408832, 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.52982153700000001, 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 718 2024-09-21T18:07:21.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.977674+0000", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.977674+0000", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.977676+0000", 2024-09-21T18:07:21.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3629999999999999e-06 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978046+0000", 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037031100000000001 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978059+0000", 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2692e-05 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978074+0000", 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5023e-05 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.078441+0000", 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10036724800000001 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:760 4.3 4:c9a5ebb4:::smithi16030472-174:head [write 507611~777843 in=777843b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386196+0000", 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.120887253, 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12138154900000001, 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 760 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386196+0000", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386196+0000", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386196+0000", 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.51e-07 2024-09-21T18:07:21.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386653+0000", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045703699999999998 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386661+0000", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386666+0000", 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7809999999999996e-06 2024-09-21T18:07:21.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487124+0000", 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10045778800000001 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:719 4.3 4:c3d868ee:::smithi16030472-165:head [write 2063988~686387 in=686387b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:20.978096+0000", 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52898717399999995, 2024-09-21T18:07:21.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.52956309700000004, 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 719 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978096+0000", 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978096+0000", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978097+0000", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5370000000000001e-06 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978463+0000", 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036566700000000002 2024-09-21T18:07:21.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978481+0000", 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8068999999999999e-05 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978497+0000", 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5590000000000002e-05 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.078781+0000", 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100284652 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:761 4.3 4:c9a5ebb4:::smithi16030472-174:head [write 2073973~444721 in=444721b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386675+0000", 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.120407977, 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121064967, 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.538 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 761 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386675+0000", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386675+0000", 2024-09-21T18:07:21.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386676+0000", 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2600000000000002e-07 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386942+0000", 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026684400000000002 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386951+0000", 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.038e-06 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386955+0000", 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4270000000000001e-06 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487489+0000", 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100534079 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:720 4.3 4:c3d868ee:::smithi16030472-165:head [setxattr _header (57) in=64b,truncate 2756348] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:20.978520+0000", 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52856308900000004, 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.52930595999999996, 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 720 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.542 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978520+0000", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978520+0000", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978522+0000", 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1100000000000001e-06 2024-09-21T18:07:21.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978530+0000", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4030000000000007e-06 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978546+0000", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5073e-05 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978559+0000", 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3049e-05 2024-09-21T18:07:21.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079005+0000", 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10044689699999999 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:762 4.3 4:c9a5ebb4:::smithi16030472-174:head [setxattr _header (57) in=64b,truncate 2907611] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.386962+0000", 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12012141699999999, 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120945965, 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.545 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 762 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386962+0000", 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386962+0000", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386962+0000", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0699999999999997e-07 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386965+0000", 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.774e-06 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386970+0000", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.254e-06 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487083+0000", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10011315599999999 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487914+0000", 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00083023799999999996 2024-09-21T18:07:21.548 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:721 4.3 4:c3d868ee:::smithi16030472-165:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:20.978577+0000", 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52850581699999999, 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.52941120500000005, 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.549 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 721 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978577+0000", 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978577+0000", 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978579+0000", 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6530000000000001e-06 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978588+0000", 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6819999999999992e-06 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978603+0000", 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.505e-05 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978619+0000", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5999e-05 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079200+0000", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100581344 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:763 4.3 4.2dd7a593 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.487123+0000", 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019960062000000001, 2024-09-21T18:07:21.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020943921000000001, 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 763 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487123+0000", 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487123+0000", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487125+0000", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4889999999999998e-06 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487134+0000", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6300000000000004e-06 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487147+0000", 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2802999999999999e-05 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.487156+0000", 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4120000000000007e-06 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:731 4.1 4:9a656cd6:::smithi16030472-168:head [write 406769~609996 in=609996b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:20.978633+0000", 2024-09-21T18:07:21.555 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52845047300000003, 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.52951036900000004, 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 731 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978633+0000", 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978633+0000", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978634+0000", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1599999999999999e-06 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978878+0000", 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024426800000000001 2024-09-21T18:07:21.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.978891+0000", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3413e-05 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.078974+0000", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10008257499999999 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079097+0000", 2024-09-21T18:07:21.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012265 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:732 4.1 4:9a656cd6:::smithi16030472-168:head [write 1439434~433714 in=433714b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.079012+0000", 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42807099399999998, 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42921511099999998, 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.559 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 732 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079012+0000", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079012+0000", 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079014+0000", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1220000000000002e-06 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079406+0000", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039191300000000003 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079419+0000", 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2729e-05 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079429+0000", 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.781e-06 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079762+0000", 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00033302899999999999 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.562 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:733 4.1 4:9a656cd6:::smithi16030472-168:head [write 2303522~503247 in=503247b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.079467+0000", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42761629600000001, 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42884313600000001, 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 733 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.563 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079467+0000", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079467+0000", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079468+0000", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.373e-06 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079669+0000", 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020135199999999999 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079685+0000", 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5526e-05 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079695+0000", 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0127e-05 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079977+0000", 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000282164 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:734 4.1 4:9a656cd6:::smithi16030472-168:head [setxattr _header (57) in=64b,truncate 2806769] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.079714+0000", 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42736879900000002, 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42868511599999998, 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.566 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 734 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079714+0000", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079714+0000", 2024-09-21T18:07:21.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079716+0000", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.389e-06 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079721+0000", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4280000000000004e-06 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079730+0000", 2024-09-21T18:07:21.568 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7760000000000003e-06 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079737+0000", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8909999999999998e-06 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080221+0000", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048395900000000001 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080283+0000", 2024-09-21T18:07:21.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1988000000000003e-05 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:769 4.1 4.711b20f5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e48)", 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.496970+0000", 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.010112588, 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.011516531999999999, 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.570 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 769 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.496970+0000", 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.496970+0000", 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.496973+0000", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.486e-06 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.497388+0000", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00041540899999999999 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.497405+0000", 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7025e-05 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.572 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:735 4.1 4:9a656cd6:::smithi16030472-168:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.079747+0000", 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.427335613, 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.428807569, 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 735 2024-09-21T18:07:21.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079747+0000", 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079747+0000", 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079748+0000", 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9199999999999999e-07 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079752+0000", 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9040000000000002e-06 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079760+0000", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1580000000000005e-06 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079768+0000", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3810000000000001e-06 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.179654+0000", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099886469000000006 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:21.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.179817+0000", 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016278399999999999 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:736 4.3 4:f7a13d16:::smithi16030472-169:head [write 423576~401598 in=401598b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.079778+0000", 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42730524199999997, 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.428867938, 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.577 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 736 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079778+0000", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079778+0000", 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.079779+0000", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4499999999999995e-07 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080093+0000", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031385899999999999 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080104+0000", 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1725999999999999e-05 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080112+0000", 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1440000000000003e-06 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080185+0000", 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2103999999999999e-05 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:737 4.3 4:f7a13d16:::smithi16030472-169:head [write 1497916~693725 in=693725b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.580 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.080133+0000", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42694998499999998, 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42859445099999999, 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 737 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080133+0000", 2024-09-21T18:07:21.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080133+0000", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080134+0000", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2929999999999999e-06 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080379+0000", 2024-09-21T18:07:21.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024433199999999999 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080389+0000", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0601999999999999e-05 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080414+0000", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4973999999999998e-05 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080476+0000", 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2045000000000007e-05 2024-09-21T18:07:21.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:738 4.3 4:f7a13d16:::smithi16030472-169:head [setxattr _header (57) in=64b,truncate 2823576] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.080428+0000", 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42665467499999998, 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42839946099999998, 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 738 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080428+0000", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080428+0000", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080429+0000", 2024-09-21T18:07:21.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0640000000000001e-06 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080435+0000", 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6339999999999998e-06 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080443+0000", 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2139999999999996e-06 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080451+0000", 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3989999999999998e-06 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080661+0000", 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021025800000000001 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:739 4.3 4:f7a13d16:::smithi16030472-169:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.080470+0000", 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42661315100000002, 2024-09-21T18:07:21.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42844011700000001, 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 739 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080470+0000", 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080470+0000", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080471+0000", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6199999999999996e-07 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080477+0000", 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.818e-06 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080487+0000", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0287e-05 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.080495+0000", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7680000000000008e-06 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.082680+0000", 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002185144 2024-09-21T18:07:21.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:740 4.1 4:af38ec1d:::smithi16030472-170:head [write 440383~476847 in=476847b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.182289+0000", 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.32479394700000003, 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32670405400000002, 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 740 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.182289+0000", 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.182289+0000", 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.182293+0000", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.183026+0000", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00073359800000000002 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.183052+0000", 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5205999999999999e-05 2024-09-21T18:07:21.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283164+0000", 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100112522 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.383415+0000", 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100250903 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.594 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:741 4.1 4:af38ec1d:::smithi16030472-170:head [write 1556398~437383 in=437383b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.283193+0000", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.223890383, 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.225882684, 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 741 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283193+0000", 2024-09-21T18:07:21.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283193+0000", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283195+0000", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4619999999999999e-06 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283785+0000", 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00059037100000000002 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283810+0000", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4151999999999998e-05 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283823+0000", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3040000000000001e-05 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.383670+0000", 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099847293000000004 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.597 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:742 4.1 4:af38ec1d:::smithi16030472-170:head [write 2474721~365662 in=365662b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.283842+0000", 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22324128600000001, 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.225315764, 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 742 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.598 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283842+0000", 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283842+0000", 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.283843+0000", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.367e-06 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284212+0000", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036935700000000001 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284225+0000", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.275e-05 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284234+0000", 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3089999999999996e-06 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.384098+0000", 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099864184999999994 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:743 4.1 4:af38ec1d:::smithi16030472-170:head [setxattr _header (57) in=64b,truncate 2840383] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.284246+0000", 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22283723, 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22499538499999999, 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.601 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 743 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284246+0000", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284246+0000", 2024-09-21T18:07:21.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284247+0000", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.147e-06 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284252+0000", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4e-06 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284260+0000", 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1329999999999999e-06 2024-09-21T18:07:21.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284268+0000", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5989999999999996e-06 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.384213+0000", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099944634000000004 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:744 4.1 4:af38ec1d:::smithi16030472-170:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.284280+0000", 2024-09-21T18:07:21.604 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.222803471, 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.225042506, 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 744 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284280+0000", 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284280+0000", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284281+0000", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7699999999999992e-07 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284284+0000", 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5559999999999999e-06 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284293+0000", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6060000000000003e-06 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284300+0000", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3440000000000003e-06 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.384349+0000", 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10004948 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.607 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:750 4.1 4:a40e217e:::smithi16030472-172:head [write 473997~627345 in=627345b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.284309+0000", 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.222773528, 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22509604, 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 750 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284309+0000", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284309+0000", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284310+0000", 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0599999999999999e-07 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284561+0000", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025059799999999999 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284573+0000", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2063e-05 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284582+0000", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3759999999999997e-06 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385138+0000", 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100555746 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:751 4.1 4:a40e217e:::smithi16030472-172:head [write 1673362~441052 in=441052b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.284593+0000", 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22248992200000001, 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.224895345, 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.611 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 751 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284593+0000", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284593+0000", 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284594+0000", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2399999999999996e-07 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284978+0000", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000384433 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.284987+0000", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1149999999999994e-06 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385063+0000", 2024-09-21T18:07:21.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100076807 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385417+0000", 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035357999999999999 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:752 4.1 4:a40e217e:::smithi16030472-172:head [write 2529567~344430 in=344430b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.385086+0000", 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121997376, 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12448883600000001, 2024-09-21T18:07:21.614 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 752 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385086+0000", 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385086+0000", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385087+0000", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.869e-06 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385330+0000", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024244300000000001 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385343+0000", 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3086e-05 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385351+0000", 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8550000000000002e-06 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385535+0000", 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000183769 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:753 4.1 4:a40e217e:::smithi16030472-172:head [setxattr _header (57) in=64b,truncate 2873997] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.385362+0000", 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121720838, 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.124377816, 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 753 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385362+0000", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385362+0000", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385363+0000", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9500000000000001e-07 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385367+0000", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5839999999999999e-06 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385373+0000", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8639999999999998e-06 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385378+0000", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4780000000000002e-06 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385688+0000", 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00031060899999999999 2024-09-21T18:07:21.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:754 4.1 4:a40e217e:::smithi16030472-172:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.385406+0000", 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12167671400000001, 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.124418105, 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 754 2024-09-21T18:07:21.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385406+0000", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385406+0000", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385407+0000", 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.44e-07 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385409+0000", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.475e-06 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385417+0000", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6279999999999996e-06 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385421+0000", 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4649999999999996e-06 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.484007+0000", 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098586045999999997 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:755 4.3 4:c92382c6:::smithi16030472-173:head [write 490804~702594 in=702594b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:21.385428+0000", 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12165463899999999, 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12448169100000001, 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:21.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 755 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385428+0000", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385428+0000", 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385429+0000", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8999999999999996e-07 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385628+0000", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019920800000000001 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385634+0000", 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9560000000000002e-06 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.385639+0000", 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7829999999999998e-06 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:21.386108+0000", 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00046942799999999998 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 33 2024-09-21T18:07:21.627 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:21.629 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:21.630 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:21.630 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:21.630 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:21.631 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:21.631 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:21.625+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:21.631 INFO:tasks.rados.rados.0.smithi160.stdout:177: writing smithi16030472-177 from 558032 to 1277975 tid 1 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:177: writing smithi16030472-177 from 1849419 to 2441467 tid 2 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: finishing write tid 1 to smithi16030472-160 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: finishing write tid 2 to smithi16030472-160 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: finishing write tid 3 to smithi16030472-160 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: finishing write tid 4 to smithi16030472-160 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: oid 160 updating version 0 to 87 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: oid 160 version 87 is already newer than 85 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: oid 160 version 87 is already newer than 86 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 160 v 87 (ObjNum 159 snap 0 seq_num 159) dirty exists 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:160: left oid 160 (ObjNum 159 snap 0 seq_num 159) 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:162: finishing write tid 4 to smithi16030472-162 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:162: oid 162 updating version 0 to 92 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:162: oid 162 updating version 92 to 93 2024-09-21T18:07:21.632 INFO:tasks.rados.rados.0.smithi160.stdout:162: oid 162 version 93 is already newer than 91 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 162 v 93 (ObjNum 161 snap 0 seq_num 161) dirty exists 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:162: left oid 162 (ObjNum 161 snap 0 seq_num 161) 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:160: done (15 left) 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:162: done (14 left) 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:178: write initial oid 178 2024-09-21T18:07:21.633 INFO:tasks.rados.rados.0.smithi160.stdout:178: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-21T18:07:21.634 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:178: writing smithi16030472-178 from 574839 to 1370031 tid 1 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:178: writing smithi16030472-178 from 1907901 to 2359960 tid 2 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:179: write initial oid 179 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:179: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:179: writing smithi16030472-179 from 591646 to 1062087 tid 1 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:179: writing smithi16030472-179 from 1566383 to 2162100 tid 2 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:179: writing smithi16030472-179 from 2888822 to 2991646 tid 3 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.635 INFO:tasks.rados.rados.0.smithi160.stdout:165: finishing write tid 1 to smithi16030472-165 2024-09-21T18:07:21.640 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:21.675 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:21.675 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:165: finishing write tid 2 to smithi16030472-165 2024-09-21T18:07:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:165: finishing write tid 3 to smithi16030472-165 2024-09-21T18:07:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:165: finishing write tid 4 to smithi16030472-165 2024-09-21T18:07:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:165: oid 165 updating version 0 to 98 2024-09-21T18:07:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:165: oid 165 updating version 98 to 99 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:165: oid 165 updating version 99 to 100 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 165 v 100 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:165: left oid 165 (ObjNum 164 snap 0 seq_num 164) 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:165: done (15 left) 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:180: write initial oid 180 2024-09-21T18:07:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:180: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-21T18:07:21.698 INFO:tasks.rados.rados.0.smithi160.stdout:180: writing smithi16030472-180 from 608453 to 1154143 tid 1 2024-09-21T18:07:21.704 INFO:tasks.rados.rados.0.smithi160.stdout:180: writing smithi16030472-180 from 1624865 to 2080593 tid 2 2024-09-21T18:07:21.710 INFO:tasks.rados.rados.0.smithi160.stdout:180: writing smithi16030472-180 from 2632598 to 3008453 tid 3 2024-09-21T18:07:21.710 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.710 INFO:tasks.rados.rados.0.smithi160.stdout:169: finishing write tid 1 to smithi16030472-169 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: finishing write tid 2 to smithi16030472-169 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: finishing write tid 3 to smithi16030472-169 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: finishing write tid 4 to smithi16030472-169 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: oid 169 updating version 0 to 101 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: oid 169 updating version 101 to 103 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: oid 169 version 103 is already newer than 102 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 169 v 103 (ObjNum 168 snap 0 seq_num 168) dirty exists 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: left oid 169 (ObjNum 168 snap 0 seq_num 168) 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:173: finishing write tid 1 to smithi16030472-173 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:169: done (15 left) 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:181: write initial oid 181 2024-09-21T18:07:21.711 INFO:tasks.rados.rados.0.smithi160.stdout:181: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-21T18:07:21.718 INFO:tasks.rados.rados.0.smithi160.stdout:181: writing smithi16030472-181 from 625260 to 1246199 tid 1 2024-09-21T18:07:21.726 INFO:tasks.rados.rados.0.smithi160.stdout:181: writing smithi16030472-181 from 1683347 to 2282733 tid 2 2024-09-21T18:07:21.730 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:21.726+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:21.731 INFO:tasks.rados.rados.0.smithi160.stdout:181: writing smithi16030472-181 from 2943668 to 3025260 tid 3 2024-09-21T18:07:21.731 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.812 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:21.812 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:21.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:21.831 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:21.826+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:21.893 INFO:tasks.rados.rados.0.smithi160.stdout:168: finishing write tid 1 to smithi16030472-168 2024-09-21T18:07:21.893 INFO:tasks.rados.rados.0.smithi160.stdout:168: finishing write tid 2 to smithi16030472-168 2024-09-21T18:07:21.893 INFO:tasks.rados.rados.0.smithi160.stdout:168: finishing write tid 3 to smithi16030472-168 2024-09-21T18:07:21.912 INFO:tasks.rados.rados.0.smithi160.stdout:177: finishing write tid 1 to smithi16030472-177 2024-09-21T18:07:21.932 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:21.927+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:21.943 INFO:tasks.rados.rados.0.smithi160.stdout:166: finishing write tid 1 to smithi16030472-166 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: finishing write tid 2 to smithi16030472-166 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: finishing write tid 3 to smithi16030472-166 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: finishing write tid 4 to smithi16030472-166 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: finishing write tid 5 to smithi16030472-166 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: oid 166 updating version 0 to 96 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: oid 166 updating version 96 to 97 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: oid 166 version 97 is already newer than 94 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: oid 166 version 97 is already newer than 95 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 97 (ObjNum 165 snap 0 seq_num 165) dirty exists 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:166: left oid 166 (ObjNum 165 snap 0 seq_num 165) 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:167: finishing write tid 1 to smithi16030472-167 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:167: finishing write tid 2 to smithi16030472-167 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:167: finishing write tid 3 to smithi16030472-167 2024-09-21T18:07:21.944 INFO:tasks.rados.rados.0.smithi160.stdout:167: finishing write tid 4 to smithi16030472-167 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:167: oid 167 updating version 0 to 98 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:167: oid 167 updating version 98 to 100 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:167: oid 167 version 100 is already newer than 99 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 167 v 100 (ObjNum 166 snap 0 seq_num 166) dirty exists 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:167: left oid 167 (ObjNum 166 snap 0 seq_num 166) 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:166: done (15 left) 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:167: done (14 left) 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:182: write initial oid 182 2024-09-21T18:07:21.945 INFO:tasks.rados.rados.0.smithi160.stdout:182: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-21T18:07:21.952 INFO:tasks.rados.rados.0.smithi160.stdout:182: writing smithi16030472-182 from 642067 to 1338255 tid 1 2024-09-21T18:07:21.960 INFO:tasks.rados.rados.0.smithi160.stdout:182: writing smithi16030472-182 from 2025476 to 2484873 tid 2 2024-09-21T18:07:21.963 INFO:tasks.rados.rados.0.smithi160.stdout:182: writing smithi16030472-182 from 2971091 to 3042067 tid 3 2024-09-21T18:07:21.963 INFO:tasks.rados.rados.0.smithi160.stdout:183: write initial oid 183 2024-09-21T18:07:21.963 INFO:tasks.rados.rados.0.smithi160.stdout:183: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-21T18:07:21.973 INFO:tasks.rados.rados.0.smithi160.stdout:183: writing smithi16030472-183 from 658874 to 1430311 tid 1 2024-09-21T18:07:21.981 INFO:tasks.rados.rados.0.smithi160.stdout:183: writing smithi16030472-183 from 2083958 to 2687013 tid 2 2024-09-21T18:07:21.981 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:21.981 INFO:tasks.rados.rados.0.smithi160.stdout:171: finishing write tid 1 to smithi16030472-171 2024-09-21T18:07:21.981 INFO:tasks.rados.rados.0.smithi160.stdout:171: finishing write tid 2 to smithi16030472-171 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: finishing write tid 3 to smithi16030472-171 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: finishing write tid 4 to smithi16030472-171 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: finishing write tid 5 to smithi16030472-171 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: oid 171 updating version 0 to 101 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: oid 171 updating version 101 to 102 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: oid 171 updating version 102 to 103 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: oid 171 updating version 103 to 104 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 171 v 104 (ObjNum 170 snap 0 seq_num 170) dirty exists 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: left oid 171 (ObjNum 170 snap 0 seq_num 170) 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:175: finishing write tid 1 to smithi16030472-175 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:158: finishing write tid 2 to smithi16030472-158 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:171: done (15 left) 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:184: write initial oid 184 2024-09-21T18:07:21.982 INFO:tasks.rados.rados.0.smithi160.stdout:184: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-21T18:07:21.989 INFO:tasks.rados.rados.0.smithi160.stdout:184: writing smithi16030472-184 from 675681 to 1238720 tid 1 2024-09-21T18:07:21.998 INFO:tasks.rados.rados.0.smithi160.stdout:184: writing smithi16030472-184 from 1858793 to 2605506 tid 2 2024-09-21T18:07:22.001 INFO:tasks.rados.rados.0.smithi160.stdout:184: writing smithi16030472-184 from 3025937 to 3075681 tid 3 2024-09-21T18:07:22.001 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: finishing write tid 2 to smithi16030472-175 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: finishing write tid 3 to smithi16030472-175 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: finishing write tid 4 to smithi16030472-175 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: finishing write tid 5 to smithi16030472-175 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: oid 175 updating version 0 to 107 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: oid 175 updating version 107 to 108 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: oid 175 version 108 is already newer than 106 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: oid 175 version 108 is already newer than 105 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 108 (ObjNum 174 snap 0 seq_num 174) dirty exists 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:175: left oid 175 (ObjNum 174 snap 0 seq_num 174) 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:178: finishing write tid 1 to smithi16030472-178 2024-09-21T18:07:22.002 INFO:tasks.rados.rados.0.smithi160.stdout:178: finishing write tid 2 to smithi16030472-178 2024-09-21T18:07:22.003 INFO:tasks.rados.rados.0.smithi160.stdout:178: finishing write tid 3 to smithi16030472-178 2024-09-21T18:07:22.003 INFO:tasks.rados.rados.0.smithi160.stdout:175: done (15 left) 2024-09-21T18:07:22.003 INFO:tasks.rados.rados.0.smithi160.stdout:185: write initial oid 185 2024-09-21T18:07:22.003 INFO:tasks.rados.rados.0.smithi160.stdout:185: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-21T18:07:22.010 INFO:tasks.rados.rados.0.smithi160.stdout:185: writing smithi16030472-185 from 692488 to 1330776 tid 1 2024-09-21T18:07:22.018 INFO:tasks.rados.rados.0.smithi160.stdout:185: writing smithi16030472-185 from 1917275 to 2523999 tid 2 2024-09-21T18:07:22.018 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.018 INFO:tasks.rados.rados.0.smithi160.stdout:173: finishing write tid 2 to smithi16030472-173 2024-09-21T18:07:22.018 INFO:tasks.rados.rados.0.smithi160.stdout:173: finishing write tid 3 to smithi16030472-173 2024-09-21T18:07:22.019 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.019 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.032 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:22.026+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:22.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:22.077 INFO:tasks.rados.rados.0.smithi160.stdout:158: finishing write tid 3 to smithi16030472-158 2024-09-21T18:07:22.077 INFO:tasks.rados.rados.0.smithi160.stdout:158: finishing write tid 4 to smithi16030472-158 2024-09-21T18:07:22.077 INFO:tasks.rados.rados.0.smithi160.stdout:158: oid 158 updating version 0 to 88 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:158: oid 158 updating version 88 to 89 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:158: oid 158 version 89 is already newer than 87 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 158 v 89 (ObjNum 157 snap 0 seq_num 157) dirty exists 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:158: left oid 158 (ObjNum 157 snap 0 seq_num 157) 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: finishing write tid 4 to smithi16030472-178 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: oid 178 updating version 0 to 110 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: oid 178 updating version 110 to 111 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: oid 178 version 111 is already newer than 109 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 178 v 111 (ObjNum 177 snap 0 seq_num 177) dirty exists 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: left oid 178 (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:158: done (15 left) 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:178: done (14 left) 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:186: write initial oid 186 2024-09-21T18:07:22.078 INFO:tasks.rados.rados.0.smithi160.stdout:186: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-21T18:07:22.085 INFO:tasks.rados.rados.0.smithi160.stdout:186: writing smithi16030472-186 from 709295 to 1422832 tid 1 2024-09-21T18:07:22.092 INFO:tasks.rados.rados.0.smithi160.stdout:186: writing smithi16030472-186 from 1859404 to 2609786 tid 2 2024-09-21T18:07:22.092 INFO:tasks.rados.rados.0.smithi160.stdout:187: write initial oid 187 2024-09-21T18:07:22.092 INFO:tasks.rados.rados.0.smithi160.stdout:187: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-21T18:07:22.100 INFO:tasks.rados.rados.0.smithi160.stdout:187: writing smithi16030472-187 from 726102 to 1514888 tid 1 2024-09-21T18:07:22.106 INFO:tasks.rados.rados.0.smithi160.stdout:187: writing smithi16030472-187 from 1917886 to 2528279 tid 2 2024-09-21T18:07:22.110 INFO:tasks.rados.rados.0.smithi160.stdout:187: writing smithi16030472-187 from 3108206 to 3126102 tid 3 2024-09-21T18:07:22.110 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: finishing write tid 4 to smithi16030472-173 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: finishing write tid 5 to smithi16030472-173 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: oid 173 updating version 0 to 106 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: oid 173 version 106 is already newer than 105 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: oid 173 updating version 106 to 107 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: oid 173 version 107 is already newer than 104 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 107 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-09-21T18:07:22.117 INFO:tasks.rados.rados.0.smithi160.stdout:173: left oid 173 (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:07:22.118 INFO:tasks.rados.rados.0.smithi160.stdout:173: done (15 left) 2024-09-21T18:07:22.118 INFO:tasks.rados.rados.0.smithi160.stdout:188: write initial oid 188 2024-09-21T18:07:22.118 INFO:tasks.rados.rados.0.smithi160.stdout:188: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-21T18:07:22.124 INFO:tasks.rados.rados.0.smithi160.stdout:188: writing smithi16030472-188 from 742909 to 1206944 tid 1 2024-09-21T18:07:22.133 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:22.127+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:22.135 INFO:tasks.rados.rados.0.smithi160.stdout:188: writing smithi16030472-188 from 1976368 to 2730419 tid 2 2024-09-21T18:07:22.135 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.135 INFO:tasks.rados.rados.0.smithi160.stdout:177: finishing write tid 2 to smithi16030472-177 2024-09-21T18:07:22.135 INFO:tasks.rados.rados.0.smithi160.stdout:177: finishing write tid 3 to smithi16030472-177 2024-09-21T18:07:22.210 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.210 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:22.234 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:22.229+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:22.236 INFO:tasks.rados.rados.0.smithi160.stdout:177: finishing write tid 4 to smithi16030472-177 2024-09-21T18:07:22.236 INFO:tasks.rados.rados.0.smithi160.stdout:177: oid 177 updating version 0 to 89 2024-09-21T18:07:22.236 INFO:tasks.rados.rados.0.smithi160.stdout:177: oid 177 version 89 is already newer than 88 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:177: oid 177 updating version 89 to 90 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 177 v 90 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:177: left oid 177 (ObjNum 176 snap 0 seq_num 176) 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:177: done (15 left) 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:189: write initial oid 189 2024-09-21T18:07:22.237 INFO:tasks.rados.rados.0.smithi160.stdout:189: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-21T18:07:22.243 INFO:tasks.rados.rados.0.smithi160.stdout:189: writing smithi16030472-189 from 759716 to 1299000 tid 1 2024-09-21T18:07:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:189: writing smithi16030472-189 from 2034850 to 2648912 tid 2 2024-09-21T18:07:22.251 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:180: finishing write tid 1 to smithi16030472-180 2024-09-21T18:07:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:180: finishing write tid 2 to smithi16030472-180 2024-09-21T18:07:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:180: finishing write tid 3 to smithi16030472-180 2024-09-21T18:07:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:180: finishing write tid 4 to smithi16030472-180 2024-09-21T18:07:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:179: finishing write tid 1 to smithi16030472-179 2024-09-21T18:07:22.335 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:22.330+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:22.361 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.361 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.375 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:22.435 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:22.433+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:22.528 INFO:tasks.rados.rados.0.smithi160.stdout:168: finishing write tid 4 to smithi16030472-168 2024-09-21T18:07:22.528 INFO:tasks.rados.rados.0.smithi160.stdout:168: finishing write tid 5 to smithi16030472-168 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:168: oid 168 updating version 0 to 125 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:168: oid 168 version 125 is already newer than 124 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:168: oid 168 updating version 125 to 126 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:168: oid 168 version 126 is already newer than 123 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 168 v 126 (ObjNum 167 snap 0 seq_num 167) dirty exists 2024-09-21T18:07:22.529 INFO:tasks.rados.rados.0.smithi160.stdout:168: left oid 168 (ObjNum 167 snap 0 seq_num 167) 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: finishing write tid 1 to smithi16030472-174 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: finishing write tid 2 to smithi16030472-174 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: finishing write tid 3 to smithi16030472-174 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: finishing write tid 4 to smithi16030472-174 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: oid 174 updating version 0 to 108 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: oid 174 updating version 108 to 109 2024-09-21T18:07:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:174: oid 174 updating version 109 to 110 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 174 v 110 (ObjNum 173 snap 0 seq_num 173) dirty exists 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:174: left oid 174 (ObjNum 173 snap 0 seq_num 173) 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:168: done (15 left) 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:174: done (14 left) 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:190: write initial oid 190 2024-09-21T18:07:22.531 INFO:tasks.rados.rados.0.smithi160.stdout:190: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-21T18:07:22.536 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:22.531+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:22.537 INFO:tasks.rados.rados.0.smithi160.stdout:190: writing smithi16030472-190 from 776523 to 1391056 tid 1 2024-09-21T18:07:22.539 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.539 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.547 INFO:tasks.rados.rados.0.smithi160.stdout:190: writing smithi16030472-190 from 1976979 to 2734699 tid 2 2024-09-21T18:07:22.547 INFO:tasks.rados.rados.0.smithi160.stdout:191: write initial oid 191 2024-09-21T18:07:22.547 INFO:tasks.rados.rados.0.smithi160.stdout:191: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-21T18:07:22.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:22.557 INFO:tasks.rados.rados.0.smithi160.stdout:191: writing smithi16030472-191 from 793330 to 1483112 tid 1 2024-09-21T18:07:22.565 INFO:tasks.rados.rados.0.smithi160.stdout:191: writing smithi16030472-191 from 2035461 to 2653192 tid 2 2024-09-21T18:07:22.568 INFO:tasks.rados.rados.0.smithi160.stdout:191: writing smithi16030472-191 from 3101545 to 3193330 tid 3 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:185: finishing write tid 1 to smithi16030472-185 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:185: finishing write tid 2 to smithi16030472-185 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: finishing write tid 1 to smithi16030472-170 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: finishing write tid 2 to smithi16030472-170 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: finishing write tid 3 to smithi16030472-170 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: finishing write tid 4 to smithi16030472-170 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: finishing write tid 5 to smithi16030472-170 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: oid 170 updating version 0 to 126 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: oid 170 updating version 126 to 128 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: oid 170 updating version 128 to 129 2024-09-21T18:07:22.569 INFO:tasks.rados.rados.0.smithi160.stdout:170: oid 170 version 129 is already newer than 127 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 170 v 129 (ObjNum 169 snap 0 seq_num 169) dirty exists 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:170: left oid 170 (ObjNum 169 snap 0 seq_num 169) 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:172: finishing write tid 1 to smithi16030472-172 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:172: finishing write tid 2 to smithi16030472-172 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:172: finishing write tid 3 to smithi16030472-172 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:172: finishing write tid 4 to smithi16030472-172 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:170: done (15 left) 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:192: write initial oid 192 2024-09-21T18:07:22.570 INFO:tasks.rados.rados.0.smithi160.stdout:192: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-21T18:07:22.574 INFO:tasks.rados.rados.0.smithi160.stdout:192: writing smithi16030472-192 from 410137 to 891521 tid 1 2024-09-21T18:07:22.583 INFO:tasks.rados.rados.0.smithi160.stdout:192: writing smithi16030472-192 from 1410296 to 2171685 tid 2 2024-09-21T18:07:22.590 INFO:tasks.rados.rados.0.smithi160.stdout:192: writing smithi16030472-192 from 2728968 to 3210137 tid 3 2024-09-21T18:07:22.590 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.590 INFO:tasks.rados.rados.0.smithi160.stdout:172: finishing write tid 5 to smithi16030472-172 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: oid 172 updating version 0 to 133 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: oid 172 version 133 is already newer than 130 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: oid 172 version 133 is already newer than 132 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: oid 172 version 133 is already newer than 131 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 172 v 133 (ObjNum 171 snap 0 seq_num 171) dirty exists 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: left oid 172 (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:176: finishing write tid 1 to smithi16030472-176 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:185: finishing write tid 3 to smithi16030472-185 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:172: done (15 left) 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:193: write initial oid 193 2024-09-21T18:07:22.591 INFO:tasks.rados.rados.0.smithi160.stdout:193: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-21T18:07:22.597 INFO:tasks.rados.rados.0.smithi160.stdout:193: writing smithi16030472-193 from 426944 to 983577 tid 1 2024-09-21T18:07:22.604 INFO:tasks.rados.rados.0.smithi160.stdout:193: writing smithi16030472-193 from 1468778 to 2090178 tid 2 2024-09-21T18:07:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:193: writing smithi16030472-193 from 2872744 to 3226944 tid 3 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:176: finishing write tid 2 to smithi16030472-176 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: finishing write tid 5 to smithi16030472-180 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: oid 180 updating version 0 to 92 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: oid 180 version 92 is already newer than 91 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: oid 180 updating version 92 to 93 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: oid 180 updating version 93 to 94 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 94 (ObjNum 179 snap 0 seq_num 179) dirty exists 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:180: left oid 180 (ObjNum 179 snap 0 seq_num 179) 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: finishing write tid 2 to smithi16030472-179 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: finishing write tid 3 to smithi16030472-179 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: finishing write tid 4 to smithi16030472-179 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: finishing write tid 5 to smithi16030472-179 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: oid 179 updating version 0 to 112 2024-09-21T18:07:22.612 INFO:tasks.rados.rados.0.smithi160.stdout:179: oid 179 updating version 112 to 114 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:179: oid 179 version 114 is already newer than 113 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:179: oid 179 updating version 114 to 115 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 115 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:179: left oid 179 (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:179: done (15 left) 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:180: done (14 left) 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:194: write initial oid 194 2024-09-21T18:07:22.613 INFO:tasks.rados.rados.0.smithi160.stdout:194: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-21T18:07:22.618 INFO:tasks.rados.rados.0.smithi160.stdout:194: writing smithi16030472-194 from 443751 to 1075633 tid 1 2024-09-21T18:07:22.626 INFO:tasks.rados.rados.0.smithi160.stdout:194: writing smithi16030472-194 from 1810907 to 2575965 tid 2 2024-09-21T18:07:22.630 INFO:tasks.rados.rados.0.smithi160.stdout:194: writing smithi16030472-194 from 3067461 to 3243751 tid 3 2024-09-21T18:07:22.631 INFO:tasks.rados.rados.0.smithi160.stdout:195: write initial oid 195 2024-09-21T18:07:22.631 INFO:tasks.rados.rados.0.smithi160.stdout:195: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-21T18:07:22.637 INFO:tasks.rados.rados.0.smithi160.stdout:195: writing smithi16030472-195 from 460558 to 1167689 tid 1 2024-09-21T18:07:22.637 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:22.631+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:22.644 INFO:tasks.rados.rados.0.smithi160.stdout:195: writing smithi16030472-195 from 1869389 to 2378105 tid 2 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:195: writing smithi16030472-195 from 3094884 to 3260558 tid 3 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: finishing write tid 1 to smithi16030472-183 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: finishing write tid 2 to smithi16030472-183 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: finishing write tid 3 to smithi16030472-183 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: finishing write tid 4 to smithi16030472-183 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: oid 183 updating version 0 to 118 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: oid 183 version 118 is already newer than 116 2024-09-21T18:07:22.649 INFO:tasks.rados.rados.0.smithi160.stdout:183: oid 183 version 118 is already newer than 117 2024-09-21T18:07:22.650 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 118 (ObjNum 182 snap 0 seq_num 182) dirty exists 2024-09-21T18:07:22.650 INFO:tasks.rados.rados.0.smithi160.stdout:183: left oid 183 (ObjNum 182 snap 0 seq_num 182) 2024-09-21T18:07:22.650 INFO:tasks.rados.rados.0.smithi160.stdout:183: done (15 left) 2024-09-21T18:07:22.650 INFO:tasks.rados.rados.0.smithi160.stdout:196: write initial oid 196 2024-09-21T18:07:22.650 INFO:tasks.rados.rados.0.smithi160.stdout:196: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-21T18:07:22.656 INFO:tasks.rados.rados.0.smithi160.stdout:196: writing smithi16030472-196 from 477365 to 1259745 tid 1 2024-09-21T18:07:22.664 INFO:tasks.rados.rados.0.smithi160.stdout:196: writing smithi16030472-196 from 1927871 to 2696598 tid 2 2024-09-21T18:07:22.667 INFO:tasks.rados.rados.0.smithi160.stdout:196: writing smithi16030472-196 from 3122307 to 3277365 tid 3 2024-09-21T18:07:22.667 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.693 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.693 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:22.738 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:22.733+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:22.750 INFO:tasks.rados.rados.0.smithi160.stdout:184: finishing write tid 1 to smithi16030472-184 2024-09-21T18:07:22.751 INFO:tasks.rados.rados.0.smithi160.stdout:184: finishing write tid 2 to smithi16030472-184 2024-09-21T18:07:22.751 INFO:tasks.rados.rados.0.smithi160.stdout:184: finishing write tid 3 to smithi16030472-184 2024-09-21T18:07:22.756 INFO:tasks.rados.rados.0.smithi160.stdout:185: finishing write tid 4 to smithi16030472-185 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:185: oid 185 updating version 0 to 111 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:185: oid 185 updating version 111 to 112 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:185: oid 185 updating version 112 to 113 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 185 v 113 (ObjNum 184 snap 0 seq_num 184) dirty exists 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:185: left oid 185 (ObjNum 184 snap 0 seq_num 184) 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:189: finishing write tid 1 to smithi16030472-189 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: finishing write tid 3 to smithi16030472-176 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: finishing write tid 4 to smithi16030472-176 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: oid 176 updating version 0 to 134 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: oid 176 updating version 134 to 135 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: oid 176 updating version 135 to 136 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 176 v 136 (ObjNum 175 snap 0 seq_num 175) dirty exists 2024-09-21T18:07:22.757 INFO:tasks.rados.rados.0.smithi160.stdout:176: left oid 176 (ObjNum 175 snap 0 seq_num 175) 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:181: finishing write tid 1 to smithi16030472-181 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:181: finishing write tid 2 to smithi16030472-181 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:181: finishing write tid 3 to smithi16030472-181 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:176: done (15 left) 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:185: done (14 left) 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:197: write initial oid 197 2024-09-21T18:07:22.758 INFO:tasks.rados.rados.0.smithi160.stdout:197: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-21T18:07:22.762 INFO:tasks.rados.rados.0.smithi160.stdout:197: writing smithi16030472-197 from 494172 to 951801 tid 1 2024-09-21T18:07:22.768 INFO:tasks.rados.rados.0.smithi160.stdout:197: writing smithi16030472-197 from 1586353 to 2098738 tid 2 2024-09-21T18:07:22.775 INFO:tasks.rados.rados.0.smithi160.stdout:197: writing smithi16030472-197 from 2749730 to 3294172 tid 3 2024-09-21T18:07:22.775 INFO:tasks.rados.rados.0.smithi160.stdout:198: write initial oid 198 2024-09-21T18:07:22.775 INFO:tasks.rados.rados.0.smithi160.stdout:198: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-21T18:07:22.781 INFO:tasks.rados.rados.0.smithi160.stdout:198: writing smithi16030472-198 from 510979 to 1043857 tid 1 2024-09-21T18:07:22.788 INFO:tasks.rados.rados.0.smithi160.stdout:198: writing smithi16030472-198 from 1528482 to 2300878 tid 2 2024-09-21T18:07:22.793 INFO:tasks.rados.rados.0.smithi160.stdout:198: writing smithi16030472-198 from 3060800 to 3310979 tid 3 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: finishing write tid 4 to smithi16030472-181 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: finishing write tid 5 to smithi16030472-181 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: oid 181 updating version 0 to 137 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: oid 181 updating version 137 to 140 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: oid 181 version 140 is already newer than 139 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: oid 181 version 140 is already newer than 138 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 181 v 140 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: left oid 181 (ObjNum 180 snap 0 seq_num 180) 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:181: done (15 left) 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:199: write initial oid 199 2024-09-21T18:07:22.794 INFO:tasks.rados.rados.0.smithi160.stdout:199: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-21T18:07:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:199: writing smithi16030472-199 from 527786 to 1252266 tid 1 2024-09-21T18:07:22.806 INFO:tasks.rados.rados.0.smithi160.stdout:199: writing smithi16030472-199 from 1703317 to 2219371 tid 2 2024-09-21T18:07:22.812 INFO:tasks.rados.rados.0.smithi160.stdout:199: writing smithi16030472-199 from 2804576 to 3239611 tid 3 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:190: finishing write tid 1 to smithi16030472-190 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: finishing write tid 1 to smithi16030472-182 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: finishing write tid 2 to smithi16030472-182 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: finishing write tid 3 to smithi16030472-182 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: finishing write tid 4 to smithi16030472-182 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: finishing write tid 5 to smithi16030472-182 2024-09-21T18:07:22.813 INFO:tasks.rados.rados.0.smithi160.stdout:182: oid 182 updating version 0 to 98 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:182: oid 182 version 98 is already newer than 96 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:182: oid 182 version 98 is already newer than 97 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:182: oid 182 version 98 is already newer than 95 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 182 v 98 (ObjNum 181 snap 0 seq_num 181) dirty exists 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:182: left oid 182 (ObjNum 181 snap 0 seq_num 181) 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:182: done (15 left) 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:200: write initial oid 200 2024-09-21T18:07:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:200: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-21T18:07:22.819 INFO:tasks.rados.rados.0.smithi160.stdout:200: writing smithi16030472-200 from 544593 to 1344322 tid 1 2024-09-21T18:07:22.826 INFO:tasks.rados.rados.0.smithi160.stdout:200: writing smithi16030472-200 from 1761799 to 2537864 tid 2 2024-09-21T18:07:22.829 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.829 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.830 INFO:tasks.rados.rados.0.smithi160.stdout:200: writing smithi16030472-200 from 2948352 to 3344593 tid 3 2024-09-21T18:07:22.830 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.839 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:22.837+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:22.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:22.855 INFO:tasks.rados.rados.0.smithi160.stdout:184: finishing write tid 4 to smithi16030472-184 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: finishing write tid 5 to smithi16030472-184 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: oid 184 updating version 0 to 120 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: oid 184 version 120 is already newer than 119 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: oid 184 updating version 120 to 121 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: oid 184 updating version 121 to 122 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 184 v 122 (ObjNum 183 snap 0 seq_num 183) dirty exists 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: left oid 184 (ObjNum 183 snap 0 seq_num 183) 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:186: finishing write tid 1 to smithi16030472-186 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:186: finishing write tid 2 to smithi16030472-186 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:184: done (15 left) 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:201: write initial oid 201 2024-09-21T18:07:22.856 INFO:tasks.rados.rados.0.smithi160.stdout:201: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-21T18:07:22.861 INFO:tasks.rados.rados.0.smithi160.stdout:201: writing smithi16030472-201 from 561400 to 1036378 tid 1 2024-09-21T18:07:22.868 INFO:tasks.rados.rados.0.smithi160.stdout:201: writing smithi16030472-201 from 1820281 to 2340004 tid 2 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout:201: writing smithi16030472-201 from 2859422 to 3317001 tid 3 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout:186: finishing write tid 3 to smithi16030472-186 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout:186: finishing write tid 4 to smithi16030472-186 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout:186: oid 186 updating version 0 to 124 2024-09-21T18:07:22.874 INFO:tasks.rados.rados.0.smithi160.stdout:186: oid 186 version 124 is already newer than 123 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:186: oid 186 updating version 124 to 125 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 186 v 125 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:186: left oid 186 (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:186: done (15 left) 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:202: write initial oid 202 2024-09-21T18:07:22.875 INFO:tasks.rados.rados.0.smithi160.stdout:202: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-21T18:07:22.880 INFO:tasks.rados.rados.0.smithi160.stdout:202: writing smithi16030472-202 from 578207 to 1128434 tid 1 2024-09-21T18:07:22.888 INFO:tasks.rados.rados.0.smithi160.stdout:202: writing smithi16030472-202 from 1762410 to 2542144 tid 2 2024-09-21T18:07:22.893 INFO:tasks.rados.rados.0.smithi160.stdout:202: writing smithi16030472-202 from 3286845 to 3378207 tid 3 2024-09-21T18:07:22.893 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:22.908 INFO:tasks.rados.rados.0.smithi160.stdout:187: finishing write tid 1 to smithi16030472-187 2024-09-21T18:07:22.939 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:22.937+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:22.941 INFO:tasks.rados.rados.0.smithi160.stdout:191: finishing write tid 1 to smithi16030472-191 2024-09-21T18:07:22.958 INFO:tasks.rados.rados.0.smithi160.stdout:187: finishing write tid 2 to smithi16030472-187 2024-09-21T18:07:22.958 INFO:tasks.rados.rados.0.smithi160.stdout:187: finishing write tid 3 to smithi16030472-187 2024-09-21T18:07:22.959 INFO:tasks.rados.rados.0.smithi160.stdout:188: finishing write tid 1 to smithi16030472-188 2024-09-21T18:07:22.971 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:22.971 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:22.978 INFO:tasks.rados.rados.0.smithi160.stdout:195: finishing write tid 1 to smithi16030472-195 2024-09-21T18:07:22.978 INFO:tasks.rados.rados.0.smithi160.stdout:195: finishing write tid 2 to smithi16030472-195 2024-09-21T18:07:22.978 INFO:tasks.rados.rados.0.smithi160.stdout:198: finishing write tid 1 to smithi16030472-198 2024-09-21T18:07:22.978 INFO:tasks.rados.rados.0.smithi160.stdout:198: finishing write tid 2 to smithi16030472-198 2024-09-21T18:07:22.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: finishing write tid 2 to smithi16030472-190 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: finishing write tid 3 to smithi16030472-190 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: finishing write tid 4 to smithi16030472-190 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: oid 190 updating version 0 to 141 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: oid 190 updating version 141 to 142 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: oid 190 updating version 142 to 143 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 190 v 143 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: left oid 190 (ObjNum 189 snap 0 seq_num 189) 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:190: done (15 left) 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:203: write initial oid 203 2024-09-21T18:07:23.012 INFO:tasks.rados.rados.0.smithi160.stdout:203: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-21T18:07:23.018 INFO:tasks.rados.rados.0.smithi160.stdout:203: writing smithi16030472-203 from 595014 to 1220490 tid 1 2024-09-21T18:07:23.025 INFO:tasks.rados.rados.0.smithi160.stdout:203: writing smithi16030472-203 from 1820892 to 2344284 tid 2 2024-09-21T18:07:23.030 INFO:tasks.rados.rados.0.smithi160.stdout:203: writing smithi16030472-203 from 2797915 to 3394391 tid 3 2024-09-21T18:07:23.031 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.040 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:23.034+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:23.059 INFO:tasks.rados.rados.0.smithi160.stdout:188: finishing write tid 2 to smithi16030472-188 2024-09-21T18:07:23.068 INFO:tasks.rados.rados.0.smithi160.stdout:198: finishing write tid 3 to smithi16030472-198 2024-09-21T18:07:23.069 INFO:tasks.rados.rados.0.smithi160.stdout:198: finishing write tid 4 to smithi16030472-198 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: finishing write tid 5 to smithi16030472-198 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: oid 198 updating version 0 to 129 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: oid 198 updating version 129 to 130 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: oid 198 version 130 is already newer than 128 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: oid 198 updating version 130 to 131 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 198 v 131 (ObjNum 197 snap 0 seq_num 197) dirty exists 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: left oid 198 (ObjNum 197 snap 0 seq_num 197) 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:198: done (15 left) 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:204: write initial oid 204 2024-09-21T18:07:23.070 INFO:tasks.rados.rados.0.smithi160.stdout:204: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-21T18:07:23.078 INFO:tasks.rados.rados.0.smithi160.stdout:204: writing smithi16030472-204 from 611821 to 1312546 tid 1 2024-09-21T18:07:23.086 INFO:tasks.rados.rados.0.smithi160.stdout:204: writing smithi16030472-204 from 1879374 to 2662777 tid 2 2024-09-21T18:07:23.091 INFO:tasks.rados.rados.0.smithi160.stdout:204: writing smithi16030472-204 from 3341691 to 3411821 tid 3 2024-09-21T18:07:23.091 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.096 INFO:tasks.rados.rados.0.smithi160.stdout:197: finishing write tid 1 to smithi16030472-197 2024-09-21T18:07:23.096 INFO:tasks.rados.rados.0.smithi160.stdout:197: finishing write tid 2 to smithi16030472-197 2024-09-21T18:07:23.141 INFO:tasks.rados.rados.0.smithi160.stdout:191: finishing write tid 2 to smithi16030472-191 2024-09-21T18:07:23.141 INFO:tasks.rados.rados.0.smithi160.stdout:191: finishing write tid 3 to smithi16030472-191 2024-09-21T18:07:23.141 INFO:tasks.rados.rados.0.smithi160.stdout:191: finishing write tid 4 to smithi16030472-191 2024-09-21T18:07:23.141 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:23.135+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:189: finishing write tid 2 to smithi16030472-189 2024-09-21T18:07:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:189: finishing write tid 3 to smithi16030472-189 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: finishing write tid 4 to smithi16030472-189 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: oid 189 updating version 0 to 116 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: oid 189 version 116 is already newer than 115 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: oid 189 version 116 is already newer than 114 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 189 v 116 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: left oid 189 (ObjNum 188 snap 0 seq_num 188) 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:189: done (15 left) 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:205: write initial oid 205 2024-09-21T18:07:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:205: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-21T18:07:23.154 INFO:tasks.rados.rados.0.smithi160.stdout:205: writing smithi16030472-205 from 628628 to 1404602 tid 1 2024-09-21T18:07:23.161 INFO:tasks.rados.rados.0.smithi160.stdout:205: writing smithi16030472-205 from 1937856 to 2464917 tid 2 2024-09-21T18:07:23.163 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.163 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.168 INFO:tasks.rados.rados.0.smithi160.stdout:205: writing smithi16030472-205 from 3252761 to 3428628 tid 3 2024-09-21T18:07:23.168 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:23.242 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:23.237+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: finishing write tid 3 to smithi16030472-195 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: finishing write tid 4 to smithi16030472-195 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: finishing write tid 5 to smithi16030472-195 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: oid 195 updating version 0 to 127 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: oid 195 updating version 127 to 128 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: oid 195 version 128 is already newer than 126 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: oid 195 updating version 128 to 129 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 129 (ObjNum 194 snap 0 seq_num 194) dirty exists 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: left oid 195 (ObjNum 194 snap 0 seq_num 194) 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:195: done (15 left) 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:206: write initial oid 206 2024-09-21T18:07:23.269 INFO:tasks.rados.rados.0.smithi160.stdout:206: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-21T18:07:23.279 INFO:tasks.rados.rados.0.smithi160.stdout:206: writing smithi16030472-206 from 645435 to 1096658 tid 1 2024-09-21T18:07:23.291 INFO:tasks.rados.rados.0.smithi160.stdout:206: writing smithi16030472-206 from 1879985 to 2667057 tid 2 2024-09-21T18:07:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:206: writing smithi16030472-206 from 3280184 to 3445435 tid 3 2024-09-21T18:07:23.297 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.298 INFO:tasks.rados.rados.0.smithi160.stdout:191: finishing write tid 5 to smithi16030472-191 2024-09-21T18:07:23.298 INFO:tasks.rados.rados.0.smithi160.stdout:191: oid 191 updating version 0 to 117 2024-09-21T18:07:23.298 INFO:tasks.rados.rados.0.smithi160.stdout:191: oid 191 updating version 117 to 118 2024-09-21T18:07:23.298 INFO:tasks.rados.rados.0.smithi160.stdout:191: oid 191 version 118 is already newer than 115 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:191: oid 191 version 118 is already newer than 116 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 191 v 118 (ObjNum 190 snap 0 seq_num 190) dirty exists 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:191: left oid 191 (ObjNum 190 snap 0 seq_num 190) 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: finishing write tid 3 to smithi16030472-197 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: finishing write tid 4 to smithi16030472-197 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: finishing write tid 5 to smithi16030472-197 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: oid 197 updating version 0 to 145 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: oid 197 version 145 is already newer than 144 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: oid 197 updating version 145 to 147 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: oid 197 version 147 is already newer than 146 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 147 (ObjNum 196 snap 0 seq_num 196) dirty exists 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:197: left oid 197 (ObjNum 196 snap 0 seq_num 196) 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:200: finishing write tid 1 to smithi16030472-200 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:200: finishing write tid 2 to smithi16030472-200 2024-09-21T18:07:23.299 INFO:tasks.rados.rados.0.smithi160.stdout:191: done (15 left) 2024-09-21T18:07:23.300 INFO:tasks.rados.rados.0.smithi160.stdout:197: done (14 left) 2024-09-21T18:07:23.300 INFO:tasks.rados.rados.0.smithi160.stdout:207: write initial oid 207 2024-09-21T18:07:23.300 INFO:tasks.rados.rados.0.smithi160.stdout:207: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-21T18:07:23.302 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.302 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.309 INFO:tasks.rados.rados.0.smithi160.stdout:207: writing smithi16030472-207 from 662242 to 1305067 tid 1 2024-09-21T18:07:23.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:23.319 INFO:tasks.rados.rados.0.smithi160.stdout:207: writing smithi16030472-207 from 2054820 to 2585550 tid 2 2024-09-21T18:07:23.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:07:23.326 INFO:tasks.rados.rados.0.smithi160.stdout:207: writing smithi16030472-207 from 3307607 to 3462242 tid 3 2024-09-21T18:07:23.327 INFO:tasks.rados.rados.0.smithi160.stdout:208: write initial oid 208 2024-09-21T18:07:23.327 INFO:tasks.rados.rados.0.smithi160.stdout:208: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-21T18:07:23.337 INFO:tasks.rados.rados.0.smithi160.stdout:208: writing smithi16030472-208 from 679049 to 1397123 tid 1 2024-09-21T18:07:23.347 INFO:tasks.rados.rados.0.smithi160.stdout:208: writing smithi16030472-208 from 2113302 to 2787690 tid 2 2024-09-21T18:07:23.351 INFO:tasks.rados.rados.0.smithi160.stdout:208: writing smithi16030472-208 from 3335030 to 3479049 tid 3 2024-09-21T18:07:23.351 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.351 INFO:tasks.rados.rados.0.smithi160.stdout:187: finishing write tid 4 to smithi16030472-187 2024-09-21T18:07:23.351 INFO:tasks.rados.rados.0.smithi160.stdout:187: finishing write tid 5 to smithi16030472-187 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: oid 187 updating version 0 to 100 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: oid 187 version 100 is already newer than 99 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: oid 187 updating version 100 to 101 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: oid 187 updating version 101 to 102 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 102 (ObjNum 186 snap 0 seq_num 186) dirty exists 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: left oid 187 (ObjNum 186 snap 0 seq_num 186) 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: finishing write tid 3 to smithi16030472-188 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: finishing write tid 4 to smithi16030472-188 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: oid 188 updating version 0 to 103 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: oid 188 updating version 103 to 104 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: oid 188 version 104 is already newer than 102 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 188 v 104 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:188: left oid 188 (ObjNum 187 snap 0 seq_num 187) 2024-09-21T18:07:23.352 INFO:tasks.rados.rados.0.smithi160.stdout:187: done (15 left) 2024-09-21T18:07:23.353 INFO:tasks.rados.rados.0.smithi160.stdout:188: done (14 left) 2024-09-21T18:07:23.353 INFO:tasks.rados.rados.0.smithi160.stdout:209: write initial oid 209 2024-09-21T18:07:23.353 INFO:tasks.rados.rados.0.smithi160.stdout:209: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-21T18:07:23.360 INFO:tasks.rados.rados.0.smithi160.stdout:209: writing smithi16030472-209 from 695856 to 1489179 tid 1 2024-09-21T18:07:23.362 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:23.361+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:23.367 INFO:tasks.rados.rados.0.smithi160.stdout:209: writing smithi16030472-209 from 2171784 to 2706183 tid 2 2024-09-21T18:07:23.373 INFO:tasks.rados.rados.0.smithi160.stdout:209: writing smithi16030472-209 from 3362453 to 3495856 tid 3 2024-09-21T18:07:23.373 INFO:tasks.rados.rados.0.smithi160.stdout:210: write initial oid 210 2024-09-21T18:07:23.373 INFO:tasks.rados.rados.0.smithi160.stdout:210: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-21T18:07:23.380 INFO:tasks.rados.rados.0.smithi160.stdout:210: writing smithi16030472-210 from 712663 to 1181235 tid 1 2024-09-21T18:07:23.387 INFO:tasks.rados.rados.0.smithi160.stdout:210: writing smithi16030472-210 from 1713913 to 2391970 tid 2 2024-09-21T18:07:23.393 INFO:tasks.rados.rados.0.smithi160.stdout:210: writing smithi16030472-210 from 2873523 to 3465256 tid 3 2024-09-21T18:07:23.393 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.393 INFO:tasks.rados.rados.0.smithi160.stdout:205: finishing write tid 1 to smithi16030472-205 2024-09-21T18:07:23.443 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:23.441+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.449 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:23.544 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:23.538+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: finishing write tid 2 to smithi16030472-205 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: finishing write tid 3 to smithi16030472-205 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: finishing write tid 4 to smithi16030472-205 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: finishing write tid 5 to smithi16030472-205 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: oid 205 updating version 0 to 134 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: oid 205 version 134 is already newer than 132 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: oid 205 version 134 is already newer than 133 2024-09-21T18:07:23.577 INFO:tasks.rados.rados.0.smithi160.stdout:205: oid 205 updating version 134 to 135 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 205 v 135 (ObjNum 204 snap 0 seq_num 204) dirty exists 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:205: left oid 205 (ObjNum 204 snap 0 seq_num 204) 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:209: finishing write tid 1 to smithi16030472-209 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:205: done (15 left) 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:211: write initial oid 211 2024-09-21T18:07:23.578 INFO:tasks.rados.rados.0.smithi160.stdout:211: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-21T18:07:23.584 INFO:tasks.rados.rados.0.smithi160.stdout:211: writing smithi16030472-211 from 729470 to 1273291 tid 1 2024-09-21T18:07:23.590 INFO:tasks.rados.rados.0.smithi160.stdout:211: writing smithi16030472-211 from 1772395 to 2310463 tid 2 2024-09-21T18:07:23.595 INFO:tasks.rados.rados.0.smithi160.stdout:211: writing smithi16030472-211 from 2900946 to 3303951 tid 3 2024-09-21T18:07:23.595 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.595 INFO:tasks.rados.rados.0.smithi160.stdout:209: finishing write tid 2 to smithi16030472-209 2024-09-21T18:07:23.603 INFO:tasks.rados.rados.0.smithi160.stdout:200: finishing write tid 3 to smithi16030472-200 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: finishing write tid 4 to smithi16030472-200 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: finishing write tid 5 to smithi16030472-200 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: oid 200 updating version 0 to 148 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: oid 200 updating version 148 to 150 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: oid 200 updating version 150 to 151 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: oid 200 version 151 is already newer than 149 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 200 v 151 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: left oid 200 (ObjNum 199 snap 0 seq_num 199) 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:200: done (15 left) 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:212: write initial oid 212 2024-09-21T18:07:23.604 INFO:tasks.rados.rados.0.smithi160.stdout:212: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-21T18:07:23.612 INFO:tasks.rados.rados.0.smithi160.stdout:212: writing smithi16030472-212 from 746277 to 1365347 tid 1 2024-09-21T18:07:23.619 INFO:tasks.rados.rados.0.smithi160.stdout:212: writing smithi16030472-212 from 1830877 to 2512603 tid 2 2024-09-21T18:07:23.625 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.625 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:212: writing smithi16030472-212 from 2928369 to 3546277 tid 3 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: finishing write tid 1 to smithi16030472-192 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: finishing write tid 2 to smithi16030472-192 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: finishing write tid 3 to smithi16030472-192 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: finishing write tid 4 to smithi16030472-192 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: finishing write tid 5 to smithi16030472-192 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: oid 192 updating version 0 to 122 2024-09-21T18:07:23.626 INFO:tasks.rados.rados.0.smithi160.stdout:192: oid 192 version 122 is already newer than 120 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:192: oid 192 version 122 is already newer than 121 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:192: oid 192 version 122 is already newer than 119 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 192 v 122 (ObjNum 191 snap 0 seq_num 191) dirty exists 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:192: left oid 192 (ObjNum 191 snap 0 seq_num 191) 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:193: finishing write tid 1 to smithi16030472-193 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:193: finishing write tid 2 to smithi16030472-193 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:202: finishing write tid 1 to smithi16030472-202 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:202: finishing write tid 2 to smithi16030472-202 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:209: finishing write tid 3 to smithi16030472-209 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:209: finishing write tid 4 to smithi16030472-209 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:192: done (15 left) 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:213: write initial oid 213 2024-09-21T18:07:23.627 INFO:tasks.rados.rados.0.smithi160.stdout:213: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-21T18:07:23.635 INFO:tasks.rados.rados.0.smithi160.stdout:213: writing smithi16030472-213 from 763084 to 1457403 tid 1 2024-09-21T18:07:23.641 INFO:tasks.rados.rados.0.smithi160.stdout:213: writing smithi16030472-213 from 1889359 to 2431096 tid 2 2024-09-21T18:07:23.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:23.645 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:23.639+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:23.648 INFO:tasks.rados.rados.0.smithi160.stdout:213: writing smithi16030472-213 from 2955792 to 3497694 tid 3 2024-09-21T18:07:23.648 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.666 INFO:tasks.rados.rados.0.smithi160.stdout:199: finishing write tid 1 to smithi16030472-199 2024-09-21T18:07:23.667 INFO:tasks.rados.rados.0.smithi160.stdout:199: finishing write tid 2 to smithi16030472-199 2024-09-21T18:07:23.667 INFO:tasks.rados.rados.0.smithi160.stdout:204: finishing write tid 1 to smithi16030472-204 2024-09-21T18:07:23.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:23.746 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:23.742+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:23.765 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.765 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:23.807 INFO:tasks.rados.rados.0.smithi160.stdout:193: finishing write tid 3 to smithi16030472-193 2024-09-21T18:07:23.807 INFO:tasks.rados.rados.0.smithi160.stdout:193: finishing write tid 4 to smithi16030472-193 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: finishing write tid 5 to smithi16030472-193 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: oid 193 updating version 0 to 124 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: oid 193 updating version 124 to 126 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: oid 193 version 126 is already newer than 123 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: oid 193 version 126 is already newer than 125 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 193 v 126 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: left oid 193 (ObjNum 192 snap 0 seq_num 192) 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:194: finishing write tid 1 to smithi16030472-194 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:193: done (15 left) 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:214: write initial oid 214 2024-09-21T18:07:23.808 INFO:tasks.rados.rados.0.smithi160.stdout:214: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-21T18:07:23.815 INFO:tasks.rados.rados.0.smithi160.stdout:214: writing smithi16030472-214 from 779891 to 1265812 tid 1 2024-09-21T18:07:23.823 INFO:tasks.rados.rados.0.smithi160.stdout:214: writing smithi16030472-214 from 1947841 to 2633236 tid 2 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:214: writing smithi16030472-214 from 3383215 to 3579891 tid 3 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: finishing write tid 2 to smithi16030472-194 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: finishing write tid 3 to smithi16030472-194 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: finishing write tid 4 to smithi16030472-194 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: finishing write tid 5 to smithi16030472-194 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: oid 194 updating version 0 to 130 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: oid 194 version 130 is already newer than 128 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: oid 194 version 130 is already newer than 127 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: oid 194 version 130 is already newer than 129 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 194 v 130 (ObjNum 193 snap 0 seq_num 193) dirty exists 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:194: left oid 194 (ObjNum 193 snap 0 seq_num 193) 2024-09-21T18:07:23.830 INFO:tasks.rados.rados.0.smithi160.stdout:202: finishing write tid 3 to smithi16030472-202 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: finishing write tid 4 to smithi16030472-202 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: finishing write tid 5 to smithi16030472-202 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: oid 202 updating version 0 to 155 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: oid 202 version 155 is already newer than 152 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: oid 202 version 155 is already newer than 153 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: oid 202 version 155 is already newer than 154 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 202 v 155 (ObjNum 201 snap 0 seq_num 201) dirty exists 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: left oid 202 (ObjNum 201 snap 0 seq_num 201) 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:206: finishing write tid 1 to smithi16030472-206 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:206: finishing write tid 2 to smithi16030472-206 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:206: finishing write tid 3 to smithi16030472-206 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:206: finishing write tid 4 to smithi16030472-206 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:194: done (15 left) 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:202: done (14 left) 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:215: write initial oid 215 2024-09-21T18:07:23.831 INFO:tasks.rados.rados.0.smithi160.stdout:215: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-21T18:07:23.840 INFO:tasks.rados.rados.0.smithi160.stdout:215: writing smithi16030472-215 from 796698 to 1357868 tid 1 2024-09-21T18:07:23.840 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:23.840 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:23.847 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:23.841+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:23.850 INFO:tasks.rados.rados.0.smithi160.stdout:215: writing smithi16030472-215 from 2006323 to 2551729 tid 2 2024-09-21T18:07:23.858 INFO:tasks.rados.rados.0.smithi160.stdout:215: writing smithi16030472-215 from 3126991 to 3596698 tid 3 2024-09-21T18:07:23.858 INFO:tasks.rados.rados.0.smithi160.stdout:216: write initial oid 216 2024-09-21T18:07:23.858 INFO:tasks.rados.rados.0.smithi160.stdout:216: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-21T18:07:23.860 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:23.866 INFO:tasks.rados.rados.0.smithi160.stdout:216: writing smithi16030472-216 from 413505 to 1049924 tid 1 2024-09-21T18:07:23.875 INFO:tasks.rados.rados.0.smithi160.stdout:216: writing smithi16030472-216 from 1664805 to 2353869 tid 2 2024-09-21T18:07:23.882 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:23.882 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:216: writing smithi16030472-216 from 3038061 to 3530132 tid 3 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: finishing write tid 5 to smithi16030472-209 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: oid 209 updating version 0 to 139 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: oid 209 version 139 is already newer than 136 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: oid 209 version 139 is already newer than 138 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: oid 209 version 139 is already newer than 137 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 209 v 139 (ObjNum 208 snap 0 seq_num 208) dirty exists 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: left oid 209 (ObjNum 208 snap 0 seq_num 208) 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:209: done (15 left) 2024-09-21T18:07:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:217: write initial oid 217 2024-09-21T18:07:23.884 INFO:tasks.rados.rados.0.smithi160.stdout:217: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-21T18:07:23.889 INFO:tasks.rados.rados.0.smithi160.stdout:217: writing smithi16030472-217 from 430312 to 1141980 tid 1 2024-09-21T18:07:23.895 INFO:tasks.rados.rados.0.smithi160.stdout:217: writing smithi16030472-217 from 1723287 to 2272362 tid 2 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:217: writing smithi16030472-217 from 2781837 to 3485180 tid 3 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: finishing write tid 3 to smithi16030472-199 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: finishing write tid 4 to smithi16030472-199 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: finishing write tid 5 to smithi16030472-199 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: oid 199 updating version 0 to 106 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: oid 199 updating version 106 to 107 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: oid 199 version 107 is already newer than 105 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: oid 199 version 107 is already newer than 104 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 107 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:07:23.903 INFO:tasks.rados.rados.0.smithi160.stdout:199: left oid 199 (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:07:23.904 INFO:tasks.rados.rados.0.smithi160.stdout:199: done (15 left) 2024-09-21T18:07:23.904 INFO:tasks.rados.rados.0.smithi160.stdout:218: write initial oid 218 2024-09-21T18:07:23.904 INFO:tasks.rados.rados.0.smithi160.stdout:218: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-21T18:07:23.904 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:23.904 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:23.904 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:23.905 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:23.910 INFO:tasks.rados.rados.0.smithi160.stdout:218: writing smithi16030472-218 from 447119 to 1234036 tid 1 2024-09-21T18:07:23.917 INFO:tasks.rados.rados.0.smithi160.stdout:218: writing smithi16030472-218 from 1665416 to 2358149 tid 2 2024-09-21T18:07:23.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:23.920 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:23.923 INFO:tasks.rados.rados.0.smithi160.stdout:218: writing smithi16030472-218 from 2976554 to 3491169 tid 3 2024-09-21T18:07:23.923 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:23.947 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:23.942+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:23.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:23.983 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:24.000 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: finishing write tid 1 to smithi16030472-196 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: finishing write tid 2 to smithi16030472-196 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: finishing write tid 3 to smithi16030472-196 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: finishing write tid 4 to smithi16030472-196 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: finishing write tid 5 to smithi16030472-196 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: oid 196 updating version 0 to 132 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: oid 196 updating version 132 to 134 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: oid 196 version 134 is already newer than 131 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: oid 196 version 134 is already newer than 133 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 196 v 134 (ObjNum 195 snap 0 seq_num 195) dirty exists 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:196: left oid 196 (ObjNum 195 snap 0 seq_num 195) 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:201: finishing write tid 1 to smithi16030472-201 2024-09-21T18:07:24.015 INFO:tasks.rados.rados.0.smithi160.stdout:201: finishing write tid 2 to smithi16030472-201 2024-09-21T18:07:24.016 INFO:tasks.rados.rados.0.smithi160.stdout:196: done (15 left) 2024-09-21T18:07:24.016 INFO:tasks.rados.rados.0.smithi160.stdout:219: write initial oid 219 2024-09-21T18:07:24.016 INFO:tasks.rados.rados.0.smithi160.stdout:219: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-21T18:07:24.020 INFO:tasks.rados.rados.0.smithi160.stdout:219: writing smithi16030472-219 from 463926 to 926092 tid 1 2024-09-21T18:07:24.023 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:24.026 INFO:tasks.rados.rados.0.smithi160.stdout:219: writing smithi16030472-219 from 1723898 to 2160289 tid 2 2024-09-21T18:07:24.033 INFO:tasks.rados.rados.0.smithi160.stdout:219: writing smithi16030472-219 from 2603977 to 3329864 tid 3 2024-09-21T18:07:24.033 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.033 INFO:tasks.rados.rados.0.smithi160.stdout:201: finishing write tid 3 to smithi16030472-201 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: finishing write tid 4 to smithi16030472-201 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: finishing write tid 5 to smithi16030472-201 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: oid 201 updating version 0 to 138 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: oid 201 version 138 is already newer than 137 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: oid 201 version 138 is already newer than 136 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: oid 201 version 138 is already newer than 135 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 201 v 138 (ObjNum 200 snap 0 seq_num 200) dirty exists 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:201: left oid 201 (ObjNum 200 snap 0 seq_num 200) 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: finishing write tid 1 to smithi16030472-203 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: finishing write tid 2 to smithi16030472-203 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: finishing write tid 3 to smithi16030472-203 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: finishing write tid 4 to smithi16030472-203 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: finishing write tid 5 to smithi16030472-203 2024-09-21T18:07:24.034 INFO:tasks.rados.rados.0.smithi160.stdout:203: oid 203 updating version 0 to 139 2024-09-21T18:07:24.035 INFO:tasks.rados.rados.0.smithi160.stdout:203: oid 203 updating version 139 to 141 2024-09-21T18:07:24.035 INFO:tasks.rados.rados.0.smithi160.stdout:203: oid 203 updating version 141 to 142 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:203: oid 203 version 142 is already newer than 140 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 142 (ObjNum 202 snap 0 seq_num 202) dirty exists 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:203: left oid 203 (ObjNum 202 snap 0 seq_num 202) 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:208: finishing write tid 1 to smithi16030472-208 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:201: done (15 left) 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:203: done (14 left) 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:220: write initial oid 220 2024-09-21T18:07:24.036 INFO:tasks.rados.rados.0.smithi160.stdout:220: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-21T18:07:24.037 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:24.038 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:24.039 INFO:tasks.rados.rados.0.smithi160.stdout:220: writing smithi16030472-220 from 480733 to 1018148 tid 1 2024-09-21T18:07:24.044 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:24.044 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:24.048 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:24.045+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:24.049 INFO:tasks.rados.rados.0.smithi160.stdout:220: writing smithi16030472-220 from 1782380 to 2478782 tid 2 2024-09-21T18:07:24.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:24.056 INFO:tasks.rados.rados.0.smithi160.stdout:220: writing smithi16030472-220 from 3031400 to 3568559 tid 3 2024-09-21T18:07:24.057 INFO:tasks.rados.rados.0.smithi160.stdout:221: write initial oid 221 2024-09-21T18:07:24.057 INFO:tasks.rados.rados.0.smithi160.stdout:221: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-21T18:07:24.064 INFO:tasks.rados.rados.0.smithi160.stdout:221: writing smithi16030472-221 from 497540 to 1110204 tid 1 2024-09-21T18:07:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:221: writing smithi16030472-221 from 1840862 to 2280922 tid 2 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:221: writing smithi16030472-221 from 3058823 to 3523607 tid 3 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: finishing write tid 5 to smithi16030472-206 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: oid 206 updating version 0 to 157 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: oid 206 updating version 157 to 158 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: oid 206 updating version 158 to 159 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: oid 206 version 159 is already newer than 156 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 206 v 159 (ObjNum 205 snap 0 seq_num 205) dirty exists 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: left oid 206 (ObjNum 205 snap 0 seq_num 205) 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:206: done (15 left) 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:222: write initial oid 222 2024-09-21T18:07:24.079 INFO:tasks.rados.rados.0.smithi160.stdout:222: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-21T18:07:24.084 INFO:tasks.rados.rados.0.smithi160.stdout:222: writing smithi16030472-222 from 514347 to 918613 tid 1 2024-09-21T18:07:24.091 INFO:tasks.rados.rados.0.smithi160.stdout:222: writing smithi16030472-222 from 1499344 to 2199415 tid 2 2024-09-21T18:07:24.097 INFO:tasks.rados.rados.0.smithi160.stdout:222: writing smithi16030472-222 from 2686246 to 3362302 tid 3 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:204: finishing write tid 2 to smithi16030472-204 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: finishing write tid 1 to smithi16030472-210 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: finishing write tid 2 to smithi16030472-210 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: finishing write tid 3 to smithi16030472-210 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: finishing write tid 4 to smithi16030472-210 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: finishing write tid 5 to smithi16030472-210 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: oid 210 updating version 0 to 108 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: oid 210 updating version 108 to 109 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: oid 210 updating version 109 to 111 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:210: oid 210 version 111 is already newer than 110 2024-09-21T18:07:24.098 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 111 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-21T18:07:24.099 INFO:tasks.rados.rados.0.smithi160.stdout:210: left oid 210 (ObjNum 209 snap 0 seq_num 209) 2024-09-21T18:07:24.099 INFO:tasks.rados.rados.0.smithi160.stdout:210: done (15 left) 2024-09-21T18:07:24.099 INFO:tasks.rados.rados.0.smithi160.stdout:223: write initial oid 223 2024-09-21T18:07:24.099 INFO:tasks.rados.rados.0.smithi160.stdout:223: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-21T18:07:24.103 INFO:tasks.rados.rados.0.smithi160.stdout:223: writing smithi16030472-223 from 531154 to 1010669 tid 1 2024-09-21T18:07:24.109 INFO:tasks.rados.rados.0.smithi160.stdout:223: writing smithi16030472-223 from 1557826 to 2001555 tid 2 2024-09-21T18:07:24.115 INFO:tasks.rados.rados.0.smithi160.stdout:223: writing smithi16030472-223 from 2713669 to 3200997 tid 3 2024-09-21T18:07:24.119 INFO:tasks.rados.rados.0.smithi160.stdout:223: writing smithi16030472-223 from 3679295 to 3731154 tid 4 2024-09-21T18:07:24.119 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.121 INFO:tasks.rados.rados.0.smithi160.stdout:208: finishing write tid 2 to smithi16030472-208 2024-09-21T18:07:24.124 INFO:tasks.rados.rados.0.smithi160.stdout:218: finishing write tid 1 to smithi16030472-218 2024-09-21T18:07:24.124 INFO:tasks.rados.rados.0.smithi160.stdout:218: finishing write tid 2 to smithi16030472-218 2024-09-21T18:07:24.149 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:24.145+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:24.227 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:24.227 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:24.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:24.286 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:24.285+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:07:24.309 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:24.305+0000 7ff40b999640 -1 osd.4 48 handle_osd_map injecting map crc failure 2024-09-21T18:07:24.327 INFO:tasks.rados.rados.0.smithi160.stdout:218: finishing write tid 3 to smithi16030472-218 2024-09-21T18:07:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:204: finishing write tid 3 to smithi16030472-204 2024-09-21T18:07:24.351 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:24.349+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:24.372 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:24.372 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:24.386 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:24.428 INFO:tasks.rados.rados.0.smithi160.stdout:218: finishing write tid 4 to smithi16030472-218 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: finishing write tid 5 to smithi16030472-218 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: oid 218 updating version 0 to 140 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: oid 218 updating version 140 to 141 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: oid 218 updating version 141 to 143 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: oid 218 version 143 is already newer than 142 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 218 v 143 (ObjNum 217 snap 0 seq_num 217) dirty exists 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: left oid 218 (ObjNum 217 snap 0 seq_num 217) 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:218: done (15 left) 2024-09-21T18:07:24.429 INFO:tasks.rados.rados.0.smithi160.stdout:224: write initial oid 224 2024-09-21T18:07:24.430 INFO:tasks.rados.rados.0.smithi160.stdout:224: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-21T18:07:24.435 INFO:tasks.rados.rados.0.smithi160.stdout:224: writing smithi16030472-224 from 547961 to 1102725 tid 1 2024-09-21T18:07:24.441 INFO:tasks.rados.rados.0.smithi160.stdout:224: writing smithi16030472-224 from 1616308 to 2320048 tid 2 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:224: writing smithi16030472-224 from 2741092 to 3439692 tid 3 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: finishing write tid 4 to smithi16030472-204 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: finishing write tid 5 to smithi16030472-204 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: oid 204 updating version 0 to 109 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: oid 204 version 109 is already newer than 107 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: oid 204 version 109 is already newer than 106 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: oid 204 version 109 is already newer than 108 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 204 v 109 (ObjNum 203 snap 0 seq_num 203) dirty exists 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: left oid 204 (ObjNum 203 snap 0 seq_num 203) 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:204: done (15 left) 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:225: write initial oid 225 2024-09-21T18:07:24.448 INFO:tasks.rados.rados.0.smithi160.stdout:225: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-21T18:07:24.452 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:24.450+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:24.454 INFO:tasks.rados.rados.0.smithi160.stdout:225: writing smithi16030472-225 from 564768 to 1194781 tid 1 2024-09-21T18:07:24.459 INFO:tasks.rados.rados.0.smithi160.stdout:225: writing smithi16030472-225 from 1674790 to 2122188 tid 2 2024-09-21T18:07:24.466 INFO:tasks.rados.rados.0.smithi160.stdout:225: writing smithi16030472-225 from 2768515 to 3394740 tid 3 2024-09-21T18:07:24.467 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.521 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:24.521 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:24.521 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:24.535 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:24.553 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:24.549+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:24.654 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:24.649+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:24.662 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:24.662 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:24.663 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:24.663 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:24.663 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:24.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: finishing write tid 3 to smithi16030472-208 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: finishing write tid 4 to smithi16030472-208 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: finishing write tid 5 to smithi16030472-208 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: oid 208 updating version 0 to 145 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: oid 208 version 145 is already newer than 143 2024-09-21T18:07:24.740 INFO:tasks.rados.rados.0.smithi160.stdout:208: oid 208 version 145 is already newer than 144 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:208: oid 208 updating version 145 to 146 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 208 v 146 (ObjNum 207 snap 0 seq_num 207) dirty exists 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:208: left oid 208 (ObjNum 207 snap 0 seq_num 207) 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:207: finishing write tid 1 to smithi16030472-207 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:207: finishing write tid 2 to smithi16030472-207 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:208: done (15 left) 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:226: write initial oid 226 2024-09-21T18:07:24.741 INFO:tasks.rados.rados.0.smithi160.stdout:226: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-21T18:07:24.749 INFO:tasks.rados.rados.0.smithi160.stdout:226: writing smithi16030472-226 from 581575 to 1286837 tid 1 2024-09-21T18:07:24.756 INFO:tasks.rados.rados.0.smithi160.stdout:226: writing smithi16030472-226 from 1733272 to 2440681 tid 2 2024-09-21T18:07:24.763 INFO:tasks.rados.rados.0.smithi160.stdout:226: writing smithi16030472-226 from 3195938 to 3633435 tid 3 2024-09-21T18:07:24.763 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.763 INFO:tasks.rados.rados.0.smithi160.stdout:207: finishing write tid 3 to smithi16030472-207 2024-09-21T18:07:24.763 INFO:tasks.rados.rados.0.smithi160.stdout:207: finishing write tid 4 to smithi16030472-207 2024-09-21T18:07:24.763 INFO:tasks.rados.rados.0.smithi160.stdout:207: finishing write tid 5 to smithi16030472-207 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:207: oid 207 updating version 0 to 163 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:207: oid 207 version 163 is already newer than 160 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:207: oid 207 version 163 is already newer than 162 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:207: oid 207 version 163 is already newer than 161 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 207 v 163 (ObjNum 206 snap 0 seq_num 206) dirty exists 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:207: left oid 207 (ObjNum 206 snap 0 seq_num 206) 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: finishing write tid 1 to smithi16030472-211 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: finishing write tid 2 to smithi16030472-211 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: finishing write tid 3 to smithi16030472-211 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: finishing write tid 4 to smithi16030472-211 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: finishing write tid 5 to smithi16030472-211 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: oid 211 updating version 0 to 165 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: oid 211 updating version 165 to 167 2024-09-21T18:07:24.764 INFO:tasks.rados.rados.0.smithi160.stdout:211: oid 211 version 167 is already newer than 166 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:211: oid 211 version 167 is already newer than 164 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 211 v 167 (ObjNum 210 snap 0 seq_num 210) dirty exists 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:211: left oid 211 (ObjNum 210 snap 0 seq_num 210) 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:214: finishing write tid 1 to smithi16030472-214 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:214: finishing write tid 2 to smithi16030472-214 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:207: done (15 left) 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:211: done (14 left) 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:227: write initial oid 227 2024-09-21T18:07:24.765 INFO:tasks.rados.rados.0.smithi160.stdout:227: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-21T18:07:24.765 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:24.761+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:24.773 INFO:tasks.rados.rados.0.smithi160.stdout:227: writing smithi16030472-227 from 598382 to 1378893 tid 1 2024-09-21T18:07:24.782 INFO:tasks.rados.rados.0.smithi160.stdout:227: writing smithi16030472-227 from 2075401 to 2526468 tid 2 2024-09-21T18:07:24.792 INFO:tasks.rados.rados.0.smithi160.stdout:227: writing smithi16030472-227 from 3107008 to 3755777 tid 3 2024-09-21T18:07:24.792 INFO:tasks.rados.rados.0.smithi160.stdout:228: write initial oid 228 2024-09-21T18:07:24.792 INFO:tasks.rados.rados.0.smithi160.stdout:228: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-21T18:07:24.800 INFO:tasks.rados.rados.0.smithi160.stdout:228: writing smithi16030472-228 from 615189 to 1070949 tid 1 2024-09-21T18:07:24.809 INFO:tasks.rados.rados.0.smithi160.stdout:228: writing smithi16030472-228 from 1733883 to 2444961 tid 2 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:228: writing smithi16030472-228 from 3134431 to 3594472 tid 3 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: finishing write tid 3 to smithi16030472-214 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: finishing write tid 4 to smithi16030472-214 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: finishing write tid 5 to smithi16030472-214 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: oid 214 updating version 0 to 171 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: oid 214 version 171 is already newer than 170 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: oid 214 version 171 is already newer than 169 2024-09-21T18:07:24.816 INFO:tasks.rados.rados.0.smithi160.stdout:214: oid 214 version 171 is already newer than 168 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 214 v 171 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:214: left oid 214 (ObjNum 213 snap 0 seq_num 213) 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:217: finishing write tid 1 to smithi16030472-217 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:217: finishing write tid 2 to smithi16030472-217 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:217: finishing write tid 3 to smithi16030472-217 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:213: finishing write tid 1 to smithi16030472-213 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:214: done (15 left) 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:229: write initial oid 229 2024-09-21T18:07:24.817 INFO:tasks.rados.rados.0.smithi160.stdout:229: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-21T18:07:24.823 INFO:tasks.rados.rados.0.smithi160.stdout:229: writing smithi16030472-229 from 631996 to 1163005 tid 1 2024-09-21T18:07:24.829 INFO:tasks.rados.rados.0.smithi160.stdout:229: writing smithi16030472-229 from 1792365 to 2247101 tid 2 2024-09-21T18:07:24.837 INFO:tasks.rados.rados.0.smithi160.stdout:229: writing smithi16030472-229 from 2761854 to 3433167 tid 3 2024-09-21T18:07:24.837 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.837 INFO:tasks.rados.rados.0.smithi160.stdout:217: finishing write tid 4 to smithi16030472-217 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: finishing write tid 5 to smithi16030472-217 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: oid 217 updating version 0 to 174 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: oid 217 version 174 is already newer than 173 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: oid 217 version 174 is already newer than 172 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: oid 217 updating version 174 to 175 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 217 v 175 (ObjNum 216 snap 0 seq_num 216) dirty exists 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:217: left oid 217 (ObjNum 216 snap 0 seq_num 216) 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: finishing write tid 2 to smithi16030472-213 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: finishing write tid 3 to smithi16030472-213 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: finishing write tid 4 to smithi16030472-213 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: finishing write tid 5 to smithi16030472-213 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: oid 213 updating version 0 to 150 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: oid 213 version 150 is already newer than 147 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: oid 213 version 150 is already newer than 149 2024-09-21T18:07:24.838 INFO:tasks.rados.rados.0.smithi160.stdout:213: oid 213 version 150 is already newer than 148 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 213 v 150 (ObjNum 212 snap 0 seq_num 212) dirty exists 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:213: left oid 213 (ObjNum 212 snap 0 seq_num 212) 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: finishing write tid 1 to smithi16030472-215 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: finishing write tid 2 to smithi16030472-215 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: finishing write tid 3 to smithi16030472-215 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: finishing write tid 4 to smithi16030472-215 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: finishing write tid 5 to smithi16030472-215 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: oid 215 updating version 0 to 154 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: oid 215 version 154 is already newer than 153 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: oid 215 version 154 is already newer than 152 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: oid 215 version 154 is already newer than 151 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 215 v 154 (ObjNum 214 snap 0 seq_num 214) dirty exists 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:215: left oid 215 (ObjNum 214 snap 0 seq_num 214) 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:216: finishing write tid 1 to smithi16030472-216 2024-09-21T18:07:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:213: done (15 left) 2024-09-21T18:07:24.840 INFO:tasks.rados.rados.0.smithi160.stdout:215: done (14 left) 2024-09-21T18:07:24.840 INFO:tasks.rados.rados.0.smithi160.stdout:217: done (13 left) 2024-09-21T18:07:24.840 INFO:tasks.rados.rados.0.smithi160.stdout:230: write initial oid 230 2024-09-21T18:07:24.840 INFO:tasks.rados.rados.0.smithi160.stdout:230: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-21T18:07:24.846 INFO:tasks.rados.rados.0.smithi160.stdout:230: writing smithi16030472-230 from 648803 to 1371414 tid 1 2024-09-21T18:07:24.846 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:24.848 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:157 3.0 3:1192aa23:::smithi16030472-40:head [write 655473~716476,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.200837+0000", 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.643518532, 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41792352100000002, 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.200837+0000", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.200837+0000", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.200842+0000", 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4059999999999998e-06 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201810+0000", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00096796800000000004 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201845+0000", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5405999999999999e-05 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201865+0000", 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9507000000000001e-05 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302143+0000", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100278271 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op must wait for map", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302153+0000", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0543000000000001e-05 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.505702+0000", 2024-09-21T18:07:24.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20354892699999999 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.505947+0000", 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024454199999999999 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.506190+0000", 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0002433 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.618407+0000", 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11221639999999999 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.618556+0000", 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014902000000000001 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.618637+0000", 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1588999999999994e-05 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.618761+0000", 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012364200000000001 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:26 3.0 3:1192aa23:::smithi16030472-40:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.201907+0000", 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.642448658999999, 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41735278599999998, 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.855 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201907+0000", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201907+0000", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201909+0000", 2024-09-21T18:07:24.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.556e-06 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201915+0000", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5239999999999997e-06 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201931+0000", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5426000000000001e-05 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201940+0000", 2024-09-21T18:07:24.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302247+0000", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100306453 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302254+0000", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.0099999999999998e-06 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.510232+0000", 2024-09-21T18:07:24.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20797880799999999 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.510472+0000", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000239526 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.618940+0000", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108467965 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.619094+0000", 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000154091 2024-09-21T18:07:24.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.619260+0000", 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016592700000000001 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:27 3.0 3:1192aa23:::smithi16030472-40:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.201957+0000", 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.642398743999999, 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.419867398, 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.860 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201957+0000", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201957+0000", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201958+0000", 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7400000000000002e-07 2024-09-21T18:07:24.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201964+0000", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7629999999999997e-06 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201975+0000", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1727e-05 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.201993+0000", 2024-09-21T18:07:24.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7876e-05 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302334+0000", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100340893 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting_for_map not empty", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.302345+0000", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0876999999999999e-05 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.510615+0000", 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20826967599999999 2024-09-21T18:07:24.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.510837+0000", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022229899999999999 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.619333+0000", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108496389 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.619439+0000", 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000105571 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.621824+0000", 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023854530000000001 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:177 3.0 3:16d9fc1d:::smithi16030472-45:head [write 739508~692721,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.614401+0000", 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.229954939000001, 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32680888000000002, 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.865 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.614401+0000", 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.614401+0000", 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.614403+0000", 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0159999999999998e-06 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.615023+0000", 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061985100000000004 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616083+0000", 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010605429999999999 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.867 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616111+0000", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7912999999999999e-05 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734630+0000", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.118518977 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734895+0000", 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026451900000000001 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735126+0000", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023157399999999999 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.742562+0000", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00743627 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.940905+0000", 2024-09-21T18:07:24.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19834216800000001 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.941066+0000", 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000160834 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.941210+0000", 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144215 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:39 3.0 3:16d9fc1d:::smithi16030472-45:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.616245+0000", 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.228110731999999, 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32778585500000001, 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616245+0000", 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616245+0000", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616247+0000", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.677e-06 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616257+0000", 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9129999999999998e-06 2024-09-21T18:07:24.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.616278+0000", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1776e-05 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.716423+0000", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100144557 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.937898+0000", 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22147482499999999 2024-09-21T18:07:24.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.938037+0000", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013916500000000001 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.941572+0000", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035348010000000002 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.941663+0000", 2024-09-21T18:07:24.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1447999999999996e-05 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944031+0000", 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023676930000000001 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:182 3.0 3:15fed93e:::smithi16030472-46:head [write 1933572~791670] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.717794+0000", 2024-09-21T18:07:24.875 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.126561924000001, 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33153000599999999, 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.717794+0000", 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.717794+0000", 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.717796+0000", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.84e-06 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.718823+0000", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010271970000000001 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.718911+0000", 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7625000000000002e-05 2024-09-21T18:07:24.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719015+0000", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000104269 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944089+0000", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22507448499999999 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944216+0000", 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000126632 2024-09-21T18:07:24.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.944421+0000", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204797 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.043455+0000", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099034144000000005 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049113+0000", 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0056585070000000001 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049201+0000", 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8141999999999997e-05 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049324+0000", 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000122368 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.880 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:42 3.0 3:15fed93e:::smithi16030472-46:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.719115+0000", 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.125241125000001, 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33050716600000002, 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719115+0000", 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719115+0000", 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719116+0000", 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719124+0000", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1799999999999999e-06 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719139+0000", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.503e-05 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719149+0000", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.049e-05 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.948443+0000", 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22929364899999999 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.948637+0000", 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019467700000000001 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049396+0000", 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100758584 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049502+0000", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00010626199999999999 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049622+0000", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119614 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:43 3.0 3:15fed93e:::smithi16030472-46:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.719161+0000", 2024-09-21T18:07:24.885 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.125194458999999, 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33371984199999999, 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719161+0000", 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719161+0000", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719162+0000", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0299999999999997e-07 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719167+0000", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1930000000000001e-06 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719176+0000", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0399999999999998e-06 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719184+0000", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8169999999999999e-06 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.040965+0000", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32178049800000003 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041083+0000", 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000118505 2024-09-21T18:07:24.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049891+0000", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0088072470000000007 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049994+0000", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000103282 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.052881+0000", 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0028873570000000001 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.889 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:189 3.0 3:1f69a690:::smithi16030472-48:head [write 789929~634821,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.719198+0000", 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.125157966, 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.330622841, 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719198+0000", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719198+0000", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719199+0000", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0699999999999996e-07 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719623+0000", 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042460300000000001 2024-09-21T18:07:24.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719641+0000", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7609999999999999e-05 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719651+0000", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0006e-05 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041145+0000", 2024-09-21T18:07:24.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.321494273 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041351+0000", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000206193 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.041581+0000", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022960999999999999 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049340+0000", 2024-09-21T18:07:24.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0077588459999999998 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049691+0000", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035121499999999999 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049764+0000", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2892000000000006e-05 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049821+0000", 2024-09-21T18:07:24.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6685999999999999e-05 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:190 3.0 3:1f69a690:::smithi16030472-48:head [write 2050536~739393] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.719667+0000", 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.124688707000001, 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33740415800000001, 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719667+0000", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719667+0000", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.719668+0000", 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3430000000000001e-06 2024-09-21T18:07:24.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720566+0000", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089756600000000003 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720582+0000", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.63e-05 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720599+0000", 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6303e-05 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.043511+0000", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32291288600000001 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.043623+0000", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111555 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.043809+0000", 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018550399999999999 2024-09-21T18:07:24.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049347+0000", 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055381950000000001 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.056635+0000", 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0072878769999999999 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.056787+0000", 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000152361 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057071+0000", 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028426800000000001 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:46 3.0 3:1f69a690:::smithi16030472-48:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.900 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.720613+0000", 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.123743185, 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33699439199999998, 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720613+0000", 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720613+0000", 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720614+0000", 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.347e-06 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720618+0000", 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.036e-06 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720629+0000", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1077000000000001e-05 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720637+0000", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.048804+0000", 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32816643499999998 2024-09-21T18:07:24.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.048943+0000", 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013919799999999999 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057199+0000", 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0082562950000000003 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057407+0000", 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020749000000000001 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057607+0000", 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020009100000000001 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:47 3.0 3:1f69a690:::smithi16030472-48:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.720647+0000", 2024-09-21T18:07:24.905 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.123708735999999, 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.34104675299999998, 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720647+0000", 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720647+0000", 2024-09-21T18:07:24.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720648+0000", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720652+0000", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3270000000000002e-06 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720660+0000", 2024-09-21T18:07:24.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0129999999999993e-06 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720668+0000", 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5329999999999997e-06 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049001+0000", 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32833301300000001 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.049099+0000", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7719000000000004e-05 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057704+0000", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0086057530000000007 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.057905+0000", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020024999999999999 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.061694+0000", 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037892479999999998 2024-09-21T18:07:24.909 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:197 3.0 3:11edfedf:::smithi16030472-50:head [write 423543~785319,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.720677+0000", 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.123678912999999, 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.43856204799999998, 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720677+0000", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720677+0000", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.720678+0000", 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1399999999999995e-07 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721595+0000", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00091727499999999995 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721611+0000", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6099999999999998e-05 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721620+0000", 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0399999999999998e-06 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.912 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.052941+0000", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.33132092000000002 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053151+0000", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000209557 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053382+0000", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000231697 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.061691+0000", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0083082199999999998 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158801+0000", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097110361000000006 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158979+0000", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000178128 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159239+0000", 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000259836 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:49 3.0 3:11edfedf:::smithi16030472-50:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.721633+0000", 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.122722398000001, 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.43801733700000001, 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.915 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721633+0000", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721633+0000", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721635+0000", 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.204e-06 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721638+0000", 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8630000000000001e-06 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721646+0000", 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0399999999999993e-06 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721654+0000", 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4159999999999998e-06 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.061799+0000", 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.34014477700000001 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.062022+0000", 2024-09-21T18:07:24.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022342 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159376+0000", 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097353630999999996 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159506+0000", 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013055099999999999 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159651+0000", 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144435 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:50 3.0 3:11edfedf:::smithi16030472-50:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.721663+0000", 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.122692618, 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.44063587100000001, 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.920 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721663+0000", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721663+0000", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721664+0000", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0599999999999999e-07 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721668+0000", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2740000000000001e-06 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721676+0000", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9500000000000001e-06 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.721684+0000", 2024-09-21T18:07:24.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.43e-06 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.146557+0000", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.42487350699999998 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.146705+0000", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014743100000000001 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159743+0000", 2024-09-21T18:07:24.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.013038261000000001 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.159973+0000", 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022975 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.162299+0000", 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0023264620000000001 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:548 3.0 3:0e2c6c87:::smithi16030472-122:head [setxattr _header (57) in=7b,truncate 2033647] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:24.924 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:18.730502+0000", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.1138541220000002, 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32676511000000003, 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.730502+0000", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.730502+0000", 2024-09-21T18:07:24.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.730503+0000", 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.494e-06 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.730510+0000", 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.55e-06 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.926 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.730519+0000", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8859999999999993e-06 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.830617+0000", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100098622 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.954533+0000", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123915737 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.927 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.954692+0000", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000158775 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.954924+0000", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023272599999999999 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.956070+0000", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.001145084 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.928 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.057074+0000", 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10100419300000001 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.057165+0000", 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1498999999999999e-05 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.057267+0000", 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101544 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:130 3.0 3:34ed9d01:::smithi16030472-136:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.729324+0000", 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.1150318480000001, 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.51501301899999996, 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729324+0000", 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729324+0000", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729326+0000", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4370000000000001e-06 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729338+0000", 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1286e-05 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729356+0000", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8159999999999999e-05 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729373+0000", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6823000000000001e-05 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.834444+0000", 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10507143400000001 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.834584+0000", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000139598 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.240531+0000", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.40594685699999999 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.240778+0000", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024796200000000001 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.244337+0000", 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035584620000000001 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:618 3.0 3:21c109ac:::smithi16030472-140:head [write 736173~553965,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.831071+0000", 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0132843820000002, 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.40929153099999999, 2024-09-21T18:07:24.934 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831071+0000", 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831071+0000", 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831072+0000", 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2999999999999999e-07 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831419+0000", 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034693499999999998 2024-09-21T18:07:24.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831435+0000", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5319999999999999e-05 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831444+0000", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-06 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.934554+0000", 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103110348 2024-09-21T18:07:24.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.934920+0000", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036534099999999998 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.935343+0000", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042328100000000002 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.139994+0000", 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20465140900000001 2024-09-21T18:07:24.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.240216+0000", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10022181500000001 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.240293+0000", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6948999999999998e-05 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.240363+0000", 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9672999999999995e-05 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.939 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:619 3.0 3:21c109ac:::smithi16030472-140:head [write 1867350~468823] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.831457+0000", 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.0128984440000002, 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.42038625200000002, 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831457+0000", 2024-09-21T18:07:24.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831457+0000", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.831458+0000", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1000000000000001e-06 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.832042+0000", 2024-09-21T18:07:24.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058397799999999997 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.832058+0000", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.554e-05 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.832067+0000", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9600000000000006e-06 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.244439+0000", 2024-09-21T18:07:24.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41237234900000003 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.244673+0000", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023421299999999999 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.245036+0000", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000362209 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.249468+0000", 2024-09-21T18:07:24.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0044326119999999998 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.251704+0000", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022362060000000001 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.251776+0000", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1339000000000003e-05 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.251844+0000", 2024-09-21T18:07:24.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7745999999999998e-05 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:973 3.0 3:332d9db6:::smithi16030472-218:head [setxattr _header (57) in=7b,truncate 3647119] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e48)", 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:24.023307+0000", 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.82104924800000001, 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.40351077699999999, 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:24.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.023307+0000", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.023307+0000", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.023308+0000", 2024-09-21T18:07:24.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9230000000000001e-06 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.023316+0000", 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1829999999999997e-06 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.023327+0000", 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0910000000000001e-05 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:24.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.123429+0000", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10010229 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.222507+0000", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099078346999999997 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.222651+0000", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000143478 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:24.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.222860+0000", 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020964099999999999 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.326537+0000", 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103676699 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.426677+0000", 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10013979000000001 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.426743+0000", 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6149999999999995e-05 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.426817+0000", 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4365999999999999e-05 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:24.950 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:24.951 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:24.952 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:24.953 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:24.953 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:24.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:24.954 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:24.948+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:230: writing smithi16030472-230 from 1967200 to 2681947 tid 2 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:230: writing smithi16030472-230 from 3421983 to 3848803 tid 3 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:231: write initial oid 231 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:231: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:231: writing smithi16030472-231 from 665610 to 1463470 tid 1 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:231: writing smithi16030472-231 from 1909329 to 2367734 tid 2 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:231: writing smithi16030472-231 from 2816700 to 3226910 tid 3 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: write initial oid 232 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: writing smithi16030472-232 from 682417 to 1155526 tid 1 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: writing smithi16030472-232 from 1567811 to 2169874 tid 2 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: writing smithi16030472-232 from 2844123 to 3465605 tid 3 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.955 INFO:tasks.rados.rados.0.smithi160.stdout:216: finishing write tid 2 to smithi16030472-216 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: finishing write tid 3 to smithi16030472-216 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: finishing write tid 4 to smithi16030472-216 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: finishing write tid 5 to smithi16030472-216 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: oid 216 updating version 0 to 158 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: oid 216 version 158 is already newer than 156 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: oid 216 version 158 is already newer than 157 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: oid 216 version 158 is already newer than 155 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 216 v 158 (ObjNum 215 snap 0 seq_num 215) dirty exists 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: left oid 216 (ObjNum 215 snap 0 seq_num 215) 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:220: finishing write tid 1 to smithi16030472-220 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:216: done (15 left) 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:233: write initial oid 233 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:233: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:233: writing smithi16030472-233 from 699224 to 1247582 tid 1 2024-09-21T18:07:24.956 INFO:tasks.rados.rados.0.smithi160.stdout:233: writing smithi16030472-233 from 2026293 to 2488367 tid 2 2024-09-21T18:07:24.957 INFO:tasks.rados.rados.0.smithi160.stdout:233: writing smithi16030472-233 from 3271546 to 3704300 tid 3 2024-09-21T18:07:24.957 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:24.957 INFO:tasks.rados.rados.0.smithi160.stdout:219: finishing write tid 1 to smithi16030472-219 2024-09-21T18:07:24.957 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:24.957 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:25.032 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:25.032 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:25.052 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:25.047+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:25.131 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:25.131 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:25.131 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:25.131 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:25.148 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:25.191 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:25.189+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:25.254 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:25.250+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:25.278 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:25.278 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:25.278 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:25.278 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:25.278 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:25.292 INFO:tasks.rados.rados.0.smithi160.stdout:219: finishing write tid 2 to smithi16030472-219 2024-09-21T18:07:25.292 INFO:tasks.rados.rados.0.smithi160.stdout:220: finishing write tid 2 to smithi16030472-220 2024-09-21T18:07:25.292 INFO:tasks.rados.rados.0.smithi160.stdout:220: finishing write tid 3 to smithi16030472-220 2024-09-21T18:07:25.292 INFO:tasks.rados.rados.0.smithi160.stdout:220: finishing write tid 4 to smithi16030472-220 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: finishing write tid 5 to smithi16030472-220 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: oid 220 updating version 0 to 176 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: oid 220 updating version 176 to 177 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: oid 220 updating version 177 to 178 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: oid 220 updating version 178 to 179 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 179 (ObjNum 219 snap 0 seq_num 219) dirty exists 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:220: left oid 220 (ObjNum 219 snap 0 seq_num 219) 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:221: finishing write tid 1 to smithi16030472-221 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:221: finishing write tid 2 to smithi16030472-221 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:221: finishing write tid 3 to smithi16030472-221 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:219: finishing write tid 3 to smithi16030472-219 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:219: finishing write tid 4 to smithi16030472-219 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:219: finishing write tid 5 to smithi16030472-219 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:219: oid 219 updating version 0 to 161 2024-09-21T18:07:25.293 INFO:tasks.rados.rados.0.smithi160.stdout:219: oid 219 version 161 is already newer than 160 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:219: oid 219 version 161 is already newer than 159 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:219: oid 219 updating version 161 to 162 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 219 v 162 (ObjNum 218 snap 0 seq_num 218) dirty exists 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:219: left oid 219 (ObjNum 218 snap 0 seq_num 218) 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:232: finishing write tid 1 to smithi16030472-232 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: finishing write tid 4 to smithi16030472-221 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: finishing write tid 5 to smithi16030472-221 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: oid 221 updating version 0 to 180 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: oid 221 updating version 180 to 181 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: oid 221 updating version 181 to 182 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: oid 221 updating version 182 to 183 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 221 v 183 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:221: left oid 221 (ObjNum 220 snap 0 seq_num 220) 2024-09-21T18:07:25.294 INFO:tasks.rados.rados.0.smithi160.stdout:222: finishing write tid 1 to smithi16030472-222 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: finishing write tid 2 to smithi16030472-222 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: finishing write tid 3 to smithi16030472-222 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: finishing write tid 4 to smithi16030472-222 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: finishing write tid 5 to smithi16030472-222 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: oid 222 updating version 0 to 185 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: oid 222 updating version 185 to 186 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: oid 222 version 186 is already newer than 184 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: oid 222 updating version 186 to 187 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 222 v 187 (ObjNum 221 snap 0 seq_num 221) dirty exists 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:222: left oid 222 (ObjNum 221 snap 0 seq_num 221) 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 1 to smithi16030472-223 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 2 to smithi16030472-223 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 3 to smithi16030472-223 2024-09-21T18:07:25.295 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 4 to smithi16030472-223 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 5 to smithi16030472-223 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:212: finishing write tid 1 to smithi16030472-212 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: finishing write tid 6 to smithi16030472-223 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: oid 223 updating version 0 to 189 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: oid 223 updating version 189 to 190 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: oid 223 updating version 190 to 192 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: oid 223 version 192 is already newer than 188 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: oid 223 version 192 is already newer than 191 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 223 v 192 (ObjNum 222 snap 0 seq_num 222) dirty exists 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:223: left oid 223 (ObjNum 222 snap 0 seq_num 222) 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:212: finishing write tid 2 to smithi16030472-212 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:212: finishing write tid 3 to smithi16030472-212 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:219: done (15 left) 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:220: done (14 left) 2024-09-21T18:07:25.296 INFO:tasks.rados.rados.0.smithi160.stdout:221: done (13 left) 2024-09-21T18:07:25.297 INFO:tasks.rados.rados.0.smithi160.stdout:222: done (12 left) 2024-09-21T18:07:25.297 INFO:tasks.rados.rados.0.smithi160.stdout:223: done (11 left) 2024-09-21T18:07:25.297 INFO:tasks.rados.rados.0.smithi160.stdout:234: write initial oid 234 2024-09-21T18:07:25.297 INFO:tasks.rados.rados.0.smithi160.stdout:234: seq_num 233 ranges {716031=623607,2084775=605732,3298969=617062} 2024-09-21T18:07:25.298 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:234: writing smithi16030472-234 from 716031 to 1339638 tid 1 2024-09-21T18:07:25.309 INFO:tasks.rados.rados.0.smithi160.stdout:234: writing smithi16030472-234 from 2084775 to 2690507 tid 2 2024-09-21T18:07:25.316 INFO:tasks.rados.rados.0.smithi160.stdout:234: writing smithi16030472-234 from 3298969 to 3916031 tid 3 2024-09-21T18:07:25.316 INFO:tasks.rados.rados.0.smithi160.stdout:235: write initial oid 235 2024-09-21T18:07:25.316 INFO:tasks.rados.rados.0.smithi160.stdout:235: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-21T18:07:25.324 INFO:tasks.rados.rados.0.smithi160.stdout:235: writing smithi16030472-235 from 732838 to 1431694 tid 1 2024-09-21T18:07:25.331 INFO:tasks.rados.rados.0.smithi160.stdout:235: writing smithi16030472-235 from 2026904 to 2492647 tid 2 2024-09-21T18:07:25.339 INFO:tasks.rados.rados.0.smithi160.stdout:235: writing smithi16030472-235 from 3210039 to 3781690 tid 3 2024-09-21T18:07:25.339 INFO:tasks.rados.rados.0.smithi160.stdout:236: write initial oid 236 2024-09-21T18:07:25.339 INFO:tasks.rados.rados.0.smithi160.stdout:236: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-21T18:07:25.347 INFO:tasks.rados.rados.0.smithi160.stdout:236: writing smithi16030472-236 from 749645 to 1523750 tid 1 2024-09-21T18:07:25.354 INFO:tasks.rados.rados.0.smithi160.stdout:236: writing smithi16030472-236 from 2085386 to 2694787 tid 2 2024-09-21T18:07:25.355 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:25.350+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:25.362 INFO:tasks.rados.rados.0.smithi160.stdout:236: writing smithi16030472-236 from 3237462 to 3949645 tid 3 2024-09-21T18:07:25.362 INFO:tasks.rados.rados.0.smithi160.stdout:237: write initial oid 237 2024-09-21T18:07:25.362 INFO:tasks.rados.rados.0.smithi160.stdout:237: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-21T18:07:25.369 INFO:tasks.rados.rados.0.smithi160.stdout:237: writing smithi16030472-237 from 766452 to 1332159 tid 1 2024-09-21T18:07:25.375 INFO:tasks.rados.rados.0.smithi160.stdout:237: writing smithi16030472-237 from 1860221 to 2329633 tid 2 2024-09-21T18:07:25.381 INFO:tasks.rados.rados.0.smithi160.stdout:237: writing smithi16030472-237 from 2981238 to 3575433 tid 3 2024-09-21T18:07:25.382 INFO:tasks.rados.rados.0.smithi160.stdout:238: write initial oid 238 2024-09-21T18:07:25.382 INFO:tasks.rados.rados.0.smithi160.stdout:238: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-21T18:07:25.389 INFO:tasks.rados.rados.0.smithi160.stdout:238: writing smithi16030472-238 from 783259 to 1424215 tid 1 2024-09-21T18:07:25.396 INFO:tasks.rados.rados.0.smithi160.stdout:238: writing smithi16030472-238 from 1918703 to 2531773 tid 2 2024-09-21T18:07:25.401 INFO:tasks.rados.rados.0.smithi160.stdout:238: writing smithi16030472-238 from 3008661 to 3414128 tid 3 2024-09-21T18:07:25.405 INFO:tasks.rados.rados.0.smithi160.stdout:238: writing smithi16030472-238 from 3821939 to 3983259 tid 4 2024-09-21T18:07:25.405 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.405 INFO:tasks.rados.rados.0.smithi160.stdout:212: finishing write tid 4 to smithi16030472-212 2024-09-21T18:07:25.405 INFO:tasks.rados.rados.0.smithi160.stdout:212: finishing write tid 5 to smithi16030472-212 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:212: oid 212 updating version 0 to 113 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:212: oid 212 updating version 113 to 115 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:212: oid 212 version 115 is already newer than 112 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:212: oid 212 version 115 is already newer than 114 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 212 v 115 (ObjNum 211 snap 0 seq_num 211) dirty exists 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:212: left oid 212 (ObjNum 211 snap 0 seq_num 211) 2024-09-21T18:07:25.406 INFO:tasks.rados.rados.0.smithi160.stdout:227: finishing write tid 1 to smithi16030472-227 2024-09-21T18:07:25.407 INFO:tasks.rados.rados.0.smithi160.stdout:212: done (15 left) 2024-09-21T18:07:25.407 INFO:tasks.rados.rados.0.smithi160.stdout:239: write initial oid 239 2024-09-21T18:07:25.407 INFO:tasks.rados.rados.0.smithi160.stdout:239: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-21T18:07:25.410 INFO:tasks.rados.rados.0.smithi160.stdout:239: writing smithi16030472-239 from 400066 to 1116271 tid 1 2024-09-21T18:07:25.416 INFO:tasks.rados.rados.0.smithi160.stdout:239: writing smithi16030472-239 from 1860832 to 2000066 tid 2 2024-09-21T18:07:25.416 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.416 INFO:tasks.rados.rados.0.smithi160.stdout:226: finishing write tid 1 to smithi16030472-226 2024-09-21T18:07:25.416 INFO:tasks.rados.rados.0.smithi160.stdout:226: finishing write tid 2 to smithi16030472-226 2024-09-21T18:07:25.436 INFO:tasks.rados.rados.0.smithi160.stdout:226: finishing write tid 3 to smithi16030472-226 2024-09-21T18:07:25.436 INFO:tasks.rados.rados.0.smithi160.stdout:226: finishing write tid 4 to smithi16030472-226 2024-09-21T18:07:25.456 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:25.450+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:21 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'11, mlcod=25'9)", 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.173286795999999, 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20317162799999999, 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.475 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297874+0000", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.297877+0000", 2024-09-21T18:07:25.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0130000000000002e-06 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298015+0000", 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013840599999999999 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298043+0000", 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7702999999999999e-05 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298059+0000", 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5512e-05 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298175+0000", 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011620699999999999 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.298212+0000", 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7409e-05 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500956+0000", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20274424699999999 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.501046+0000", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9130999999999998e-05 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:25.479 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.172108799, 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:07:25.483 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.170894740000001, 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:25.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:07:25.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.487 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.169543820000001, 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:07:25.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:07:25.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:07:25.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:93 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'67, mlcod=28'64)", 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.270920908000001, 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203107066, 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200240+0000", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200243+0000", 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.01e-06 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200311+0000", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8039000000000004e-05 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200338+0000", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6934999999999999e-05 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.200352+0000", 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.377e-05 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.299647+0000", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099295849000000005 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.299678+0000", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0565000000000003e-05 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403249+0000", 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10357140400000001 2024-09-21T18:07:25.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403347+0000", 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7493999999999999e-05 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:94 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'68, mlcod=28'64)", 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.269699283000001, 2024-09-21T18:07:25.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20356700799999999, 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201461+0000", 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201464+0000", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.033e-06 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201477+0000", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2743e-05 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201504+0000", 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6435999999999999e-05 2024-09-21T18:07:25.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.201517+0000", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3336000000000001e-05 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300456+0000", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098939036999999994 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300486+0000", 2024-09-21T18:07:25.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0295999999999999e-05 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.404948+0000", 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104461295 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405028+0000", 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.0832000000000001e-05 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:95 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'69, mlcod=28'64)", 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.268662751000001, 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20260623799999999, 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202498+0000", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202501+0000", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202512+0000", 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1590000000000001e-05 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202533+0000", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0665999999999998e-05 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.202546+0000", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2806000000000001e-05 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300947+0000", 2024-09-21T18:07:25.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098401269999999999 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.300977+0000", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0079000000000001e-05 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405035+0000", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10405750800000001 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.405104+0000", 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9555999999999996e-05 2024-09-21T18:07:25.504 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:98 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'72, mlcod=28'64)", 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 36.265188014000003, 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203347474, 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205973+0000", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205976+0000", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.033e-06 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.205989+0000", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2847000000000001e-05 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.206015+0000", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6265999999999999e-05 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.206028+0000", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3199e-05 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403443+0000", 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.197415325 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.403473+0000", 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9543000000000002e-05 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409245+0000", 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057717460000000003 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:49.409320+0000", 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5514999999999998e-05 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.881963183000003, 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.509 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:07:25.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:117 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'87, mlcod=28'80)", 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.781176256000002, 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20472252399999999, 2024-09-21T18:07:25.513 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689984+0000", 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689988+0000", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2600000000000001e-06 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690085+0000", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7133999999999995e-05 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690105+0000", 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9826e-05 2024-09-21T18:07:25.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690118+0000", 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.314e-05 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.790369+0000", 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100250926 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.790444+0000", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5288000000000002e-05 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894614+0000", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104170426 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894707+0000", 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2523999999999997e-05 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:118 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'88, mlcod=28'80)", 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.781012392999997, 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20465380899999999, 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690148+0000", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690150+0000", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.72e-06 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690160+0000", 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0406e-05 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690177+0000", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6296000000000001e-05 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690187+0000", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.039e-05 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.891400+0000", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20121330400000001 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.891457+0000", 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.6508999999999997e-05 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894718+0000", 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0032612829999999998 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894802+0000", 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3900999999999995e-05 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.521 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:119 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'89, mlcod=28'80)", 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.780956617999998, 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.204678522, 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:25.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690204+0000", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690205+0000", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.24e-06 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690246+0000", 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0209000000000001e-05 2024-09-21T18:07:25.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690261+0000", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5596000000000001e-05 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690269+0000", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1529999999999997e-06 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892340+0000", 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20207024700000001 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892378+0000", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8173000000000002e-05 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894808+0000", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0024301629999999999 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894883+0000", 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.4740999999999995e-05 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:120 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'90, mlcod=28'80)", 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.78087489, 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20471718799999999, 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690286+0000", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690287+0000", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0640000000000001e-06 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690344+0000", 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7000999999999999e-05 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690385+0000", 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1526000000000002e-05 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.690394+0000", 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2060000000000007e-06 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892905+0000", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20251159399999999 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892933+0000", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7572e-05 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.894888+0000", 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0019556840000000001 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.895003+0000", 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114541 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 12.783370552999999, 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.530 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 12.782893343, 2024-09-21T18:07:25.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:07:25.538 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 12.78245839, 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:07:25.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:07:25.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:468 3.2 e46/31 3:4999db3b:::smithi16030472-106:head v 46'40, mlcod=46'38)", 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:18.626446+0000", 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.8447144409999998, 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20243491499999999, 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.626446+0000", 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.626446+0000", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.626450+0000", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0400000000000003e-06 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.627425+0000", 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00097423300000000002 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.627476+0000", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1922999999999997e-05 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.627493+0000", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6103000000000002e-05 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.627641+0000", 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014888799999999999 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.627684+0000", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2599000000000003e-05 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.828819+0000", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20113476 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.828881+0000", 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.2368999999999999e-05 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.546 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:501 3.2 e47/31 3:7a41f658:::smithi16030472-113:head v 47'56, mlcod=46'38)", 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:18.934113+0000", 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.5370480459999998, 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203044693, 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.934113+0000", 2024-09-21T18:07:25.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.934113+0000", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.934115+0000", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.21e-06 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.130070+0000", 2024-09-21T18:07:25.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.195954723 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.130097+0000", 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7813e-05 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.130113+0000", 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6059999999999999e-05 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.130242+0000", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000128861 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.130293+0000", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0386000000000002e-05 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.137039+0000", 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0067466710000000001 2024-09-21T18:07:25.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.137157+0000", 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011796899999999999 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:942 3.2 e48/31 3:5b304e71:::smithi16030472-212:head v 48'114, mlcod=48'110)", 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:24.740035+0000", 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.73112595899999999, 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20315293300000001, 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.551 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740035+0000", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740035+0000", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740036+0000", 2024-09-21T18:07:25.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.122e-06 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740648+0000", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000612584 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740666+0000", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7776000000000001e-05 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740677+0000", 2024-09-21T18:07:25.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.057e-05 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.840259+0000", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099581973000000004 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.840296+0000", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7698999999999999e-05 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.943118+0000", 2024-09-21T18:07:25.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10282139999999999 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.943188+0000", 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9808999999999995e-05 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:943 3.2 e48/31 3:5b304e71:::smithi16030472-212:head v 48'115, mlcod=48'110)", 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:24.740693+0000", 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.73046736599999995, 2024-09-21T18:07:25.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20255035599999999, 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740693+0000", 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740693+0000", 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:25.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740694+0000", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.051e-06 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740701+0000", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7970000000000003e-06 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.740713+0000", 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1678999999999999e-05 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.840794+0000", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10008120199999999 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.885520+0000", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044726193999999997 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.885542+0000", 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1375000000000001e-05 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.943194+0000", 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.057652154999999997 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.943244+0000", 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9903e-05 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:25.559 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:25.562 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:25.562 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:25.557+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:226: finishing write tid 5 to smithi16030472-226 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:226: oid 226 updating version 0 to 145 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:226: oid 226 version 145 is already newer than 144 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:226: oid 226 updating version 145 to 146 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:226: oid 226 updating version 146 to 147 2024-09-21T18:07:25.562 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 226 v 147 (ObjNum 225 snap 0 seq_num 225) dirty exists 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:226: left oid 226 (ObjNum 225 snap 0 seq_num 225) 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:228: finishing write tid 1 to smithi16030472-228 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:226: done (15 left) 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:240: write initial oid 240 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:240: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:240: writing smithi16030472-240 from 416873 to 1208327 tid 1 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:240: writing smithi16030472-240 from 1919314 to 2016873 tid 2 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.563 INFO:tasks.rados.rados.0.smithi160.stdout:228: finishing write tid 2 to smithi16030472-228 2024-09-21T18:07:25.637 INFO:tasks.rados.rados.0.smithi160.stdout:227: finishing write tid 2 to smithi16030472-227 2024-09-21T18:07:25.648 INFO:tasks.rados.rados.0.smithi160.stdout:232: finishing write tid 2 to smithi16030472-232 2024-09-21T18:07:25.648 INFO:tasks.rados.rados.0.smithi160.stdout:232: finishing write tid 3 to smithi16030472-232 2024-09-21T18:07:25.662 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:25.660+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:25.737 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:25.737 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:25.737 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:25.738 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:25.752 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:25.763 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:25.758+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:25.774 INFO:tasks.rados.rados.0.smithi160.stdout:228: finishing write tid 3 to smithi16030472-228 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: finishing write tid 4 to smithi16030472-228 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: finishing write tid 5 to smithi16030472-228 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: oid 228 updating version 0 to 148 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: oid 228 updating version 148 to 150 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: oid 228 updating version 150 to 151 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: oid 228 version 151 is already newer than 149 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 228 v 151 (ObjNum 227 snap 0 seq_num 227) dirty exists 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: left oid 228 (ObjNum 227 snap 0 seq_num 227) 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:230: finishing write tid 1 to smithi16030472-230 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:230: finishing write tid 2 to smithi16030472-230 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:228: done (15 left) 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:241: write initial oid 241 2024-09-21T18:07:25.775 INFO:tasks.rados.rados.0.smithi160.stdout:241: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-21T18:07:25.779 INFO:tasks.rados.rados.0.smithi160.stdout:241: writing smithi16030472-241 from 433680 to 900383 tid 1 2024-09-21T18:07:25.786 INFO:tasks.rados.rados.0.smithi160.stdout:241: writing smithi16030472-241 from 1577796 to 2033680 tid 2 2024-09-21T18:07:25.786 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.859 INFO:tasks.rados.rados.0.smithi160.stdout:230: finishing write tid 3 to smithi16030472-230 2024-09-21T18:07:25.860 INFO:tasks.rados.rados.0.smithi160.stdout:230: finishing write tid 4 to smithi16030472-230 2024-09-21T18:07:25.864 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:25.858+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:25.873 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:25.873 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:25.873 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:25.873 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:25.873 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:25.888 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: finishing write tid 4 to smithi16030472-232 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: finishing write tid 5 to smithi16030472-232 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: oid 232 updating version 0 to 164 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: oid 232 updating version 164 to 166 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: oid 232 version 166 is already newer than 165 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: oid 232 version 166 is already newer than 163 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 232 v 166 (ObjNum 231 snap 0 seq_num 231) dirty exists 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:232: left oid 232 (ObjNum 231 snap 0 seq_num 231) 2024-09-21T18:07:25.954 INFO:tasks.rados.rados.0.smithi160.stdout:224: finishing write tid 1 to smithi16030472-224 2024-09-21T18:07:25.955 INFO:tasks.rados.rados.0.smithi160.stdout:224: finishing write tid 2 to smithi16030472-224 2024-09-21T18:07:25.955 INFO:tasks.rados.rados.0.smithi160.stdout:232: done (15 left) 2024-09-21T18:07:25.955 INFO:tasks.rados.rados.0.smithi160.stdout:242: write initial oid 242 2024-09-21T18:07:25.955 INFO:tasks.rados.rados.0.smithi160.stdout:242: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-21T18:07:25.961 INFO:tasks.rados.rados.0.smithi160.stdout:242: writing smithi16030472-242 from 450487 to 992439 tid 1 2024-09-21T18:07:25.965 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:25.961+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:242: writing smithi16030472-242 from 1636278 to 2050487 tid 2 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: finishing write tid 3 to smithi16030472-224 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:227: finishing write tid 3 to smithi16030472-227 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: finishing write tid 4 to smithi16030472-224 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:240: finishing write tid 1 to smithi16030472-240 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: finishing write tid 5 to smithi16030472-224 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: oid 224 updating version 0 to 195 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: oid 224 updating version 195 to 196 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: oid 224 version 196 is already newer than 194 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:224: oid 224 version 196 is already newer than 193 2024-09-21T18:07:25.969 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 224 v 196 (ObjNum 223 snap 0 seq_num 223) dirty exists 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:224: left oid 224 (ObjNum 223 snap 0 seq_num 223) 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:225: finishing write tid 1 to smithi16030472-225 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:231: finishing write tid 1 to smithi16030472-231 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:224: done (15 left) 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:243: write initial oid 243 2024-09-21T18:07:25.970 INFO:tasks.rados.rados.0.smithi160.stdout:243: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-21T18:07:25.976 INFO:tasks.rados.rados.0.smithi160.stdout:243: writing smithi16030472-243 from 467294 to 1084495 tid 1 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:243: writing smithi16030472-243 from 1578407 to 2067294 tid 2 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:240: finishing write tid 2 to smithi16030472-240 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:240: finishing write tid 3 to smithi16030472-240 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: finishing write tid 4 to smithi16030472-227 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: finishing write tid 5 to smithi16030472-227 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: oid 227 updating version 0 to 116 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: oid 227 updating version 116 to 117 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: oid 227 updating version 117 to 118 2024-09-21T18:07:25.983 INFO:tasks.rados.rados.0.smithi160.stdout:227: oid 227 updating version 118 to 119 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 227 v 119 (ObjNum 226 snap 0 seq_num 226) dirty exists 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:227: left oid 227 (ObjNum 226 snap 0 seq_num 226) 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:229: finishing write tid 1 to smithi16030472-229 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:229: finishing write tid 2 to smithi16030472-229 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:227: done (15 left) 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:244: write initial oid 244 2024-09-21T18:07:25.984 INFO:tasks.rados.rados.0.smithi160.stdout:244: seq_num 243 ranges {484101=692450,1636889=447212} 2024-09-21T18:07:25.991 INFO:tasks.rados.rados.0.smithi160.stdout:244: writing smithi16030472-244 from 484101 to 1176551 tid 1 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:244: writing smithi16030472-244 from 1636889 to 2084101 tid 2 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: finishing write tid 5 to smithi16030472-230 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: oid 230 updating version 0 to 154 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: oid 230 version 154 is already newer than 153 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: oid 230 updating version 154 to 155 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: oid 230 version 155 is already newer than 152 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 155 (ObjNum 229 snap 0 seq_num 229) dirty exists 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:230: left oid 230 (ObjNum 229 snap 0 seq_num 229) 2024-09-21T18:07:25.998 INFO:tasks.rados.rados.0.smithi160.stdout:237: finishing write tid 1 to smithi16030472-237 2024-09-21T18:07:25.999 INFO:tasks.rados.rados.0.smithi160.stdout:237: finishing write tid 2 to smithi16030472-237 2024-09-21T18:07:25.999 INFO:tasks.rados.rados.0.smithi160.stdout:229: finishing write tid 3 to smithi16030472-229 2024-09-21T18:07:25.999 INFO:tasks.rados.rados.0.smithi160.stdout:230: done (15 left) 2024-09-21T18:07:25.999 INFO:tasks.rados.rados.0.smithi160.stdout:245: write initial oid 245 2024-09-21T18:07:25.999 INFO:tasks.rados.rados.0.smithi160.stdout:245: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-21T18:07:26.005 INFO:tasks.rados.rados.0.smithi160.stdout:245: writing smithi16030472-245 from 500908 to 984960 tid 1 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:245: writing smithi16030472-245 from 1411724 to 2100908 tid 2 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: finishing write tid 2 to smithi16030472-231 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: finishing write tid 3 to smithi16030472-231 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: finishing write tid 4 to smithi16030472-231 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: finishing write tid 5 to smithi16030472-231 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: oid 231 updating version 0 to 201 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: oid 231 version 201 is already newer than 199 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: oid 231 version 201 is already newer than 198 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:231: oid 231 version 201 is already newer than 200 2024-09-21T18:07:26.013 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 231 v 201 (ObjNum 230 snap 0 seq_num 230) dirty exists 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:231: left oid 231 (ObjNum 230 snap 0 seq_num 230) 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: finishing write tid 1 to smithi16030472-233 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: finishing write tid 2 to smithi16030472-233 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: finishing write tid 3 to smithi16030472-233 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: finishing write tid 4 to smithi16030472-233 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: finishing write tid 5 to smithi16030472-233 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: oid 233 updating version 0 to 203 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: oid 233 version 203 is already newer than 202 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: oid 233 updating version 203 to 205 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: oid 233 version 205 is already newer than 204 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 233 v 205 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: left oid 233 (ObjNum 232 snap 0 seq_num 232) 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:231: done (15 left) 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:233: done (14 left) 2024-09-21T18:07:26.014 INFO:tasks.rados.rados.0.smithi160.stdout:246: write initial oid 246 2024-09-21T18:07:26.015 INFO:tasks.rados.rados.0.smithi160.stdout:246: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-21T18:07:26.015 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:121 3.0 e43/31 3:1440a676:::smithi16030472-31:head v 43'8, mlcod=43'6)", 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 13.119543123, 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20492206299999999, 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893673+0000", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.893677+0000", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.653e-06 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.894421+0000", 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00074468899999999998 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.894451+0000", 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9096000000000001e-05 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.994619+0000", 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100168902 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.094872+0000", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100252722 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.094932+0000", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0152000000000003e-05 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.098481+0000", 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.003548697 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.098595+0000", 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000114152 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:175 3.0 e44/31 3:364cb6d5:::smithi16030472-44:head v 44'14, mlcod=44'10)", 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 12.379175436000001, 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10156053399999999, 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.020 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634041+0000", 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634044+0000", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4759999999999998e-06 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634062+0000", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7717e-05 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.634096+0000", 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3679000000000001e-05 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734214+0000", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100118467 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734422+0000", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020803400000000001 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.734466+0000", 2024-09-21T18:07:26.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3639e-05 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735528+0000", 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00106251 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:13.735601+0000", 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3011999999999998e-05 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:197 3.0 e44/31 3:11edfedf:::smithi16030472-50:head v 44'20, mlcod=44'17)", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 11.959580309, 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104761221, 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053636+0000", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.053639+0000", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2729999999999998e-06 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054586+0000", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00094677300000000004 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054621+0000", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5095999999999997e-05 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.054646+0000", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4530000000000001e-05 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.154924+0000", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10027860700000001 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.154961+0000", 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6636e-05 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158262+0000", 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0033014049999999999 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:14.158397+0000", 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013490099999999999 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:548 3.0 e47/31 3:0e2c6c87:::smithi16030472-122:head v 47'54, mlcod=47'52)", 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:18.955170+0000", 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 7.0580465300000004, 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101556591, 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955170+0000", 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955170+0000", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955174+0000", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2400000000000001e-06 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955189+0000", 2024-09-21T18:07:26.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4997000000000001e-05 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955221+0000", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2376000000000003e-05 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:18.955238+0000", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.055526+0000", 2024-09-21T18:07:26.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.1002884 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.055562+0000", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5389000000000003e-05 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.056629+0000", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0010672050000000001 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.056726+0000", 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7493999999999999e-05 2024-09-21T18:07:26.032 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:572 3.0 e47/31 3:14ea3e66:::smithi16030472-128:head v 47'57, mlcod=47'54)", 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.243873+0000", 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.769343696, 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100757111, 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243873+0000", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243873+0000", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243877+0000", 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0099999999999997e-06 2024-09-21T18:07:26.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243893+0000", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6507e-05 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243926+0000", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3315000000000002e-05 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243944+0000", 2024-09-21T18:07:26.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7986999999999999e-05 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.244572+0000", 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000627864 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.244631+0000", 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8956000000000003e-05 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.344565+0000", 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099934219000000005 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.344630+0000", 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.4252999999999996e-05 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:574 3.0 e47/31 3:1a999e0e:::smithi16030472-129:head v 47'58, mlcod=47'54)", 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.243982+0000", 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.7692338569999997, 2024-09-21T18:07:26.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100687761, 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243982+0000", 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243982+0000", 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.243984+0000", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.376e-06 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.244405+0000", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00042125399999999998 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.244430+0000", 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.48e-05 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.244442+0000", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2536e-05 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.245220+0000", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00077764499999999999 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.245266+0000", 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6397000000000003e-05 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.344633+0000", 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099366987000000004 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.344670+0000", 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6766e-05 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:602 3.0 e47/31 3:34ed9d01:::smithi16030472-136:head v 47'67, mlcod=47'65)", 2024-09-21T18:07:26.041 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.728460+0000", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.2847567370000004, 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106390972, 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.728460+0000", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.728460+0000", 2024-09-21T18:07:26.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.728463+0000", 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3170000000000002e-06 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729079+0000", 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00061572700000000005 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729108+0000", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9e-05 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729123+0000", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5e-05 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729279+0000", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015652400000000001 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.729326+0000", 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.6712999999999998e-05 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.834759+0000", 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105433265 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.834850+0000", 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1426000000000002e-05 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.045 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:603 3.0 e47/31 3:34ed9d01:::smithi16030472-136:head v 47'68, mlcod=47'65)", 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:19.731904+0000", 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 6.2813124340000002, 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10400868000000001, 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.731904+0000", 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.731904+0000", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.731907+0000", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1659999999999998e-06 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.732490+0000", 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058309900000000001 2024-09-21T18:07:26.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.732518+0000", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8119e-05 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.732539+0000", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0599999999999999e-05 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.832782+0000", 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100242731 2024-09-21T18:07:26.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.832819+0000", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7070000000000003e-05 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.835836+0000", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0030169559999999999 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:19.835912+0000", 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6939000000000003e-05 2024-09-21T18:07:26.049 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:620 3.0 e47/31 3:21c109ac:::smithi16030472-140:head v 47'74, mlcod=47'71)", 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.248751+0000", 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.7644653379999999, 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104228917, 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.248751+0000", 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.248751+0000", 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.248754+0000", 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1599999999999998e-06 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.248773+0000", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8896e-05 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.248800+0000", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6812999999999999e-05 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.348950+0000", 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100150119 2024-09-21T18:07:26.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.349140+0000", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019046800000000001 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.349183+0000", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2543000000000002e-05 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.352749+0000", 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035660470000000001 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.352980+0000", 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00023087100000000001 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:656 3.0 e47/31 3:38160a63:::smithi16030472-149:head v 47'83, mlcod=47'80)", 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.561674+0000", 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.4515423380000003, 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20867144400000001, 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561674+0000", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561674+0000", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561677+0000", 2024-09-21T18:07:26.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5829999999999998e-06 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561693+0000", 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5026e-05 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561721+0000", 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8954000000000001e-05 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561740+0000", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.895e-05 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.662004+0000", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100263561 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.662048+0000", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4273000000000001e-05 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770115+0000", 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10806691 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770345+0000", 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000230187 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:686 3.0 e47/31 3:3c3db2a9:::smithi16030472-157:head v 47'84, mlcod=47'80)", 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.561776+0000", 2024-09-21T18:07:26.058 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.451440023, 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20864221099999999, 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561776+0000", 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561776+0000", 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.561777+0000", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.173e-06 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562358+0000", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00058045899999999997 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562401+0000", 2024-09-21T18:07:26.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3590000000000001e-05 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562414+0000", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2543e-05 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.762593+0000", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20017907400000001 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.762630+0000", 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7286000000000002e-05 2024-09-21T18:07:26.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770350+0000", 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0077199310000000002 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770418+0000", 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8155000000000001e-05 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.062 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:687 3.0 e47/31 3:3c3db2a9:::smithi16030472-157:head v 47'85, mlcod=47'80)", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.562436+0000", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.4507800120000001, 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.208024816, 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562436+0000", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562436+0000", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.562437+0000", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.263e-06 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563112+0000", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067451800000000004 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563127+0000", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4887e-05 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563135+0000", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2700000000000004e-06 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.764314+0000", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20117889 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.764348+0000", 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4153000000000003e-05 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770426+0000", 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0060779229999999998 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770461+0000", 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4912e-05 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:688 3.0 e47/31 3:3c3db2a9:::smithi16030472-157:head v 47'86, mlcod=47'80)", 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.563149+0000", 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.450067582, 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20734688300000001, 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563149+0000", 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563149+0000", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563149+0000", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9699999999999995e-07 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563157+0000", 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9500000000000001e-06 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563165+0000", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8159999999999994e-06 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.563172+0000", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3099999999999997e-06 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.766505+0000", 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.203333918 2024-09-21T18:07:26.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.766540+0000", 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4536999999999998e-05 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770463+0000", 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0039233180000000003 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770496+0000", 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2237000000000003e-05 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:690 3.0 e47/31 3:15f546d9:::smithi16030472-158:head v 47'87, mlcod=47'81)", 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.662625+0000", 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3505909489999999, 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107915419, 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.071 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.662625+0000", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.662625+0000", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.662628+0000", 2024-09-21T18:07:26.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9830000000000001e-06 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.663286+0000", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065796799999999999 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.663311+0000", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4796000000000001e-05 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.663329+0000", 2024-09-21T18:07:26.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8340000000000001e-05 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.766880+0000", 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10355110300000001 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.766907+0000", 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6988999999999999e-05 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770498+0000", 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0035900469999999999 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.770541+0000", 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3192999999999998e-05 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:703 3.0 e47/31 3:04c0fdf4:::smithi16030472-161:head v 47'89, mlcod=47'81)", 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:20.690740+0000", 2024-09-21T18:07:26.075 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 5.3224765139999999, 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10609484700000001, 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.690740+0000", 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.690740+0000", 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.690743+0000", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0800000000000002e-06 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.691927+0000", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00118458 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.691963+0000", 2024-09-21T18:07:26.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5716000000000002e-05 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.792087+0000", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012426300000001 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.792230+0000", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014271499999999999 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.792286+0000", 2024-09-21T18:07:26.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5535999999999998e-05 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.796704+0000", 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.004418712 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:20.796835+0000", 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000130245 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:973 3.0 e48/31 3:332d9db6:::smithi16030472-218:head v 48'143, mlcod=48'140)", 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:24.223100+0000", 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.7901159820000001, 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101640388, 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223100+0000", 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223100+0000", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223103+0000", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9469999999999999e-06 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223115+0000", 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1929999999999999e-05 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223144+0000", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9065999999999999e-05 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.223158+0000", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4263e-05 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.323429+0000", 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100270163 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.323492+0000", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3288999999999997e-05 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.324673+0000", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0011808739999999999 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:24.324741+0000", 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7855999999999995e-05 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1011 3.0 e49/31 3:101418ef:::smithi16030472-226:head v 49'144, mlcod=48'142)", 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:25.136372+0000", 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.876844344, 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105801197, 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.136372+0000", 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.136372+0000", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.136373+0000", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8999999999999995e-07 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.137150+0000", 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00077707299999999998 2024-09-21T18:07:26.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.137173+0000", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3453999999999999e-05 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.137190+0000", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6325999999999999e-05 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.238097+0000", 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100907886 2024-09-21T18:07:26.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.238118+0000", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0458999999999998e-05 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.242129+0000", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0040108660000000001 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.242173+0000", 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4342999999999999e-05 2024-09-21T18:07:26.087 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1012 3.0 e49/31 3:101418ef:::smithi16030472-226:head v 49'145, mlcod=48'142)", 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:25.140108+0000", 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.87310777799999995, 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104837323, 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140108+0000", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140108+0000", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140113+0000", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0659999999999997e-06 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140872+0000", 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00075942800000000003 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140905+0000", 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3068999999999998e-05 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.140934+0000", 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9280000000000001e-05 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.239477+0000", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098542728999999996 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.239510+0000", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3117999999999997e-05 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.244673+0000", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0051630360000000002 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.244946+0000", 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00027259700000000002 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1013 3.0 e49/31 3:101418ef:::smithi16030472-226:head v 49'146, mlcod=48'142)", 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:25.143178+0000", 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.870038058, 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103393262, 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143178+0000", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143178+0000", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143181+0000", 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2569999999999999e-06 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143645+0000", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000464041 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143675+0000", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9856e-05 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.143700+0000", 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4513000000000001e-05 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.241014+0000", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097314036000000007 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.241043+0000", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8751e-05 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.246484+0000", 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0054417190000000002 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.246571+0000", 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7089000000000006e-05 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1014 3.0 e49/31 3:101418ef:::smithi16030472-226:head v 49'147, mlcod=48'142)", 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:25.145909+0000", 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.86730679200000005, 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100734955, 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:26.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145909+0000", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145909+0000", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145913+0000", 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6789999999999998e-06 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145930+0000", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6667000000000002e-05 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145968+0000", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8405999999999998e-05 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.145999+0000", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0456000000000001e-05 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:26.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.241928+0000", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.095929337000000003 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.241951+0000", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3365000000000001e-05 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.246578+0000", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0046263659999999998 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.246644+0000", 2024-09-21T18:07:26.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.6679000000000003e-05 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:26.100 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:26.102 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:246: writing smithi16030472-246 from 517715 to 1077016 tid 1 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:246: writing smithi16030472-246 from 1870206 to 2117715 tid 2 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:247: write initial oid 247 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:247: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:247: writing smithi16030472-247 from 534522 to 1169072 tid 1 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:247: writing smithi16030472-247 from 1812335 to 2134522 tid 2 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:240: finishing write tid 4 to smithi16030472-240 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:240: oid 240 updating version 0 to 167 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:240: oid 240 updating version 167 to 168 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:240: oid 240 updating version 168 to 169 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 240 v 169 (ObjNum 239 snap 0 seq_num 239) dirty exists 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:240: left oid 240 (ObjNum 239 snap 0 seq_num 239) 2024-09-21T18:07:26.103 INFO:tasks.rados.rados.0.smithi160.stdout:241: finishing write tid 1 to smithi16030472-241 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout:240: done (15 left) 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout:248: write initial oid 248 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout:248: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout:248: writing smithi16030472-248 from 551329 to 1261128 tid 1 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout:248: writing smithi16030472-248 from 1870817 to 2151329 tid 2 2024-09-21T18:07:26.104 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.104 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:26.096+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:26.202 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:26.198+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:26.248 INFO:tasks.rados.rados.0.smithi160.stdout:247: finishing write tid 1 to smithi16030472-247 2024-09-21T18:07:26.248 INFO:tasks.rados.rados.0.smithi160.stdout:247: finishing write tid 2 to smithi16030472-247 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: finishing write tid 3 to smithi16030472-247 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: finishing write tid 4 to smithi16030472-247 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: oid 247 updating version 0 to 160 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: oid 247 version 160 is already newer than 159 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: oid 247 version 160 is already newer than 158 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 160 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: left oid 247 (ObjNum 246 snap 0 seq_num 246) 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:247: done (15 left) 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:249: write initial oid 249 2024-09-21T18:07:26.249 INFO:tasks.rados.rados.0.smithi160.stdout:249: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-21T18:07:26.257 INFO:tasks.rados.rados.0.smithi160.stdout:249: writing smithi16030472-249 from 568136 to 1353184 tid 1 2024-09-21T18:07:26.262 INFO:tasks.rados.rados.0.smithi160.stdout:249: writing smithi16030472-249 from 1929299 to 2168136 tid 2 2024-09-21T18:07:26.262 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.275 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:26.275 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:26.276 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:26.276 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:26.279 INFO:tasks.rados.rados.0.smithi160.stdout:237: finishing write tid 3 to smithi16030472-237 2024-09-21T18:07:26.290 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:26.294 INFO:tasks.rados.rados.0.smithi160.stdout:234: finishing write tid 1 to smithi16030472-234 2024-09-21T18:07:26.294 INFO:tasks.rados.rados.0.smithi160.stdout:235: finishing write tid 1 to smithi16030472-235 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: finishing write tid 2 to smithi16030472-235 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: finishing write tid 3 to smithi16030472-235 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: finishing write tid 4 to smithi16030472-235 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: finishing write tid 5 to smithi16030472-235 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: oid 235 updating version 0 to 126 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: oid 235 version 126 is already newer than 125 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: oid 235 updating version 126 to 127 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: oid 235 version 127 is already newer than 124 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 235 v 127 (ObjNum 234 snap 0 seq_num 234) dirty exists 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:235: left oid 235 (ObjNum 234 snap 0 seq_num 234) 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 1 to smithi16030472-238 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 2 to smithi16030472-238 2024-09-21T18:07:26.295 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 3 to smithi16030472-238 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 4 to smithi16030472-238 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 5 to smithi16030472-238 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: finishing write tid 6 to smithi16030472-238 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: oid 238 updating version 0 to 128 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: oid 238 updating version 128 to 132 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: oid 238 version 132 is already newer than 131 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: oid 238 version 132 is already newer than 129 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: oid 238 version 132 is already newer than 130 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 238 v 132 (ObjNum 237 snap 0 seq_num 237) dirty exists 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: left oid 238 (ObjNum 237 snap 0 seq_num 237) 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:235: done (15 left) 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:238: done (14 left) 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:250: write initial oid 250 2024-09-21T18:07:26.296 INFO:tasks.rados.rados.0.smithi160.stdout:250: seq_num 249 ranges {584943=460297,1587781=597162} 2024-09-21T18:07:26.303 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:26.298+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:26.303 INFO:tasks.rados.rados.0.smithi160.stdout:250: writing smithi16030472-250 from 584943 to 1045240 tid 1 2024-09-21T18:07:26.312 INFO:tasks.rados.rados.0.smithi160.stdout:250: writing smithi16030472-250 from 1587781 to 2184943 tid 2 2024-09-21T18:07:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:251: write initial oid 251 2024-09-21T18:07:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:251: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-21T18:07:26.320 INFO:tasks.rados.rados.0.smithi160.stdout:251: writing smithi16030472-251 from 601750 to 1137296 tid 1 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:251: writing smithi16030472-251 from 1929910 to 2201750 tid 2 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: finishing write tid 1 to smithi16030472-239 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: finishing write tid 2 to smithi16030472-239 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: finishing write tid 3 to smithi16030472-239 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: finishing write tid 4 to smithi16030472-239 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: oid 239 updating version 0 to 133 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: oid 239 updating version 133 to 135 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: oid 239 version 135 is already newer than 134 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 239 v 135 (ObjNum 238 snap 0 seq_num 238) dirty exists 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:239: left oid 239 (ObjNum 238 snap 0 seq_num 238) 2024-09-21T18:07:26.329 INFO:tasks.rados.rados.0.smithi160.stdout:243: finishing write tid 1 to smithi16030472-243 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:234: finishing write tid 2 to smithi16030472-234 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:234: finishing write tid 3 to smithi16030472-234 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:236: finishing write tid 1 to smithi16030472-236 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:236: finishing write tid 2 to smithi16030472-236 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:236: finishing write tid 3 to smithi16030472-236 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:236: finishing write tid 4 to smithi16030472-236 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:239: done (15 left) 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:252: write initial oid 252 2024-09-21T18:07:26.330 INFO:tasks.rados.rados.0.smithi160.stdout:252: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-21T18:07:26.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:07:26.339 INFO:tasks.rados.rados.0.smithi160.stdout:252: writing smithi16030472-252 from 618557 to 1345705 tid 1 2024-09-21T18:07:26.346 INFO:tasks.rados.rados.0.smithi160.stdout:252: writing smithi16030472-252 from 2104745 to 2218557 tid 2 2024-09-21T18:07:26.346 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.379 INFO:tasks.rados.rados.0.smithi160.stdout:237: finishing write tid 4 to smithi16030472-237 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: finishing write tid 5 to smithi16030472-237 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: oid 237 updating version 0 to 157 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: oid 237 updating version 157 to 159 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: oid 237 version 159 is already newer than 158 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: oid 237 version 159 is already newer than 156 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 237 v 159 (ObjNum 236 snap 0 seq_num 236) dirty exists 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: left oid 237 (ObjNum 236 snap 0 seq_num 236) 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:237: done (15 left) 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:253: write initial oid 253 2024-09-21T18:07:26.380 INFO:tasks.rados.rados.0.smithi160.stdout:253: seq_num 252 ranges {635364=402397,1763227=472137} 2024-09-21T18:07:26.388 INFO:tasks.rados.rados.0.smithi160.stdout:253: writing smithi16030472-253 from 635364 to 1037761 tid 1 2024-09-21T18:07:26.397 INFO:tasks.rados.rados.0.smithi160.stdout:253: writing smithi16030472-253 from 1763227 to 2235364 tid 2 2024-09-21T18:07:26.397 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.404 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:26.398+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: finishing write tid 5 to smithi16030472-236 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: oid 236 updating version 0 to 209 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: oid 236 version 209 is already newer than 208 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: oid 236 version 209 is already newer than 207 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: oid 236 version 209 is already newer than 206 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 236 v 209 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: left oid 236 (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:07:26.413 INFO:tasks.rados.rados.0.smithi160.stdout:236: done (15 left) 2024-09-21T18:07:26.414 INFO:tasks.rados.rados.0.smithi160.stdout:254: write initial oid 254 2024-09-21T18:07:26.414 INFO:tasks.rados.rados.0.smithi160.stdout:254: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-21T18:07:26.418 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:26.418 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:26.418 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:26.422 INFO:tasks.rados.rados.0.smithi160.stdout:254: writing smithi16030472-254 from 652171 to 1129817 tid 1 2024-09-21T18:07:26.431 INFO:tasks.rados.rados.0.smithi160.stdout:254: writing smithi16030472-254 from 1821709 to 2252171 tid 2 2024-09-21T18:07:26.431 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.432 INFO:tasks.rados.rados.0.smithi160.stdout:225: finishing write tid 2 to smithi16030472-225 2024-09-21T18:07:26.432 INFO:tasks.rados.rados.0.smithi160.stdout:225: finishing write tid 3 to smithi16030472-225 2024-09-21T18:07:26.432 INFO:tasks.rados.rados.0.smithi160.stdout:225: finishing write tid 4 to smithi16030472-225 2024-09-21T18:07:26.433 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:26.505 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:26.501+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:26.506 INFO:tasks.rados.rados.0.smithi160.stdout:229: finishing write tid 4 to smithi16030472-229 2024-09-21T18:07:26.506 INFO:tasks.rados.rados.0.smithi160.stdout:243: finishing write tid 2 to smithi16030472-243 2024-09-21T18:07:26.512 INFO:tasks.rados.rados.0.smithi160.stdout:229: finishing write tid 5 to smithi16030472-229 2024-09-21T18:07:26.512 INFO:tasks.rados.rados.0.smithi160.stdout:229: oid 229 updating version 0 to 120 2024-09-21T18:07:26.512 INFO:tasks.rados.rados.0.smithi160.stdout:229: oid 229 updating version 120 to 121 2024-09-21T18:07:26.512 INFO:tasks.rados.rados.0.smithi160.stdout:229: oid 229 updating version 121 to 122 2024-09-21T18:07:26.512 INFO:tasks.rados.rados.0.smithi160.stdout:229: oid 229 updating version 122 to 126 2024-09-21T18:07:26.513 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 229 v 126 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-09-21T18:07:26.513 INFO:tasks.rados.rados.0.smithi160.stdout:229: left oid 229 (ObjNum 228 snap 0 seq_num 228) 2024-09-21T18:07:26.513 INFO:tasks.rados.rados.0.smithi160.stdout:229: done (15 left) 2024-09-21T18:07:26.513 INFO:tasks.rados.rados.0.smithi160.stdout:255: write initial oid 255 2024-09-21T18:07:26.513 INFO:tasks.rados.rados.0.smithi160.stdout:255: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-21T18:07:26.523 INFO:tasks.rados.rados.0.smithi160.stdout:255: writing smithi16030472-255 from 668978 to 1221873 tid 1 2024-09-21T18:07:26.532 INFO:tasks.rados.rados.0.smithi160.stdout:255: writing smithi16030472-255 from 1763838 to 2268978 tid 2 2024-09-21T18:07:26.532 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: finishing write tid 2 to smithi16030472-241 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: finishing write tid 3 to smithi16030472-241 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: finishing write tid 4 to smithi16030472-241 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: oid 241 updating version 0 to 170 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: oid 241 updating version 170 to 171 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: oid 241 updating version 171 to 172 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 172 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-21T18:07:26.533 INFO:tasks.rados.rados.0.smithi160.stdout:241: left oid 241 (ObjNum 240 snap 0 seq_num 240) 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:242: finishing write tid 1 to smithi16030472-242 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:242: finishing write tid 2 to smithi16030472-242 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:241: done (15 left) 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:256: write initial oid 256 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:242: finishing write tid 3 to smithi16030472-242 2024-09-21T18:07:26.534 INFO:tasks.rados.rados.0.smithi160.stdout:256: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-21T18:07:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:256: writing smithi16030472-256 from 685785 to 1313929 tid 1 2024-09-21T18:07:26.552 INFO:tasks.rados.rados.0.smithi160.stdout:256: writing smithi16030472-256 from 1822320 to 2285785 tid 2 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: finishing write tid 4 to smithi16030472-242 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: oid 242 updating version 0 to 173 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: oid 242 updating version 173 to 174 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: oid 242 updating version 174 to 175 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 242 v 175 (ObjNum 241 snap 0 seq_num 241) dirty exists 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: left oid 242 (ObjNum 241 snap 0 seq_num 241) 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:242: done (15 left) 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:257: write initial oid 257 2024-09-21T18:07:26.553 INFO:tasks.rados.rados.0.smithi160.stdout:257: seq_num 256 ranges {702592=703393,1880802=421790} 2024-09-21T18:07:26.562 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:26.562 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:26.564 INFO:tasks.rados.rados.0.smithi160.stdout:257: writing smithi16030472-257 from 702592 to 1405985 tid 1 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:257: writing smithi16030472-257 from 1880802 to 2302592 tid 2 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:225: finishing write tid 5 to smithi16030472-225 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:225: oid 225 updating version 0 to 198 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:225: oid 225 updating version 198 to 200 2024-09-21T18:07:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:225: oid 225 version 200 is already newer than 199 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:225: oid 225 version 200 is already newer than 197 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 200 (ObjNum 224 snap 0 seq_num 224) dirty exists 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:225: left oid 225 (ObjNum 224 snap 0 seq_num 224) 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:225: done (15 left) 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:258: write initial oid 258 2024-09-21T18:07:26.573 INFO:tasks.rados.rados.0.smithi160.stdout:258: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-21T18:07:26.578 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:26.584 INFO:tasks.rados.rados.0.smithi160.stdout:258: writing smithi16030472-258 from 719399 to 1498041 tid 1 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:258: writing smithi16030472-258 from 1939284 to 2319399 tid 2 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:244: finishing write tid 1 to smithi16030472-244 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:244: finishing write tid 2 to smithi16030472-244 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:244: finishing write tid 3 to smithi16030472-244 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: finishing write tid 1 to smithi16030472-245 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: finishing write tid 2 to smithi16030472-245 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: finishing write tid 3 to smithi16030472-245 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:243: finishing write tid 3 to smithi16030472-243 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: finishing write tid 4 to smithi16030472-245 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: oid 245 updating version 0 to 178 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: oid 245 version 178 is already newer than 176 2024-09-21T18:07:26.591 INFO:tasks.rados.rados.0.smithi160.stdout:245: oid 245 version 178 is already newer than 177 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 245 v 178 (ObjNum 244 snap 0 seq_num 244) dirty exists 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:245: left oid 245 (ObjNum 244 snap 0 seq_num 244) 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: finishing write tid 4 to smithi16030472-243 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: oid 243 updating version 0 to 137 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: oid 243 version 137 is already newer than 136 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: oid 243 updating version 137 to 138 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 138 (ObjNum 242 snap 0 seq_num 242) dirty exists 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: left oid 243 (ObjNum 242 snap 0 seq_num 242) 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:243: done (15 left) 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:245: done (14 left) 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:259: write initial oid 259 2024-09-21T18:07:26.592 INFO:tasks.rados.rados.0.smithi160.stdout:259: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-21T18:07:26.599 INFO:tasks.rados.rados.0.smithi160.stdout:259: writing smithi16030472-259 from 736206 to 1190097 tid 1 2024-09-21T18:07:26.607 INFO:tasks.rados.rados.0.smithi160.stdout:259: writing smithi16030472-259 from 1881413 to 2336206 tid 2 2024-09-21T18:07:26.607 INFO:tasks.rados.rados.0.smithi160.stdout:260: write initial oid 260 2024-09-21T18:07:26.607 INFO:tasks.rados.rados.0.smithi160.stdout:260: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-21T18:07:26.617 INFO:tasks.rados.rados.0.smithi160.stdout:260: writing smithi16030472-260 from 753013 to 1398506 tid 1 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:260: writing smithi16030472-260 from 2056248 to 2353013 tid 2 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:251: finishing write tid 1 to smithi16030472-251 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:251: finishing write tid 2 to smithi16030472-251 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: finishing write tid 4 to smithi16030472-234 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: finishing write tid 5 to smithi16030472-234 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: oid 234 updating version 0 to 125 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: oid 234 version 125 is already newer than 123 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: oid 234 updating version 125 to 127 2024-09-21T18:07:26.623 INFO:tasks.rados.rados.0.smithi160.stdout:234: oid 234 version 127 is already newer than 124 2024-09-21T18:07:26.624 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 234 v 127 (ObjNum 233 snap 0 seq_num 233) dirty exists 2024-09-21T18:07:26.624 INFO:tasks.rados.rados.0.smithi160.stdout:234: left oid 234 (ObjNum 233 snap 0 seq_num 233) 2024-09-21T18:07:26.624 INFO:tasks.rados.rados.0.smithi160.stdout:234: done (15 left) 2024-09-21T18:07:26.624 INFO:tasks.rados.rados.0.smithi160.stdout:261: write initial oid 261 2024-09-21T18:07:26.624 INFO:tasks.rados.rados.0.smithi160.stdout:261: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-21T18:07:26.625 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:26.622+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:26.631 INFO:tasks.rados.rados.0.smithi160.stdout:261: writing smithi16030472-261 from 769820 to 1490562 tid 1 2024-09-21T18:07:26.636 INFO:tasks.rados.rados.0.smithi160.stdout:261: writing smithi16030472-261 from 2114730 to 2369820 tid 2 2024-09-21T18:07:26.636 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.637 INFO:tasks.rados.rados.0.smithi160.stdout:255: finishing write tid 1 to smithi16030472-255 2024-09-21T18:07:26.638 INFO:tasks.rados.rados.0.smithi160.stdout:255: finishing write tid 2 to smithi16030472-255 2024-09-21T18:07:26.706 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:26.701+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:26.711 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:26.711 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:26.711 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:26.711 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:26.725 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:26.735 INFO:tasks.rados.rados.0.smithi160.stdout:255: finishing write tid 3 to smithi16030472-255 2024-09-21T18:07:26.739 INFO:tasks.rados.rados.0.smithi160.stdout:255: finishing write tid 4 to smithi16030472-255 2024-09-21T18:07:26.739 INFO:tasks.rados.rados.0.smithi160.stdout:255: oid 255 updating version 0 to 162 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:255: oid 255 updating version 162 to 163 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:255: oid 255 version 163 is already newer than 161 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 255 v 163 (ObjNum 254 snap 0 seq_num 254) dirty exists 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:255: left oid 255 (ObjNum 254 snap 0 seq_num 254) 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:255: done (15 left) 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:262: write initial oid 262 2024-09-21T18:07:26.740 INFO:tasks.rados.rados.0.smithi160.stdout:262: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-21T18:07:26.749 INFO:tasks.rados.rados.0.smithi160.stdout:262: writing smithi16030472-262 from 786627 to 1582618 tid 1 2024-09-21T18:07:26.754 INFO:tasks.rados.rados.0.smithi160.stdout:262: writing smithi16030472-262 from 2173212 to 2386627 tid 2 2024-09-21T18:07:26.754 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.754 INFO:tasks.rados.rados.0.smithi160.stdout:244: finishing write tid 4 to smithi16030472-244 2024-09-21T18:07:26.754 INFO:tasks.rados.rados.0.smithi160.stdout:244: oid 244 updating version 0 to 210 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:244: oid 244 updating version 210 to 211 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:244: oid 244 updating version 211 to 212 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 244 v 212 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:244: left oid 244 (ObjNum 243 snap 0 seq_num 243) 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:244: done (15 left) 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:263: write initial oid 263 2024-09-21T18:07:26.755 INFO:tasks.rados.rados.0.smithi160.stdout:263: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-21T18:07:26.759 INFO:tasks.rados.rados.0.smithi160.stdout:263: writing smithi16030472-263 from 403434 to 874674 tid 1 2024-09-21T18:07:26.764 INFO:tasks.rados.rados.0.smithi160.stdout:263: writing smithi16030472-263 from 1315341 to 1716097 tid 2 2024-09-21T18:07:26.764 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.764 INFO:tasks.rados.rados.0.smithi160.stdout:251: finishing write tid 3 to smithi16030472-251 2024-09-21T18:07:26.764 INFO:tasks.rados.rados.0.smithi160.stdout:251: finishing write tid 4 to smithi16030472-251 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:251: oid 251 updating version 0 to 179 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:251: oid 251 updating version 179 to 180 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:251: oid 251 updating version 180 to 181 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 251 v 181 (ObjNum 250 snap 0 seq_num 250) dirty exists 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:251: left oid 251 (ObjNum 250 snap 0 seq_num 250) 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:253: finishing write tid 1 to smithi16030472-253 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:251: done (15 left) 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:264: write initial oid 264 2024-09-21T18:07:26.765 INFO:tasks.rados.rados.0.smithi160.stdout:264: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-21T18:07:26.769 INFO:tasks.rados.rados.0.smithi160.stdout:264: writing smithi16030472-264 from 420241 to 966730 tid 1 2024-09-21T18:07:26.775 INFO:tasks.rados.rados.0.smithi160.stdout:264: writing smithi16030472-264 from 1373823 to 1918237 tid 2 2024-09-21T18:07:26.776 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.776 INFO:tasks.rados.rados.0.smithi160.stdout:253: finishing write tid 2 to smithi16030472-253 2024-09-21T18:07:26.776 INFO:tasks.rados.rados.0.smithi160.stdout:253: finishing write tid 3 to smithi16030472-253 2024-09-21T18:07:26.807 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:26.804+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: finishing write tid 1 to smithi16030472-246 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: finishing write tid 2 to smithi16030472-246 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: finishing write tid 3 to smithi16030472-246 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: finishing write tid 4 to smithi16030472-246 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: oid 246 updating version 0 to 141 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: oid 246 version 141 is already newer than 139 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: oid 246 version 141 is already newer than 140 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 246 v 141 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: left oid 246 (ObjNum 245 snap 0 seq_num 245) 2024-09-21T18:07:26.863 INFO:tasks.rados.rados.0.smithi160.stdout:246: done (15 left) 2024-09-21T18:07:26.864 INFO:tasks.rados.rados.0.smithi160.stdout:265: write initial oid 265 2024-09-21T18:07:26.864 INFO:tasks.rados.rados.0.smithi160.stdout:265: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-21T18:07:26.870 INFO:tasks.rados.rados.0.smithi160.stdout:265: writing smithi16030472-265 from 437048 to 1058786 tid 1 2024-09-21T18:07:26.879 INFO:tasks.rados.rados.0.smithi160.stdout:265: writing smithi16030472-265 from 1832305 to 2236730 tid 2 2024-09-21T18:07:26.879 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:26.894 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:26.894 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:26.894 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:26.895 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:26.895 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:26.908 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:26.904+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:26.910 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:26.962 INFO:tasks.rados.rados.0.smithi160.stdout:262: finishing write tid 1 to smithi16030472-262 2024-09-21T18:07:26.962 INFO:tasks.rados.rados.0.smithi160.stdout:262: finishing write tid 2 to smithi16030472-262 2024-09-21T18:07:27.009 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:27.004+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:27.038 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:27.038 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:27.038 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:482 3.3 e46/31 3:ccecab68:::smithi16030472-109:head v 46'66, mlcod=46'58)", 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.9170534539999995, 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212039585, 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.039 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118628+0000", 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.040 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.118630+0000", 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8640000000000001e-06 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119297+0000", 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066705300000000003 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.041 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119316+0000", 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9245999999999998e-05 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.119327+0000", 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0842999999999999e-05 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.042 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124681+0000", 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0053538709999999996 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.124708+0000", 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6931999999999999e-05 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330606+0000", 2024-09-21T18:07:27.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20589761400000001 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330668+0000", 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2162000000000006e-05 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:27.044 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.704762916, 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.045 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:07:27.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:07:27.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.11.0:320 4.1 e48/33 4:b502f07b:::smithi16030472-206:head v 48'39, mlcod=48'37)", 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:23.929189+0000", 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 3.106492517, 2024-09-21T18:07:27.049 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212693718, 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.929189+0000", 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.050 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.929189+0000", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.929193+0000", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7270000000000001e-06 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.935248+0000", 2024-09-21T18:07:27.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060552180000000002 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.935283+0000", 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4476e-05 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:23.935301+0000", 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7966999999999999e-05 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.135709+0000", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20040798300000001 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.135781+0000", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2185000000000001e-05 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.141789+0000", 2024-09-21T18:07:27.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060083869999999996 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.141883+0000", 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3775000000000003e-05 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:07:27.054 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.8649109199999998, 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.058 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:07:27.059 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:946 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'148, mlcod=48'143)", 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.271975+0000", 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.763707095, 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21167150900000001, 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271975+0000", 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271975+0000", 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271977+0000", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7470000000000001e-06 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.272744+0000", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00076630700000000003 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.272769+0000", 2024-09-21T18:07:27.062 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5162999999999999e-05 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.272788+0000", 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8729e-05 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.374420+0000", 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10163222600000001 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.063 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.374457+0000", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7385999999999997e-05 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483553+0000", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109096046 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483646+0000", 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2905000000000006e-05 2024-09-21T18:07:27.064 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:948 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'150, mlcod=48'145)", 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.273620+0000", 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.7620618970000002, 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21257763599999999, 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.065 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273620+0000", 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273620+0000", 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.066 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273621+0000", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4729999999999999e-06 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273632+0000", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.095e-05 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273650+0000", 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7669999999999999e-05 2024-09-21T18:07:27.067 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273663+0000", 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3169999999999999e-05 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.478919+0000", 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205256198 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.068 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.478952+0000", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2391999999999998e-05 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486106+0000", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0071544249999999998 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486197+0000", 2024-09-21T18:07:27.069 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1358000000000002e-05 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:955 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'151, mlcod=48'145)", 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.273682+0000", 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.7620000949999999, 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21259088500000001, 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.070 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273682+0000", 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273682+0000", 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.071 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.273683+0000", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-06 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274440+0000", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00075723100000000003 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274458+0000", 2024-09-21T18:07:27.072 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7462e-05 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274471+0000", 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3403e-05 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479518+0000", 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20504684000000001 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.073 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479557+0000", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8772000000000001e-05 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486205+0000", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0066482720000000002 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486272+0000", 2024-09-21T18:07:27.074 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7655000000000002e-05 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.761190472, 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.075 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.076 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:07:27.077 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.078 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:07:27.079 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.760415729, 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.080 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.084 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.7597677100000002, 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.085 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:07:27.086 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.087 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:07:27.088 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:07:27.089 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.745896015, 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.090 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.091 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:07:27.092 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.093 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.094 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.7429549990000002, 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.095 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.096 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.097 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:07:27.098 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.099 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.739938151, 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.100 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.101 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.102 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:07:27.103 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.104 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.7377265689999999, 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.105 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:27.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:975 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'159, mlcod=48'145)", 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.400175+0000", 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.6355065259999999, 2024-09-21T18:07:27.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29601684900000003, 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.400175+0000", 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.400175+0000", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.400176+0000", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6599999999999996e-07 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.400996+0000", 2024-09-21T18:07:27.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00082038599999999999 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.401022+0000", 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5409999999999999e-05 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.401043+0000", 2024-09-21T18:07:27.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1798999999999999e-05 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593958+0000", 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.192914744 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593990+0000", 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.218e-05 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696155+0000", 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102164908 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696192+0000", 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6856000000000001e-05 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.115 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.6323894770000003, 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.116 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:07:27.117 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:07:27.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.120 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:977 3.3 e49/31 3:f8256ac6:::smithi16030472-219:head v 49'161, mlcod=48'145)", 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.580449+0000", 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.455232997, 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.216467094, 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.580449+0000", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.580449+0000", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.580452+0000", 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0800000000000002e-06 2024-09-21T18:07:27.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.580979+0000", 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052738800000000005 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581008+0000", 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8989999999999999e-05 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581027+0000", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9106e-05 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696314+0000", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11528707000000001 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696369+0000", 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4632000000000001e-05 2024-09-21T18:07:27.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796882+0000", 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100513238 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796916+0000", 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3590000000000002e-05 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.125 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:978 3.3 e49/31 3:f8256ac6:::smithi16030472-219:head v 49'162, mlcod=48'145)", 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.581054+0000", 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.4546272899999999, 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21657932799999999, 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581054+0000", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581054+0000", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581056+0000", 2024-09-21T18:07:27.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.497e-06 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581063+0000", 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2259999999999999e-06 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581072+0000", 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.8200000000000003e-06 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.581079+0000", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.469e-06 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796393+0000", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21531333599999999 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796427+0000", 2024-09-21T18:07:27.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.797552+0000", 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001124683 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.797634+0000", 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1800999999999998e-05 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.130 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.940967103, 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:07:27.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:07:27.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.135 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.9379371029999999, 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:27.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:07:27.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:27.140 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:27.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:27.143 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:27.144 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:27.138+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:27.144 INFO:tasks.rados.rados.0.smithi160.stdout:254: finishing write tid 1 to smithi16030472-254 2024-09-21T18:07:27.144 INFO:tasks.rados.rados.0.smithi160.stdout:254: finishing write tid 2 to smithi16030472-254 2024-09-21T18:07:27.144 INFO:tasks.rados.rados.0.smithi160.stdout:254: finishing write tid 3 to smithi16030472-254 2024-09-21T18:07:27.144 INFO:tasks.rados.rados.0.smithi160.stdout:254: finishing write tid 4 to smithi16030472-254 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:254: oid 254 updating version 0 to 214 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:254: oid 254 version 214 is already newer than 213 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:254: oid 254 updating version 214 to 215 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 254 v 215 (ObjNum 253 snap 0 seq_num 253) dirty exists 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:254: left oid 254 (ObjNum 253 snap 0 seq_num 253) 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:256: finishing write tid 1 to smithi16030472-256 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:256: finishing write tid 2 to smithi16030472-256 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:254: done (15 left) 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:266: write initial oid 266 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:266: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:266: writing smithi16030472-266 from 453855 to 1150842 tid 1 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:266: writing smithi16030472-266 from 1890787 to 2438870 tid 2 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.145 INFO:tasks.rados.rados.0.smithi160.stdout:248: finishing write tid 1 to smithi16030472-248 2024-09-21T18:07:27.146 INFO:tasks.rados.rados.0.smithi160.stdout:248: finishing write tid 2 to smithi16030472-248 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:253: finishing write tid 4 to smithi16030472-253 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:253: oid 253 updating version 0 to 184 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:253: oid 253 version 184 is already newer than 183 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:253: oid 253 version 184 is already newer than 182 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 253 v 184 (ObjNum 252 snap 0 seq_num 252) dirty exists 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:253: left oid 253 (ObjNum 252 snap 0 seq_num 252) 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:257: finishing write tid 1 to smithi16030472-257 2024-09-21T18:07:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:257: finishing write tid 2 to smithi16030472-257 2024-09-21T18:07:27.158 INFO:tasks.rados.rados.0.smithi160.stdout:257: finishing write tid 3 to smithi16030472-257 2024-09-21T18:07:27.158 INFO:tasks.rados.rados.0.smithi160.stdout:253: done (15 left) 2024-09-21T18:07:27.158 INFO:tasks.rados.rados.0.smithi160.stdout:267: write initial oid 267 2024-09-21T18:07:27.158 INFO:tasks.rados.rados.0.smithi160.stdout:267: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-21T18:07:27.169 INFO:tasks.rados.rados.0.smithi160.stdout:267: writing smithi16030472-267 from 470662 to 1242898 tid 1 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:267: writing smithi16030472-267 from 1832916 to 2241010 tid 2 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:257: finishing write tid 4 to smithi16030472-257 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:257: oid 257 updating version 0 to 185 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:257: oid 257 updating version 185 to 186 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:257: oid 257 updating version 186 to 187 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 257 v 187 (ObjNum 256 snap 0 seq_num 256) dirty exists 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:257: left oid 257 (ObjNum 256 snap 0 seq_num 256) 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:249: finishing write tid 1 to smithi16030472-249 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:249: finishing write tid 2 to smithi16030472-249 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:249: finishing write tid 3 to smithi16030472-249 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:249: finishing write tid 4 to smithi16030472-249 2024-09-21T18:07:27.180 INFO:tasks.rados.rados.0.smithi160.stdout:249: oid 249 updating version 0 to 146 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:249: oid 249 version 146 is already newer than 145 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:249: oid 249 version 146 is already newer than 144 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 249 v 146 (ObjNum 248 snap 0 seq_num 248) dirty exists 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:249: left oid 249 (ObjNum 248 snap 0 seq_num 248) 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:250: finishing write tid 1 to smithi16030472-250 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:252: finishing write tid 1 to smithi16030472-252 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: finishing write tid 3 to smithi16030472-262 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: finishing write tid 4 to smithi16030472-262 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: oid 262 updating version 0 to 165 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: oid 262 version 165 is already newer than 164 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: oid 262 updating version 165 to 166 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 262 v 166 (ObjNum 261 snap 0 seq_num 261) dirty exists 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:262: left oid 262 (ObjNum 261 snap 0 seq_num 261) 2024-09-21T18:07:27.181 INFO:tasks.rados.rados.0.smithi160.stdout:249: done (15 left) 2024-09-21T18:07:27.182 INFO:tasks.rados.rados.0.smithi160.stdout:257: done (14 left) 2024-09-21T18:07:27.182 INFO:tasks.rados.rados.0.smithi160.stdout:262: done (13 left) 2024-09-21T18:07:27.182 INFO:tasks.rados.rados.0.smithi160.stdout:268: write initial oid 268 2024-09-21T18:07:27.182 INFO:tasks.rados.rados.0.smithi160.stdout:268: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-21T18:07:27.191 INFO:tasks.rados.rados.0.smithi160.stdout:268: writing smithi16030472-268 from 487469 to 1051307 tid 1 2024-09-21T18:07:27.204 INFO:tasks.rados.rados.0.smithi160.stdout:268: writing smithi16030472-268 from 1607751 to 2159503 tid 2 2024-09-21T18:07:27.204 INFO:tasks.rados.rados.0.smithi160.stdout:269: write initial oid 269 2024-09-21T18:07:27.204 INFO:tasks.rados.rados.0.smithi160.stdout:269: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-21T18:07:27.215 INFO:tasks.rados.rados.0.smithi160.stdout:269: writing smithi16030472-269 from 504276 to 1143363 tid 1 2024-09-21T18:07:27.225 INFO:tasks.rados.rados.0.smithi160.stdout:269: writing smithi16030472-269 from 1666233 to 2361643 tid 2 2024-09-21T18:07:27.225 INFO:tasks.rados.rados.0.smithi160.stdout:270: write initial oid 270 2024-09-21T18:07:27.225 INFO:tasks.rados.rados.0.smithi160.stdout:270: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-21T18:07:27.234 INFO:tasks.rados.rados.0.smithi160.stdout:270: writing smithi16030472-270 from 521083 to 1235419 tid 1 2024-09-21T18:07:27.241 INFO:tasks.rados.rados.0.smithi160.stdout:270: writing smithi16030472-270 from 1724715 to 2280136 tid 2 2024-09-21T18:07:27.241 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.243 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:27.238+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:256: finishing write tid 3 to smithi16030472-256 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:256: finishing write tid 4 to smithi16030472-256 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:256: oid 256 updating version 0 to 217 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:256: oid 256 updating version 217 to 218 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:256: oid 256 version 218 is already newer than 216 2024-09-21T18:07:27.247 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 256 v 218 (ObjNum 255 snap 0 seq_num 255) dirty exists 2024-09-21T18:07:27.248 INFO:tasks.rados.rados.0.smithi160.stdout:256: left oid 256 (ObjNum 255 snap 0 seq_num 255) 2024-09-21T18:07:27.248 INFO:tasks.rados.rados.0.smithi160.stdout:258: finishing write tid 1 to smithi16030472-258 2024-09-21T18:07:27.248 INFO:tasks.rados.rados.0.smithi160.stdout:256: done (15 left) 2024-09-21T18:07:27.248 INFO:tasks.rados.rados.0.smithi160.stdout:271: write initial oid 271 2024-09-21T18:07:27.248 INFO:tasks.rados.rados.0.smithi160.stdout:271: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-21T18:07:27.256 INFO:tasks.rados.rados.0.smithi160.stdout:271: writing smithi16030472-271 from 537890 to 1327475 tid 1 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:271: writing smithi16030472-271 from 1783197 to 2482276 tid 2 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: finishing write tid 2 to smithi16030472-258 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: finishing write tid 3 to smithi16030472-258 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: finishing write tid 4 to smithi16030472-258 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: oid 258 updating version 0 to 220 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: oid 258 version 220 is already newer than 219 2024-09-21T18:07:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:258: oid 258 updating version 220 to 221 2024-09-21T18:07:27.265 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 221 (ObjNum 257 snap 0 seq_num 257) dirty exists 2024-09-21T18:07:27.265 INFO:tasks.rados.rados.0.smithi160.stdout:258: left oid 258 (ObjNum 257 snap 0 seq_num 257) 2024-09-21T18:07:27.265 INFO:tasks.rados.rados.0.smithi160.stdout:258: done (15 left) 2024-09-21T18:07:27.265 INFO:tasks.rados.rados.0.smithi160.stdout:272: write initial oid 272 2024-09-21T18:07:27.265 INFO:tasks.rados.rados.0.smithi160.stdout:272: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-21T18:07:27.271 INFO:tasks.rados.rados.0.smithi160.stdout:272: writing smithi16030472-272 from 554697 to 1019531 tid 1 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout:272: writing smithi16030472-272 from 1725326 to 2284416 tid 2 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout:252: finishing write tid 2 to smithi16030472-252 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout:248: finishing write tid 3 to smithi16030472-248 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout:248: finishing write tid 4 to smithi16030472-248 2024-09-21T18:07:27.281 INFO:tasks.rados.rados.0.smithi160.stdout:248: oid 248 updating version 0 to 143 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:248: oid 248 version 143 is already newer than 142 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:248: oid 248 updating version 143 to 144 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 248 v 144 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:248: left oid 248 (ObjNum 247 snap 0 seq_num 247) 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:252: finishing write tid 3 to smithi16030472-252 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:248: done (15 left) 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:273: write initial oid 273 2024-09-21T18:07:27.282 INFO:tasks.rados.rados.0.smithi160.stdout:273: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-21T18:07:27.288 INFO:tasks.rados.rados.0.smithi160.stdout:273: writing smithi16030472-273 from 571504 to 1111587 tid 1 2024-09-21T18:07:27.299 INFO:tasks.rados.rados.0.smithi160.stdout:273: writing smithi16030472-273 from 1783808 to 2486556 tid 2 2024-09-21T18:07:27.299 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.299 INFO:tasks.rados.rados.0.smithi160.stdout:252: finishing write tid 4 to smithi16030472-252 2024-09-21T18:07:27.299 INFO:tasks.rados.rados.0.smithi160.stdout:252: oid 252 updating version 0 to 150 2024-09-21T18:07:27.299 INFO:tasks.rados.rados.0.smithi160.stdout:252: oid 252 version 150 is already newer than 148 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:252: oid 252 version 150 is already newer than 149 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 252 v 150 (ObjNum 251 snap 0 seq_num 251) dirty exists 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:252: left oid 252 (ObjNum 251 snap 0 seq_num 251) 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: finishing write tid 1 to smithi16030472-261 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: finishing write tid 2 to smithi16030472-261 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: finishing write tid 3 to smithi16030472-261 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: finishing write tid 4 to smithi16030472-261 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: oid 261 updating version 0 to 152 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: oid 261 version 152 is already newer than 151 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: oid 261 updating version 152 to 153 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 261 v 153 (ObjNum 260 snap 0 seq_num 260) dirty exists 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:261: left oid 261 (ObjNum 260 snap 0 seq_num 260) 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:263: finishing write tid 1 to smithi16030472-263 2024-09-21T18:07:27.300 INFO:tasks.rados.rados.0.smithi160.stdout:263: finishing write tid 2 to smithi16030472-263 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: finishing write tid 3 to smithi16030472-263 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: finishing write tid 4 to smithi16030472-263 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: oid 263 updating version 0 to 156 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: oid 263 version 156 is already newer than 155 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: oid 263 version 156 is already newer than 154 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 263 v 156 (ObjNum 262 snap 0 seq_num 262) dirty exists 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:263: left oid 263 (ObjNum 262 snap 0 seq_num 262) 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: finishing write tid 1 to smithi16030472-264 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: finishing write tid 2 to smithi16030472-264 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: finishing write tid 3 to smithi16030472-264 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: finishing write tid 4 to smithi16030472-264 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: oid 264 updating version 0 to 157 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: oid 264 updating version 157 to 159 2024-09-21T18:07:27.301 INFO:tasks.rados.rados.0.smithi160.stdout:264: oid 264 version 159 is already newer than 158 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 159 (ObjNum 263 snap 0 seq_num 263) dirty exists 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:264: left oid 264 (ObjNum 263 snap 0 seq_num 263) 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:252: done (15 left) 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:261: done (14 left) 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:263: done (13 left) 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:264: done (12 left) 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:274: write initial oid 274 2024-09-21T18:07:27.302 INFO:tasks.rados.rados.0.smithi160.stdout:274: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-21T18:07:27.302 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:27.303 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:274: writing smithi16030472-274 from 588311 to 1203643 tid 1 2024-09-21T18:07:27.313 INFO:tasks.rados.rados.0.smithi160.stdout:274: writing smithi16030472-274 from 1842290 to 2405049 tid 2 2024-09-21T18:07:27.313 INFO:tasks.rados.rados.0.smithi160.stdout:275: write initial oid 275 2024-09-21T18:07:27.313 INFO:tasks.rados.rados.0.smithi160.stdout:275: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-21T18:07:27.315 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:27.315 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:27.316 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:27.316 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:07:27.318 INFO:tasks.rados.rados.0.smithi160.stdout:275: writing smithi16030472-275 from 605118 to 1012052 tid 1 2024-09-21T18:07:27.323 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:27.324 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:27.326 INFO:tasks.rados.rados.0.smithi160.stdout:275: writing smithi16030472-275 from 1617125 to 2323542 tid 2 2024-09-21T18:07:27.326 INFO:tasks.rados.rados.0.smithi160.stdout:276: write initial oid 276 2024-09-21T18:07:27.327 INFO:tasks.rados.rados.0.smithi160.stdout:276: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-21T18:07:27.331 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:27.332 INFO:tasks.rados.rados.0.smithi160.stdout:276: writing smithi16030472-276 from 621925 to 1104108 tid 1 2024-09-21T18:07:27.339 INFO:tasks.rados.rados.0.smithi160.stdout:276: writing smithi16030472-276 from 1559254 to 2125682 tid 2 2024-09-21T18:07:27.339 INFO:tasks.rados.rados.0.smithi160.stdout:277: write initial oid 277 2024-09-21T18:07:27.339 INFO:tasks.rados.rados.0.smithi160.stdout:277: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-21T18:07:27.344 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:27.341+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:27.345 INFO:tasks.rados.rados.0.smithi160.stdout:277: writing smithi16030472-277 from 638732 to 1196164 tid 1 2024-09-21T18:07:27.351 INFO:tasks.rados.rados.0.smithi160.stdout:277: writing smithi16030472-277 from 1617736 to 2327822 tid 2 2024-09-21T18:07:27.351 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.375 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:27.376 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:27.396 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:27.396 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:27.412 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:27.413 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:27.422 INFO:tasks.rados.rados.0.smithi160.stdout:259: finishing write tid 1 to smithi16030472-259 2024-09-21T18:07:27.422 INFO:tasks.rados.rados.0.smithi160.stdout:259: finishing write tid 2 to smithi16030472-259 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: finishing write tid 3 to smithi16030472-259 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: finishing write tid 4 to smithi16030472-259 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: oid 259 updating version 0 to 223 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: oid 259 version 223 is already newer than 222 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: oid 259 updating version 223 to 224 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 259 v 224 (ObjNum 258 snap 0 seq_num 258) dirty exists 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: left oid 259 (ObjNum 258 snap 0 seq_num 258) 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:259: done (15 left) 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:278: write initial oid 278 2024-09-21T18:07:27.423 INFO:tasks.rados.rados.0.smithi160.stdout:278: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-21T18:07:27.432 INFO:tasks.rados.rados.0.smithi160.stdout:278: writing smithi16030472-278 from 655539 to 1288220 tid 1 2024-09-21T18:07:27.433 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:27.433 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:278: writing smithi16030472-278 from 2076218 to 2646315 tid 2 2024-09-21T18:07:27.439 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.445 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:27.441+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:27.453 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:27.453 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:27.454 INFO:tasks.rados.rados.0.smithi160.stdout:265: finishing write tid 1 to smithi16030472-265 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: finishing write tid 2 to smithi16030472-265 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: finishing write tid 3 to smithi16030472-265 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: finishing write tid 4 to smithi16030472-265 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: oid 265 updating version 0 to 226 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: oid 265 updating version 226 to 227 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: oid 265 version 227 is already newer than 225 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 265 v 227 (ObjNum 264 snap 0 seq_num 264) dirty exists 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: left oid 265 (ObjNum 264 snap 0 seq_num 264) 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:265: done (15 left) 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:279: write initial oid 279 2024-09-21T18:07:27.455 INFO:tasks.rados.rados.0.smithi160.stdout:279: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-21T18:07:27.458 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:27.458 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:27.462 INFO:tasks.rados.rados.0.smithi160.stdout:279: writing smithi16030472-279 from 672346 to 1380276 tid 1 2024-09-21T18:07:27.469 INFO:tasks.rados.rados.0.smithi160.stdout:279: writing smithi16030472-279 from 2134700 to 2672346 tid 2 2024-09-21T18:07:27.469 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:27.515 INFO:tasks.rados.rados.0.smithi160.stdout:260: finishing write tid 1 to smithi16030472-260 2024-09-21T18:07:27.516 INFO:tasks.rados.rados.0.smithi160.stdout:260: finishing write tid 2 to smithi16030472-260 2024-09-21T18:07:27.516 INFO:tasks.rados.rados.0.smithi160.stdout:260: finishing write tid 3 to smithi16030472-260 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: finishing write tid 2 to smithi16030472-250 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: finishing write tid 3 to smithi16030472-250 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: finishing write tid 4 to smithi16030472-250 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: oid 250 updating version 0 to 148 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: oid 250 version 148 is already newer than 147 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: oid 250 updating version 148 to 149 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 250 v 149 (ObjNum 249 snap 0 seq_num 249) dirty exists 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: left oid 250 (ObjNum 249 snap 0 seq_num 249) 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:250: done (15 left) 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:280: write initial oid 280 2024-09-21T18:07:27.518 INFO:tasks.rados.rados.0.smithi160.stdout:280: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-21T18:07:27.526 INFO:tasks.rados.rados.0.smithi160.stdout:280: writing smithi16030472-280 from 689153 to 1472332 tid 1 2024-09-21T18:07:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:280: writing smithi16030472-280 from 2076829 to 2534242 tid 2 2024-09-21T18:07:27.533 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.542 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:27.545 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:27.540+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:27.568 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:27.568 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:27.596 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:27.596 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:27.599 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:27.610 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:27.618 INFO:tasks.rados.rados.0.smithi160.stdout:260: finishing write tid 4 to smithi16030472-260 2024-09-21T18:07:27.618 INFO:tasks.rados.rados.0.smithi160.stdout:260: oid 260 updating version 0 to 188 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:260: oid 260 updating version 188 to 190 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:260: oid 260 version 190 is already newer than 189 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 260 v 190 (ObjNum 259 snap 0 seq_num 259) dirty exists 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:260: left oid 260 (ObjNum 259 snap 0 seq_num 259) 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:268: finishing write tid 1 to smithi16030472-268 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:268: finishing write tid 2 to smithi16030472-268 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:268: finishing write tid 3 to smithi16030472-268 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:260: done (15 left) 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:281: write initial oid 281 2024-09-21T18:07:27.619 INFO:tasks.rados.rados.0.smithi160.stdout:281: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-21T18:07:27.626 INFO:tasks.rados.rados.0.smithi160.stdout:281: writing smithi16030472-281 from 705960 to 1164388 tid 1 2024-09-21T18:07:27.627 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:27.627 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:27.635 INFO:tasks.rados.rados.0.smithi160.stdout:281: writing smithi16030472-281 from 1735311 to 2452735 tid 2 2024-09-21T18:07:27.635 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.635 INFO:tasks.rados.rados.0.smithi160.stdout:268: finishing write tid 4 to smithi16030472-268 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:268: oid 268 updating version 0 to 192 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:268: oid 268 updating version 192 to 193 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:268: oid 268 version 193 is already newer than 191 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 268 v 193 (ObjNum 267 snap 0 seq_num 267) dirty exists 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:268: left oid 268 (ObjNum 267 snap 0 seq_num 267) 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: finishing write tid 1 to smithi16030472-273 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: finishing write tid 2 to smithi16030472-273 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: finishing write tid 3 to smithi16030472-273 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: finishing write tid 4 to smithi16030472-273 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: oid 273 updating version 0 to 194 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: oid 273 updating version 194 to 195 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: oid 273 updating version 195 to 196 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 273 v 196 (ObjNum 272 snap 0 seq_num 272) dirty exists 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:273: left oid 273 (ObjNum 272 snap 0 seq_num 272) 2024-09-21T18:07:27.636 INFO:tasks.rados.rados.0.smithi160.stdout:268: done (15 left) 2024-09-21T18:07:27.637 INFO:tasks.rados.rados.0.smithi160.stdout:273: done (14 left) 2024-09-21T18:07:27.637 INFO:tasks.rados.rados.0.smithi160.stdout:282: write initial oid 282 2024-09-21T18:07:27.637 INFO:tasks.rados.rados.0.smithi160.stdout:282: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-21T18:07:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:282: writing smithi16030472-282 from 722767 to 1256444 tid 1 2024-09-21T18:07:27.646 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:27.641+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:27.649 INFO:tasks.rados.rados.0.smithi160.stdout:282: writing smithi16030472-282 from 1793793 to 2254875 tid 2 2024-09-21T18:07:27.650 INFO:tasks.rados.rados.0.smithi160.stdout:283: write initial oid 283 2024-09-21T18:07:27.650 INFO:tasks.rados.rados.0.smithi160.stdout:283: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-21T18:07:27.655 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:27.655 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:27.659 INFO:tasks.rados.rados.0.smithi160.stdout:283: writing smithi16030472-283 from 739574 to 1464853 tid 1 2024-09-21T18:07:27.667 INFO:tasks.rados.rados.0.smithi160.stdout:283: writing smithi16030472-283 from 1968628 to 2689721 tid 2 2024-09-21T18:07:27.667 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.682 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:27.682 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:27.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:27.719 INFO:tasks.rados.rados.0.smithi160.stdout:275: finishing write tid 1 to smithi16030472-275 2024-09-21T18:07:27.720 INFO:tasks.rados.rados.0.smithi160.stdout:275: finishing write tid 2 to smithi16030472-275 2024-09-21T18:07:27.722 INFO:tasks.rados.rados.0.smithi160.stdout:270: finishing write tid 1 to smithi16030472-270 2024-09-21T18:07:27.723 INFO:tasks.rados.rados.0.smithi160.stdout:275: finishing write tid 3 to smithi16030472-275 2024-09-21T18:07:27.726 INFO:tasks.rados.rados.0.smithi160.stdout:270: finishing write tid 2 to smithi16030472-270 2024-09-21T18:07:27.726 INFO:tasks.rados.rados.0.smithi160.stdout:270: finishing write tid 3 to smithi16030472-270 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: finishing write tid 4 to smithi16030472-270 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: oid 270 updating version 0 to 162 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: oid 270 version 162 is already newer than 160 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: oid 270 version 162 is already newer than 161 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 270 v 162 (ObjNum 269 snap 0 seq_num 269) dirty exists 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: left oid 270 (ObjNum 269 snap 0 seq_num 269) 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:270: done (15 left) 2024-09-21T18:07:27.727 INFO:tasks.rados.rados.0.smithi160.stdout:284: write initial oid 284 2024-09-21T18:07:27.728 INFO:tasks.rados.rados.0.smithi160.stdout:284: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-21T18:07:27.734 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:27.734 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:27.735 INFO:tasks.rados.rados.0.smithi160.stdout:284: writing smithi16030472-284 from 756381 to 1156909 tid 1 2024-09-21T18:07:27.737 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:27.737 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:27.744 INFO:tasks.rados.rados.0.smithi160.stdout:284: writing smithi16030472-284 from 1910757 to 2375508 tid 2 2024-09-21T18:07:27.744 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.744 INFO:tasks.rados.rados.0.smithi160.stdout:274: finishing write tid 1 to smithi16030472-274 2024-09-21T18:07:27.744 INFO:tasks.rados.rados.0.smithi160.stdout:274: finishing write tid 2 to smithi16030472-274 2024-09-21T18:07:27.744 INFO:tasks.rados.rados.0.smithi160.stdout:274: finishing write tid 3 to smithi16030472-274 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: finishing write tid 4 to smithi16030472-274 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: oid 274 updating version 0 to 165 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: oid 274 version 165 is already newer than 163 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: oid 274 version 165 is already newer than 164 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 274 v 165 (ObjNum 273 snap 0 seq_num 273) dirty exists 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: left oid 274 (ObjNum 273 snap 0 seq_num 273) 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:266: finishing write tid 1 to smithi16030472-266 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:266: finishing write tid 2 to smithi16030472-266 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:266: finishing write tid 3 to smithi16030472-266 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:267: finishing write tid 1 to smithi16030472-267 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:274: done (15 left) 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:285: write initial oid 285 2024-09-21T18:07:27.745 INFO:tasks.rados.rados.0.smithi160.stdout:285: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-21T18:07:27.747 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:27.744+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:27.749 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:27.752 INFO:tasks.rados.rados.0.smithi160.stdout:285: writing smithi16030472-285 from 773188 to 1248965 tid 1 2024-09-21T18:07:27.762 INFO:tasks.rados.rados.0.smithi160.stdout:285: writing smithi16030472-285 from 1969239 to 2694001 tid 2 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:266: finishing write tid 4 to smithi16030472-266 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:266: oid 266 updating version 0 to 169 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:266: oid 266 version 169 is already newer than 168 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:266: oid 266 version 169 is already newer than 167 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 169 (ObjNum 265 snap 0 seq_num 265) dirty exists 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:266: left oid 266 (ObjNum 265 snap 0 seq_num 265) 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: finishing write tid 2 to smithi16030472-267 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: finishing write tid 3 to smithi16030472-267 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: finishing write tid 4 to smithi16030472-267 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: oid 267 updating version 0 to 170 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: oid 267 updating version 170 to 171 2024-09-21T18:07:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:267: oid 267 updating version 171 to 172 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 267 v 172 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:267: left oid 267 (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:269: finishing write tid 1 to smithi16030472-269 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:266: done (15 left) 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:267: done (14 left) 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:286: write initial oid 286 2024-09-21T18:07:27.764 INFO:tasks.rados.rados.0.smithi160.stdout:286: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-21T18:07:27.765 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:27.771 INFO:tasks.rados.rados.0.smithi160.stdout:286: writing smithi16030472-286 from 789995 to 1341021 tid 1 2024-09-21T18:07:27.779 INFO:tasks.rados.rados.0.smithi160.stdout:286: writing smithi16030472-286 from 2027721 to 2496141 tid 2 2024-09-21T18:07:27.779 INFO:tasks.rados.rados.0.smithi160.stdout:287: write initial oid 287 2024-09-21T18:07:27.779 INFO:tasks.rados.rados.0.smithi160.stdout:287: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-21T18:07:27.786 INFO:tasks.rados.rados.0.smithi160.stdout:287: writing smithi16030472-287 from 406802 to 1033077 tid 1 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:287: writing smithi16030472-287 from 1686203 to 2414634 tid 2 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: finishing write tid 2 to smithi16030472-269 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: finishing write tid 3 to smithi16030472-269 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: finishing write tid 4 to smithi16030472-269 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: oid 269 updating version 0 to 173 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: oid 269 updating version 173 to 175 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:269: oid 269 version 175 is already newer than 174 2024-09-21T18:07:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 269 v 175 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-09-21T18:07:27.796 INFO:tasks.rados.rados.0.smithi160.stdout:269: left oid 269 (ObjNum 268 snap 0 seq_num 268) 2024-09-21T18:07:27.796 INFO:tasks.rados.rados.0.smithi160.stdout:272: finishing write tid 1 to smithi16030472-272 2024-09-21T18:07:27.796 INFO:tasks.rados.rados.0.smithi160.stdout:269: done (15 left) 2024-09-21T18:07:27.796 INFO:tasks.rados.rados.0.smithi160.stdout:288: write initial oid 288 2024-09-21T18:07:27.796 INFO:tasks.rados.rados.0.smithi160.stdout:288: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-21T18:07:27.801 INFO:tasks.rados.rados.0.smithi160.stdout:288: writing smithi16030472-288 from 423609 to 1125133 tid 1 2024-09-21T18:07:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:288: writing smithi16030472-288 from 1628332 to 2100421 tid 2 2024-09-21T18:07:27.812 INFO:tasks.rados.rados.0.smithi160.stdout:288: writing smithi16030472-288 from 2648987 to 2823609 tid 3 2024-09-21T18:07:27.812 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.847 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:27.842+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:27.872 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:27.872 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:27.886 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:27.948 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:27.944+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:27.964 INFO:tasks.rados.rados.0.smithi160.stdout:271: finishing write tid 1 to smithi16030472-271 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: finishing write tid 2 to smithi16030472-271 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: finishing write tid 3 to smithi16030472-271 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: finishing write tid 4 to smithi16030472-271 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: oid 271 updating version 0 to 176 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: oid 271 updating version 176 to 177 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: oid 271 updating version 177 to 178 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 271 v 178 (ObjNum 270 snap 0 seq_num 270) dirty exists 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: left oid 271 (ObjNum 270 snap 0 seq_num 270) 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:276: finishing write tid 1 to smithi16030472-276 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:276: finishing write tid 2 to smithi16030472-276 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:271: done (15 left) 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:289: write initial oid 289 2024-09-21T18:07:27.965 INFO:tasks.rados.rados.0.smithi160.stdout:289: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-21T18:07:27.971 INFO:tasks.rados.rados.0.smithi160.stdout:289: writing smithi16030472-289 from 440416 to 1217189 tid 1 2024-09-21T18:07:27.978 INFO:tasks.rados.rados.0.smithi160.stdout:289: writing smithi16030472-289 from 1686814 to 2418914 tid 2 2024-09-21T18:07:27.978 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:275: finishing write tid 4 to smithi16030472-275 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:275: oid 275 updating version 0 to 197 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:275: oid 275 updating version 197 to 199 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:275: oid 275 version 199 is already newer than 198 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 275 v 199 (ObjNum 274 snap 0 seq_num 274) dirty exists 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:275: left oid 275 (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: finishing write tid 2 to smithi16030472-272 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: finishing write tid 3 to smithi16030472-272 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: finishing write tid 4 to smithi16030472-272 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: oid 272 updating version 0 to 228 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: oid 272 updating version 228 to 229 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:272: oid 272 updating version 229 to 230 2024-09-21T18:07:27.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 272 v 230 (ObjNum 271 snap 0 seq_num 271) dirty exists 2024-09-21T18:07:27.983 INFO:tasks.rados.rados.0.smithi160.stdout:272: left oid 272 (ObjNum 271 snap 0 seq_num 271) 2024-09-21T18:07:27.983 INFO:tasks.rados.rados.0.smithi160.stdout:272: done (15 left) 2024-09-21T18:07:27.983 INFO:tasks.rados.rados.0.smithi160.stdout:275: done (14 left) 2024-09-21T18:07:27.983 INFO:tasks.rados.rados.0.smithi160.stdout:290: write initial oid 290 2024-09-21T18:07:27.983 INFO:tasks.rados.rados.0.smithi160.stdout:290: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-21T18:07:27.989 INFO:tasks.rados.rados.0.smithi160.stdout:290: writing smithi16030472-290 from 457223 to 1025598 tid 1 2024-09-21T18:07:27.995 INFO:tasks.rados.rados.0.smithi160.stdout:290: writing smithi16030472-290 from 1461649 to 1937407 tid 2 2024-09-21T18:07:28.001 INFO:tasks.rados.rados.0.smithi160.stdout:290: writing smithi16030472-290 from 2420186 to 2857223 tid 3 2024-09-21T18:07:28.002 INFO:tasks.rados.rados.0.smithi160.stdout:291: write initial oid 291 2024-09-21T18:07:28.002 INFO:tasks.rados.rados.0.smithi160.stdout:291: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-21T18:07:28.009 INFO:tasks.rados.rados.0.smithi160.stdout:291: writing smithi16030472-291 from 474030 to 1117654 tid 1 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:291: writing smithi16030472-291 from 1520131 to 2255900 tid 2 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:277: finishing write tid 1 to smithi16030472-277 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:277: finishing write tid 2 to smithi16030472-277 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:277: finishing write tid 3 to smithi16030472-277 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:277: finishing write tid 4 to smithi16030472-277 2024-09-21T18:07:28.018 INFO:tasks.rados.rados.0.smithi160.stdout:277: oid 277 updating version 0 to 231 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:277: oid 277 updating version 231 to 233 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:277: oid 277 version 233 is already newer than 232 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 277 v 233 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:277: left oid 277 (ObjNum 276 snap 0 seq_num 276) 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:286: finishing write tid 1 to smithi16030472-286 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:277: done (15 left) 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:292: write initial oid 292 2024-09-21T18:07:28.019 INFO:tasks.rados.rados.0.smithi160.stdout:292: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-21T18:07:28.024 INFO:tasks.rados.rados.0.smithi160.stdout:292: writing smithi16030472-292 from 490837 to 1209710 tid 1 2024-09-21T18:07:28.032 INFO:tasks.rados.rados.0.smithi160.stdout:292: writing smithi16030472-292 from 1862260 to 2341687 tid 2 2024-09-21T18:07:28.036 INFO:tasks.rados.rados.0.smithi160.stdout:292: writing smithi16030472-292 from 2875032 to 2890837 tid 3 2024-09-21T18:07:28.036 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.048 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:28.045+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:28.057 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.057 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.072 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:28.143 INFO:tasks.rados.rados.0.smithi160.stdout:278: finishing write tid 1 to smithi16030472-278 2024-09-21T18:07:28.149 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:28.145+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:28.197 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.198 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.212 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:28.244 INFO:tasks.rados.rados.0.smithi160.stdout:278: finishing write tid 2 to smithi16030472-278 2024-09-21T18:07:28.251 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:28.245+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:28.253 INFO:tasks.rados.rados.0.smithi160.stdout:276: finishing write tid 3 to smithi16030472-276 2024-09-21T18:07:28.329 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.330 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.343 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:28.351 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:28.348+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:28.354 INFO:tasks.rados.rados.0.smithi160.stdout:276: finishing write tid 4 to smithi16030472-276 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:276: oid 276 updating version 0 to 180 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:276: oid 276 updating version 180 to 181 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:276: oid 276 version 181 is already newer than 179 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 181 (ObjNum 275 snap 0 seq_num 275) dirty exists 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:276: left oid 276 (ObjNum 275 snap 0 seq_num 275) 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:276: done (15 left) 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:293: write initial oid 293 2024-09-21T18:07:28.355 INFO:tasks.rados.rados.0.smithi160.stdout:293: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-21T18:07:28.362 INFO:tasks.rados.rados.0.smithi160.stdout:293: writing smithi16030472-293 from 507644 to 1301766 tid 1 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:293: writing smithi16030472-293 from 1920742 to 2543827 tid 2 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:281: finishing write tid 1 to smithi16030472-281 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: finishing write tid 3 to smithi16030472-278 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:282: finishing write tid 1 to smithi16030472-282 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: finishing write tid 4 to smithi16030472-278 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: oid 278 updating version 0 to 182 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: oid 278 version 182 is already newer than 181 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: oid 278 updating version 182 to 183 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 278 v 183 (ObjNum 277 snap 0 seq_num 277) dirty exists 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:278: left oid 278 (ObjNum 277 snap 0 seq_num 277) 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:282: finishing write tid 2 to smithi16030472-282 2024-09-21T18:07:28.371 INFO:tasks.rados.rados.0.smithi160.stdout:282: finishing write tid 3 to smithi16030472-282 2024-09-21T18:07:28.372 INFO:tasks.rados.rados.0.smithi160.stdout:283: finishing write tid 1 to smithi16030472-283 2024-09-21T18:07:28.372 INFO:tasks.rados.rados.0.smithi160.stdout:284: finishing write tid 1 to smithi16030472-284 2024-09-21T18:07:28.372 INFO:tasks.rados.rados.0.smithi160.stdout:278: done (15 left) 2024-09-21T18:07:28.372 INFO:tasks.rados.rados.0.smithi160.stdout:294: write initial oid 294 2024-09-21T18:07:28.372 INFO:tasks.rados.rados.0.smithi160.stdout:294: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2024-09-21T18:07:28.378 INFO:tasks.rados.rados.0.smithi160.stdout:294: writing smithi16030472-294 from 524451 to 993822 tid 1 2024-09-21T18:07:28.386 INFO:tasks.rados.rados.0.smithi160.stdout:294: writing smithi16030472-294 from 1579224 to 2062320 tid 2 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout:294: writing smithi16030472-294 from 2529878 to 2924451 tid 3 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout:285: finishing write tid 1 to smithi16030472-285 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout:285: finishing write tid 2 to smithi16030472-285 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout:292: finishing write tid 1 to smithi16030472-292 2024-09-21T18:07:28.393 INFO:tasks.rados.rados.0.smithi160.stdout:292: finishing write tid 2 to smithi16030472-292 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: finishing write tid 2 to smithi16030472-286 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: finishing write tid 3 to smithi16030472-286 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: finishing write tid 4 to smithi16030472-286 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: oid 286 updating version 0 to 200 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: oid 286 updating version 200 to 202 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: oid 286 version 202 is already newer than 201 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 286 v 202 (ObjNum 285 snap 0 seq_num 285) dirty exists 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: left oid 286 (ObjNum 285 snap 0 seq_num 285) 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:288: finishing write tid 1 to smithi16030472-288 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:288: finishing write tid 2 to smithi16030472-288 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:286: done (15 left) 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:295: write initial oid 295 2024-09-21T18:07:28.394 INFO:tasks.rados.rados.0.smithi160.stdout:295: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-21T18:07:28.401 INFO:tasks.rados.rados.0.smithi160.stdout:295: writing smithi16030472-295 from 541258 to 1085878 tid 1 2024-09-21T18:07:28.411 INFO:tasks.rados.rados.0.smithi160.stdout:295: writing smithi16030472-295 from 1637706 to 2264460 tid 2 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:295: writing smithi16030472-295 from 2840948 to 2941258 tid 3 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: finishing write tid 3 to smithi16030472-288 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:279: finishing write tid 1 to smithi16030472-279 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: finishing write tid 4 to smithi16030472-288 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: finishing write tid 5 to smithi16030472-288 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: oid 288 updating version 0 to 205 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: oid 288 version 205 is already newer than 204 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: oid 288 updating version 205 to 206 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: oid 288 version 206 is already newer than 203 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 288 v 206 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-09-21T18:07:28.416 INFO:tasks.rados.rados.0.smithi160.stdout:288: left oid 288 (ObjNum 287 snap 0 seq_num 287) 2024-09-21T18:07:28.417 INFO:tasks.rados.rados.0.smithi160.stdout:288: done (15 left) 2024-09-21T18:07:28.417 INFO:tasks.rados.rados.0.smithi160.stdout:296: write initial oid 296 2024-09-21T18:07:28.417 INFO:tasks.rados.rados.0.smithi160.stdout:296: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-21T18:07:28.424 INFO:tasks.rados.rados.0.smithi160.stdout:296: writing smithi16030472-296 from 558065 to 1177934 tid 1 2024-09-21T18:07:28.430 INFO:tasks.rados.rados.0.smithi160.stdout:296: writing smithi16030472-296 from 1579835 to 2066600 tid 2 2024-09-21T18:07:28.436 INFO:tasks.rados.rados.0.smithi160.stdout:296: writing smithi16030472-296 from 2468371 to 2958065 tid 3 2024-09-21T18:07:28.436 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.452 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:28.446+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:28.461 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.462 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.464 INFO:tasks.rados.rados.0.smithi160.stdout:281: finishing write tid 2 to smithi16030472-281 2024-09-21T18:07:28.479 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: finishing write tid 2 to smithi16030472-279 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: finishing write tid 3 to smithi16030472-279 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: finishing write tid 4 to smithi16030472-279 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: oid 279 updating version 0 to 167 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: oid 279 updating version 167 to 168 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: oid 279 version 168 is already newer than 166 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 279 v 168 (ObjNum 278 snap 0 seq_num 278) dirty exists 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: left oid 279 (ObjNum 278 snap 0 seq_num 278) 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:279: done (15 left) 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:297: write initial oid 297 2024-09-21T18:07:28.497 INFO:tasks.rados.rados.0.smithi160.stdout:297: seq_num 296 ranges {574872=695118,2038317=630423} 2024-09-21T18:07:28.505 INFO:tasks.rados.rados.0.smithi160.stdout:297: writing smithi16030472-297 from 574872 to 1269990 tid 1 2024-09-21T18:07:28.514 INFO:tasks.rados.rados.0.smithi160.stdout:297: writing smithi16030472-297 from 2038317 to 2668740 tid 2 2024-09-21T18:07:28.514 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.514 INFO:tasks.rados.rados.0.smithi160.stdout:290: finishing write tid 1 to smithi16030472-290 2024-09-21T18:07:28.514 INFO:tasks.rados.rados.0.smithi160.stdout:280: finishing write tid 1 to smithi16030472-280 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: finishing write tid 2 to smithi16030472-280 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: finishing write tid 3 to smithi16030472-280 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: finishing write tid 4 to smithi16030472-280 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: oid 280 updating version 0 to 169 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: oid 280 updating version 169 to 171 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: oid 280 version 171 is already newer than 170 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 280 v 171 (ObjNum 279 snap 0 seq_num 279) dirty exists 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: left oid 280 (ObjNum 279 snap 0 seq_num 279) 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:287: finishing write tid 1 to smithi16030472-287 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:280: done (15 left) 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:298: write initial oid 298 2024-09-21T18:07:28.515 INFO:tasks.rados.rados.0.smithi160.stdout:298: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-21T18:07:28.521 INFO:tasks.rados.rados.0.smithi160.stdout:298: writing smithi16030472-298 from 591679 to 1078399 tid 1 2024-09-21T18:07:28.529 INFO:tasks.rados.rados.0.smithi160.stdout:298: writing smithi16030472-298 from 1813152 to 2303586 tid 2 2024-09-21T18:07:28.529 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.553 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:28.548+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:28.602 INFO:tasks.rados.rados.0.smithi160.stdout:293: finishing write tid 1 to smithi16030472-293 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: finishing write tid 2 to smithi16030472-293 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: finishing write tid 3 to smithi16030472-293 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: finishing write tid 4 to smithi16030472-293 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: oid 293 updating version 0 to 207 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: oid 293 updating version 207 to 209 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: oid 293 version 209 is already newer than 208 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 209 (ObjNum 292 snap 0 seq_num 292) dirty exists 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: left oid 293 (ObjNum 292 snap 0 seq_num 292) 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:293: done (15 left) 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:299: write initial oid 299 2024-09-21T18:07:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:299: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-21T18:07:28.612 INFO:tasks.rados.rados.0.smithi160.stdout:299: writing smithi16030472-299 from 608486 to 1170455 tid 1 2024-09-21T18:07:28.614 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.614 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.622 INFO:tasks.rados.rados.0.smithi160.stdout:299: writing smithi16030472-299 from 1871634 to 2505726 tid 2 2024-09-21T18:07:28.625 INFO:tasks.rados.rados.0.smithi160.stdout:299: writing smithi16030472-299 from 2950640 to 3008486 tid 3 2024-09-21T18:07:28.625 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.631 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:28.671 INFO:tasks.rados.rados.0.smithi160.stdout:281: finishing write tid 3 to smithi16030472-281 2024-09-21T18:07:28.672 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:28.666+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:28.672 INFO:tasks.rados.rados.0.smithi160.stdout:282: finishing write tid 4 to smithi16030472-282 2024-09-21T18:07:28.672 INFO:tasks.rados.rados.0.smithi160.stdout:282: oid 282 updating version 0 to 184 2024-09-21T18:07:28.672 INFO:tasks.rados.rados.0.smithi160.stdout:282: oid 282 updating version 184 to 186 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:282: oid 282 version 186 is already newer than 185 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 282 v 186 (ObjNum 281 snap 0 seq_num 281) dirty exists 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:282: left oid 282 (ObjNum 281 snap 0 seq_num 281) 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:282: done (15 left) 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:300: write initial oid 300 2024-09-21T18:07:28.673 INFO:tasks.rados.rados.0.smithi160.stdout:300: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-21T18:07:28.679 INFO:tasks.rados.rados.0.smithi160.stdout:300: writing smithi16030472-300 from 625293 to 1262511 tid 1 2024-09-21T18:07:28.685 INFO:tasks.rados.rados.0.smithi160.stdout:300: writing smithi16030472-300 from 1813763 to 2307866 tid 2 2024-09-21T18:07:28.689 INFO:tasks.rados.rados.0.smithi160.stdout:300: writing smithi16030472-300 from 2978063 to 3025293 tid 3 2024-09-21T18:07:28.689 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.689 INFO:tasks.rados.rados.0.smithi160.stdout:284: finishing write tid 2 to smithi16030472-284 2024-09-21T18:07:28.754 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:28.750+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:284: finishing write tid 3 to smithi16030472-284 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:284: finishing write tid 4 to smithi16030472-284 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:284: oid 284 updating version 0 to 189 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:284: oid 284 version 189 is already newer than 188 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:284: oid 284 updating version 189 to 190 2024-09-21T18:07:28.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 284 v 190 (ObjNum 283 snap 0 seq_num 283) dirty exists 2024-09-21T18:07:28.775 INFO:tasks.rados.rados.0.smithi160.stdout:284: left oid 284 (ObjNum 283 snap 0 seq_num 283) 2024-09-21T18:07:28.775 INFO:tasks.rados.rados.0.smithi160.stdout:284: done (15 left) 2024-09-21T18:07:28.775 INFO:tasks.rados.rados.0.smithi160.stdout:301: write initial oid 301 2024-09-21T18:07:28.775 INFO:tasks.rados.rados.0.smithi160.stdout:301: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-21T18:07:28.787 INFO:tasks.rados.rados.0.smithi160.stdout:301: writing smithi16030472-301 from 642100 to 1354567 tid 1 2024-09-21T18:07:28.794 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.794 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.797 INFO:tasks.rados.rados.0.smithi160.stdout:301: writing smithi16030472-301 from 1872245 to 2510006 tid 2 2024-09-21T18:07:28.797 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: finishing write tid 2 to smithi16030472-290 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: finishing write tid 3 to smithi16030472-290 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: finishing write tid 4 to smithi16030472-290 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: finishing write tid 5 to smithi16030472-290 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: oid 290 updating version 0 to 237 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: oid 290 version 237 is already newer than 236 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: oid 290 version 237 is already newer than 235 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: oid 290 version 237 is already newer than 234 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 290 v 237 (ObjNum 289 snap 0 seq_num 289) dirty exists 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: left oid 290 (ObjNum 289 snap 0 seq_num 289) 2024-09-21T18:07:28.811 INFO:tasks.rados.rados.0.smithi160.stdout:290: done (15 left) 2024-09-21T18:07:28.812 INFO:tasks.rados.rados.0.smithi160.stdout:302: write initial oid 302 2024-09-21T18:07:28.812 INFO:tasks.rados.rados.0.smithi160.stdout:302: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-21T18:07:28.827 INFO:tasks.rados.rados.0.smithi160.stdout:302: writing smithi16030472-302 from 658907 to 1446623 tid 1 2024-09-21T18:07:28.838 INFO:tasks.rados.rados.0.smithi160.stdout:302: writing smithi16030472-302 from 1930727 to 2428499 tid 2 2024-09-21T18:07:28.844 INFO:tasks.rados.rados.0.smithi160.stdout:302: writing smithi16030472-302 from 3032909 to 3058907 tid 3 2024-09-21T18:07:28.844 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.844 INFO:tasks.rados.rados.0.smithi160.stdout:296: finishing write tid 1 to smithi16030472-296 2024-09-21T18:07:28.844 INFO:tasks.rados.rados.0.smithi160.stdout:296: finishing write tid 2 to smithi16030472-296 2024-09-21T18:07:28.844 INFO:tasks.rados.rados.0.smithi160.stdout:296: finishing write tid 3 to smithi16030472-296 2024-09-21T18:07:28.855 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:28.854+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:28.873 INFO:tasks.rados.rados.0.smithi160.stdout:285: finishing write tid 3 to smithi16030472-285 2024-09-21T18:07:28.874 INFO:tasks.rados.rados.0.smithi160.stdout:285: finishing write tid 4 to smithi16030472-285 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:285: oid 285 updating version 0 to 190 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:285: oid 285 version 190 is already newer than 189 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:285: oid 285 updating version 190 to 191 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 285 v 191 (ObjNum 284 snap 0 seq_num 284) dirty exists 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:285: left oid 285 (ObjNum 284 snap 0 seq_num 284) 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:285: done (15 left) 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:303: write initial oid 303 2024-09-21T18:07:28.875 INFO:tasks.rados.rados.0.smithi160.stdout:303: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-21T18:07:28.885 INFO:tasks.rados.rados.0.smithi160.stdout:303: writing smithi16030472-303 from 675714 to 1138679 tid 1 2024-09-21T18:07:28.896 INFO:tasks.rados.rados.0.smithi160.stdout:303: writing smithi16030472-303 from 1589209 to 2230639 tid 2 2024-09-21T18:07:28.904 INFO:tasks.rados.rados.0.smithi160.stdout:303: writing smithi16030472-303 from 2943979 to 3075714 tid 3 2024-09-21T18:07:28.904 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.904 INFO:tasks.rados.rados.0.smithi160.stdout:283: finishing write tid 2 to smithi16030472-283 2024-09-21T18:07:28.904 INFO:tasks.rados.rados.0.smithi160.stdout:283: finishing write tid 3 to smithi16030472-283 2024-09-21T18:07:28.904 INFO:tasks.rados.rados.0.smithi160.stdout:283: finishing write tid 4 to smithi16030472-283 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:283: oid 283 updating version 0 to 193 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:283: oid 283 version 193 is already newer than 192 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:283: oid 283 version 193 is already newer than 187 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 283 v 193 (ObjNum 282 snap 0 seq_num 282) dirty exists 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:283: left oid 283 (ObjNum 282 snap 0 seq_num 282) 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:283: done (15 left) 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:304: write initial oid 304 2024-09-21T18:07:28.905 INFO:tasks.rados.rados.0.smithi160.stdout:304: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-21T18:07:28.916 INFO:tasks.rados.rados.0.smithi160.stdout:304: writing smithi16030472-304 from 692521 to 1230735 tid 1 2024-09-21T18:07:28.930 INFO:tasks.rados.rados.0.smithi160.stdout:304: writing smithi16030472-304 from 1931338 to 2716426 tid 2 2024-09-21T18:07:28.930 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:28.931 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:28.931 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:28.946 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:28.956 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:28.952+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: finishing write tid 2 to smithi16030472-287 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: finishing write tid 3 to smithi16030472-287 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: finishing write tid 4 to smithi16030472-287 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: oid 287 updating version 0 to 174 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: oid 287 version 174 is already newer than 172 2024-09-21T18:07:28.984 INFO:tasks.rados.rados.0.smithi160.stdout:287: oid 287 version 174 is already newer than 173 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 287 v 174 (ObjNum 286 snap 0 seq_num 286) dirty exists 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:287: left oid 287 (ObjNum 286 snap 0 seq_num 286) 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:289: finishing write tid 1 to smithi16030472-289 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:287: done (15 left) 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:305: write initial oid 305 2024-09-21T18:07:28.985 INFO:tasks.rados.rados.0.smithi160.stdout:305: seq_num 304 ranges {709328=613463,1989820=645099} 2024-09-21T18:07:28.994 INFO:tasks.rados.rados.0.smithi160.stdout:305: writing smithi16030472-305 from 709328 to 1322791 tid 1 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:305: writing smithi16030472-305 from 1989820 to 2634919 tid 2 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:289: finishing write tid 2 to smithi16030472-289 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:289: finishing write tid 3 to smithi16030472-289 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:289: finishing write tid 4 to smithi16030472-289 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:289: oid 289 updating version 0 to 176 2024-09-21T18:07:29.004 INFO:tasks.rados.rados.0.smithi160.stdout:289: oid 289 version 176 is already newer than 175 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:289: oid 289 updating version 176 to 177 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 177 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:289: left oid 289 (ObjNum 288 snap 0 seq_num 288) 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:291: finishing write tid 1 to smithi16030472-291 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:291: finishing write tid 2 to smithi16030472-291 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:289: done (15 left) 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:306: write initial oid 306 2024-09-21T18:07:29.005 INFO:tasks.rados.rados.0.smithi160.stdout:306: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-21T18:07:29.012 INFO:tasks.rados.rados.0.smithi160.stdout:306: writing smithi16030472-306 from 726135 to 1131200 tid 1 2024-09-21T18:07:29.023 INFO:tasks.rados.rados.0.smithi160.stdout:306: writing smithi16030472-306 from 1764655 to 2553412 tid 2 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:306: writing smithi16030472-306 from 3026248 to 3126135 tid 3 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: finishing write tid 4 to smithi16030472-281 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: oid 281 updating version 0 to 183 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: oid 281 updating version 183 to 185 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: oid 281 version 185 is already newer than 184 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 281 v 185 (ObjNum 280 snap 0 seq_num 280) dirty exists 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: left oid 281 (ObjNum 280 snap 0 seq_num 280) 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:281: done (15 left) 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:307: write initial oid 307 2024-09-21T18:07:29.028 INFO:tasks.rados.rados.0.smithi160.stdout:307: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-21T18:07:29.037 INFO:tasks.rados.rados.0.smithi160.stdout:307: writing smithi16030472-307 from 742942 to 1223256 tid 1 2024-09-21T18:07:29.046 INFO:tasks.rados.rados.0.smithi160.stdout:307: writing smithi16030472-307 from 1823137 to 2471905 tid 2 2024-09-21T18:07:29.046 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.058 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:29.053+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:29.106 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.106 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.121 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:29.130 INFO:tasks.rados.rados.0.smithi160.stdout:297: finishing write tid 1 to smithi16030472-297 2024-09-21T18:07:29.130 INFO:tasks.rados.rados.0.smithi160.stdout:297: finishing write tid 2 to smithi16030472-297 2024-09-21T18:07:29.130 INFO:tasks.rados.rados.0.smithi160.stdout:291: finishing write tid 3 to smithi16030472-291 2024-09-21T18:07:29.130 INFO:tasks.rados.rados.0.smithi160.stdout:297: finishing write tid 3 to smithi16030472-297 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: finishing write tid 4 to smithi16030472-291 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: oid 291 updating version 0 to 179 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: oid 291 version 179 is already newer than 178 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: oid 291 updating version 179 to 180 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 291 v 180 (ObjNum 290 snap 0 seq_num 290) dirty exists 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: left oid 291 (ObjNum 290 snap 0 seq_num 290) 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: finishing write tid 4 to smithi16030472-297 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: oid 297 updating version 0 to 243 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: oid 297 version 243 is already newer than 241 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: oid 297 version 243 is already newer than 242 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 297 v 243 (ObjNum 296 snap 0 seq_num 296) dirty exists 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: left oid 297 (ObjNum 296 snap 0 seq_num 296) 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:291: done (15 left) 2024-09-21T18:07:29.131 INFO:tasks.rados.rados.0.smithi160.stdout:297: done (14 left) 2024-09-21T18:07:29.132 INFO:tasks.rados.rados.0.smithi160.stdout:308: write initial oid 308 2024-09-21T18:07:29.132 INFO:tasks.rados.rados.0.smithi160.stdout:308: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-21T18:07:29.137 INFO:tasks.rados.rados.0.smithi160.stdout:308: writing smithi16030472-308 from 759749 to 1315312 tid 1 2024-09-21T18:07:29.144 INFO:tasks.rados.rados.0.smithi160.stdout:308: writing smithi16030472-308 from 1765266 to 2557692 tid 2 2024-09-21T18:07:29.147 INFO:tasks.rados.rados.0.smithi160.stdout:308: writing smithi16030472-308 from 2964741 to 3159749 tid 3 2024-09-21T18:07:29.148 INFO:tasks.rados.rados.0.smithi160.stdout:309: write initial oid 309 2024-09-21T18:07:29.148 INFO:tasks.rados.rados.0.smithi160.stdout:309: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-09-21T18:07:29.155 INFO:tasks.rados.rados.0.smithi160.stdout:309: writing smithi16030472-309 from 776556 to 1407368 tid 1 2024-09-21T18:07:29.158 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:29.154+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:29.161 INFO:tasks.rados.rados.0.smithi160.stdout:309: writing smithi16030472-309 from 1823748 to 2476185 tid 2 2024-09-21T18:07:29.165 INFO:tasks.rados.rados.0.smithi160.stdout:309: writing smithi16030472-309 from 3108517 to 3176556 tid 3 2024-09-21T18:07:29.165 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.165 INFO:tasks.rados.rados.0.smithi160.stdout:298: finishing write tid 1 to smithi16030472-298 2024-09-21T18:07:29.165 INFO:tasks.rados.rados.0.smithi160.stdout:294: finishing write tid 1 to smithi16030472-294 2024-09-21T18:07:29.166 INFO:tasks.rados.rados.0.smithi160.stdout:294: finishing write tid 2 to smithi16030472-294 2024-09-21T18:07:29.232 INFO:tasks.rados.rados.0.smithi160.stdout:296: finishing write tid 4 to smithi16030472-296 2024-09-21T18:07:29.232 INFO:tasks.rados.rados.0.smithi160.stdout:296: finishing write tid 5 to smithi16030472-296 2024-09-21T18:07:29.232 INFO:tasks.rados.rados.0.smithi160.stdout:296: oid 296 updating version 0 to 238 2024-09-21T18:07:29.232 INFO:tasks.rados.rados.0.smithi160.stdout:296: oid 296 updating version 238 to 241 2024-09-21T18:07:29.232 INFO:tasks.rados.rados.0.smithi160.stdout:296: oid 296 version 241 is already newer than 240 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:296: oid 296 version 241 is already newer than 239 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 296 v 241 (ObjNum 295 snap 0 seq_num 295) dirty exists 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:296: left oid 296 (ObjNum 295 snap 0 seq_num 295) 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:296: done (15 left) 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:310: write initial oid 310 2024-09-21T18:07:29.233 INFO:tasks.rados.rados.0.smithi160.stdout:310: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-21T18:07:29.243 INFO:tasks.rados.rados.0.smithi160.stdout:310: writing smithi16030472-310 from 793363 to 1499424 tid 1 2024-09-21T18:07:29.251 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.251 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.254 INFO:tasks.rados.rados.0.smithi160.stdout:310: writing smithi16030472-310 from 2282230 to 3078325 tid 2 2024-09-21T18:07:29.254 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.255 INFO:tasks.rados.rados.0.smithi160.stdout:295: finishing write tid 1 to smithi16030472-295 2024-09-21T18:07:29.255 INFO:tasks.rados.rados.0.smithi160.stdout:303: finishing write tid 1 to smithi16030472-303 2024-09-21T18:07:29.255 INFO:tasks.rados.rados.0.smithi160.stdout:303: finishing write tid 2 to smithi16030472-303 2024-09-21T18:07:29.255 INFO:tasks.rados.rados.0.smithi160.stdout:303: finishing write tid 3 to smithi16030472-303 2024-09-21T18:07:29.255 INFO:tasks.rados.rados.0.smithi160.stdout:292: finishing write tid 3 to smithi16030472-292 2024-09-21T18:07:29.259 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:29.257+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:29.266 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: finishing write tid 4 to smithi16030472-292 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: finishing write tid 5 to smithi16030472-292 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: oid 292 updating version 0 to 195 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: oid 292 version 195 is already newer than 194 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: oid 292 version 195 is already newer than 192 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: oid 292 version 195 is already newer than 191 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 292 v 195 (ObjNum 291 snap 0 seq_num 291) dirty exists 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: left oid 292 (ObjNum 291 snap 0 seq_num 291) 2024-09-21T18:07:29.334 INFO:tasks.rados.rados.0.smithi160.stdout:292: done (15 left) 2024-09-21T18:07:29.335 INFO:tasks.rados.rados.0.smithi160.stdout:311: write initial oid 311 2024-09-21T18:07:29.335 INFO:tasks.rados.rados.0.smithi160.stdout:311: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-21T18:07:29.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:07:29.343 INFO:tasks.rados.rados.0.smithi160.stdout:311: writing smithi16030472-311 from 410170 to 1191480 tid 1 2024-09-21T18:07:29.354 INFO:tasks.rados.rados.0.smithi160.stdout:311: writing smithi16030472-311 from 1940712 to 2596818 tid 2 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:311: writing smithi16030472-311 from 3163363 to 3210170 tid 3 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: finishing write tid 3 to smithi16030472-294 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: finishing write tid 4 to smithi16030472-294 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: finishing write tid 5 to smithi16030472-294 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: oid 294 updating version 0 to 184 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: oid 294 version 184 is already newer than 183 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: oid 294 version 184 is already newer than 181 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: oid 294 version 184 is already newer than 182 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 184 (ObjNum 293 snap 0 seq_num 293) dirty exists 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: left oid 294 (ObjNum 293 snap 0 seq_num 293) 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:300: finishing write tid 1 to smithi16030472-300 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:300: finishing write tid 2 to smithi16030472-300 2024-09-21T18:07:29.360 INFO:tasks.rados.rados.0.smithi160.stdout:294: done (15 left) 2024-09-21T18:07:29.361 INFO:tasks.rados.rados.0.smithi160.stdout:312: write initial oid 312 2024-09-21T18:07:29.361 INFO:tasks.rados.rados.0.smithi160.stdout:312: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-21T18:07:29.361 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:29.357+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:29.366 INFO:tasks.rados.rados.0.smithi160.stdout:312: writing smithi16030472-312 from 426977 to 883536 tid 1 2024-09-21T18:07:29.377 INFO:tasks.rados.rados.0.smithi160.stdout:312: writing smithi16030472-312 from 1482841 to 2282605 tid 2 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:312: writing smithi16030472-312 from 2958080 to 3226977 tid 3 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: finishing write tid 2 to smithi16030472-298 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: finishing write tid 3 to smithi16030472-298 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: finishing write tid 4 to smithi16030472-298 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: oid 298 updating version 0 to 244 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: oid 298 updating version 244 to 246 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: oid 298 version 246 is already newer than 245 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 298 v 246 (ObjNum 297 snap 0 seq_num 297) dirty exists 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: left oid 298 (ObjNum 297 snap 0 seq_num 297) 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:299: finishing write tid 1 to smithi16030472-299 2024-09-21T18:07:29.385 INFO:tasks.rados.rados.0.smithi160.stdout:298: done (15 left) 2024-09-21T18:07:29.386 INFO:tasks.rados.rados.0.smithi160.stdout:313: write initial oid 313 2024-09-21T18:07:29.386 INFO:tasks.rados.rados.0.smithi160.stdout:313: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-21T18:07:29.393 INFO:tasks.rados.rados.0.smithi160.stdout:313: writing smithi16030472-313 from 443784 to 1091945 tid 1 2024-09-21T18:07:29.403 INFO:tasks.rados.rados.0.smithi160.stdout:313: writing smithi16030472-313 from 1657676 to 2317451 tid 2 2024-09-21T18:07:29.409 INFO:tasks.rados.rados.0.smithi160.stdout:313: writing smithi16030472-313 from 2818209 to 3243784 tid 3 2024-09-21T18:07:29.409 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.435 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.436 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.451 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:29.457 INFO:tasks.rados.rados.0.smithi160.stdout:299: finishing write tid 2 to smithi16030472-299 2024-09-21T18:07:29.457 INFO:tasks.rados.rados.0.smithi160.stdout:299: finishing write tid 3 to smithi16030472-299 2024-09-21T18:07:29.457 INFO:tasks.rados.rados.0.smithi160.stdout:299: finishing write tid 4 to smithi16030472-299 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: finishing write tid 5 to smithi16030472-299 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: oid 299 updating version 0 to 247 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: oid 299 updating version 247 to 250 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: oid 299 version 250 is already newer than 249 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: oid 299 version 250 is already newer than 248 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 299 v 250 (ObjNum 298 snap 0 seq_num 298) dirty exists 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: left oid 299 (ObjNum 298 snap 0 seq_num 298) 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:301: finishing write tid 1 to smithi16030472-301 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:301: finishing write tid 2 to smithi16030472-301 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:299: done (15 left) 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:314: write initial oid 314 2024-09-21T18:07:29.458 INFO:tasks.rados.rados.0.smithi160.stdout:314: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-21T18:07:29.460 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:29.457+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:29.465 INFO:tasks.rados.rados.0.smithi160.stdout:314: writing smithi16030472-314 from 460591 to 1184001 tid 1 2024-09-21T18:07:29.471 INFO:tasks.rados.rados.0.smithi160.stdout:314: writing smithi16030472-314 from 1716158 to 2119591 tid 2 2024-09-21T18:07:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:314: writing smithi16030472-314 from 2729279 to 3169419 tid 3 2024-09-21T18:07:29.479 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.561 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:29.558+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:301: finishing write tid 3 to smithi16030472-301 2024-09-21T18:07:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:301: finishing write tid 4 to smithi16030472-301 2024-09-21T18:07:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:301: oid 301 updating version 0 to 252 2024-09-21T18:07:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:301: oid 301 updating version 252 to 253 2024-09-21T18:07:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:301: oid 301 version 253 is already newer than 251 2024-09-21T18:07:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 301 v 253 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-21T18:07:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:301: left oid 301 (ObjNum 300 snap 0 seq_num 300) 2024-09-21T18:07:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:301: done (15 left) 2024-09-21T18:07:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:315: write initial oid 315 2024-09-21T18:07:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:315: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-21T18:07:29.587 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.587 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.589 INFO:tasks.rados.rados.0.smithi160.stdout:315: writing smithi16030472-315 from 477398 to 1276057 tid 1 2024-09-21T18:07:29.598 INFO:tasks.rados.rados.0.smithi160.stdout:315: writing smithi16030472-315 from 1774640 to 2438084 tid 2 2024-09-21T18:07:29.602 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:29.604 INFO:tasks.rados.rados.0.smithi160.stdout:315: writing smithi16030472-315 from 2873055 to 3277398 tid 3 2024-09-21T18:07:29.604 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: finishing write tid 1 to smithi16030472-305 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: finishing write tid 2 to smithi16030472-305 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: finishing write tid 3 to smithi16030472-305 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: finishing write tid 4 to smithi16030472-305 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: oid 305 updating version 0 to 212 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: oid 305 version 212 is already newer than 211 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: oid 305 version 212 is already newer than 210 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 305 v 212 (ObjNum 304 snap 0 seq_num 304) dirty exists 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: left oid 305 (ObjNum 304 snap 0 seq_num 304) 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:305: done (15 left) 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:316: write initial oid 316 2024-09-21T18:07:29.605 INFO:tasks.rados.rados.0.smithi160.stdout:316: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-21T18:07:29.611 INFO:tasks.rados.rados.0.smithi160.stdout:316: writing smithi16030472-316 from 494205 to 968113 tid 1 2024-09-21T18:07:29.617 INFO:tasks.rados.rados.0.smithi160.stdout:316: writing smithi16030472-316 from 1433122 to 1840224 tid 2 2024-09-21T18:07:29.625 INFO:tasks.rados.rados.0.smithi160.stdout:316: writing smithi16030472-316 from 2384125 to 2846809 tid 3 2024-09-21T18:07:29.625 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.625 INFO:tasks.rados.rados.0.smithi160.stdout:300: finishing write tid 3 to smithi16030472-300 2024-09-21T18:07:29.625 INFO:tasks.rados.rados.0.smithi160.stdout:300: finishing write tid 4 to smithi16030472-300 2024-09-21T18:07:29.646 INFO:tasks.rados.rados.0.smithi160.stdout:295: finishing write tid 2 to smithi16030472-295 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: finishing write tid 4 to smithi16030472-303 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: finishing write tid 5 to smithi16030472-303 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: oid 303 updating version 0 to 195 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: oid 303 updating version 195 to 197 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: oid 303 version 197 is already newer than 196 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: oid 303 version 197 is already newer than 194 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 303 v 197 (ObjNum 302 snap 0 seq_num 302) dirty exists 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: left oid 303 (ObjNum 302 snap 0 seq_num 302) 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:304: finishing write tid 1 to smithi16030472-304 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:304: finishing write tid 2 to smithi16030472-304 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:304: finishing write tid 3 to smithi16030472-304 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:303: done (15 left) 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:317: write initial oid 317 2024-09-21T18:07:29.647 INFO:tasks.rados.rados.0.smithi160.stdout:317: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-21T18:07:29.656 INFO:tasks.rados.rados.0.smithi160.stdout:317: writing smithi16030472-317 from 511012 to 1060169 tid 1 2024-09-21T18:07:29.663 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:29.662+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:29.666 INFO:tasks.rados.rados.0.smithi160.stdout:317: writing smithi16030472-317 from 1775251 to 2326011 tid 2 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:317: writing smithi16030472-317 from 3095195 to 3311012 tid 3 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:304: finishing write tid 4 to smithi16030472-304 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:304: oid 304 updating version 0 to 200 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:304: oid 304 version 200 is already newer than 199 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:304: oid 304 version 200 is already newer than 198 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 304 v 200 (ObjNum 303 snap 0 seq_num 303) dirty exists 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:304: left oid 304 (ObjNum 303 snap 0 seq_num 303) 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:306: finishing write tid 1 to smithi16030472-306 2024-09-21T18:07:29.675 INFO:tasks.rados.rados.0.smithi160.stdout:306: finishing write tid 2 to smithi16030472-306 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: finishing write tid 3 to smithi16030472-306 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: finishing write tid 4 to smithi16030472-306 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: finishing write tid 5 to smithi16030472-306 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: oid 306 updating version 0 to 204 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: oid 306 version 204 is already newer than 203 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: oid 306 version 204 is already newer than 201 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: oid 306 version 204 is already newer than 202 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 306 v 204 (ObjNum 305 snap 0 seq_num 305) dirty exists 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: left oid 306 (ObjNum 305 snap 0 seq_num 305) 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:295: finishing write tid 3 to smithi16030472-295 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:304: done (15 left) 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:306: done (14 left) 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:318: write initial oid 318 2024-09-21T18:07:29.676 INFO:tasks.rados.rados.0.smithi160.stdout:318: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-21T18:07:29.684 INFO:tasks.rados.rados.0.smithi160.stdout:318: writing smithi16030472-318 from 527819 to 1152225 tid 1 2024-09-21T18:07:29.693 INFO:tasks.rados.rados.0.smithi160.stdout:318: writing smithi16030472-318 from 1833733 to 2244504 tid 2 2024-09-21T18:07:29.701 INFO:tasks.rados.rados.0.smithi160.stdout:318: writing smithi16030472-318 from 2722618 to 3324199 tid 3 2024-09-21T18:07:29.701 INFO:tasks.rados.rados.0.smithi160.stdout:319: write initial oid 319 2024-09-21T18:07:29.701 INFO:tasks.rados.rados.0.smithi160.stdout:319: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-21T18:07:29.709 INFO:tasks.rados.rados.0.smithi160.stdout:319: writing smithi16030472-319 from 544626 to 1244281 tid 1 2024-09-21T18:07:29.717 INFO:tasks.rados.rados.0.smithi160.stdout:319: writing smithi16030472-319 from 1892215 to 2446644 tid 2 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:319: writing smithi16030472-319 from 3150041 to 3344626 tid 3 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: finishing write tid 4 to smithi16030472-295 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: finishing write tid 5 to smithi16030472-295 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: oid 295 updating version 0 to 197 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: oid 295 version 197 is already newer than 196 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: oid 295 version 197 is already newer than 193 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: oid 295 updating version 197 to 198 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 198 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: left oid 295 (ObjNum 294 snap 0 seq_num 294) 2024-09-21T18:07:29.723 INFO:tasks.rados.rados.0.smithi160.stdout:295: done (15 left) 2024-09-21T18:07:29.724 INFO:tasks.rados.rados.0.smithi160.stdout:320: write initial oid 320 2024-09-21T18:07:29.724 INFO:tasks.rados.rados.0.smithi160.stdout:320: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-21T18:07:29.725 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.725 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.730 INFO:tasks.rados.rados.0.smithi160.stdout:320: writing smithi16030472-320 from 561433 to 1336337 tid 1 2024-09-21T18:07:29.736 INFO:tasks.rados.rados.0.smithi160.stdout:320: writing smithi16030472-320 from 1950697 to 2365137 tid 2 2024-09-21T18:07:29.741 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:29.742 INFO:tasks.rados.rados.0.smithi160.stdout:320: writing smithi16030472-320 from 2777464 to 3361433 tid 3 2024-09-21T18:07:29.742 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: finishing write tid 1 to smithi16030472-302 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: finishing write tid 2 to smithi16030472-302 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: finishing write tid 3 to smithi16030472-302 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:316: finishing write tid 1 to smithi16030472-316 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: finishing write tid 4 to smithi16030472-302 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: finishing write tid 5 to smithi16030472-302 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: oid 302 updating version 0 to 257 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: oid 302 version 257 is already newer than 256 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: oid 302 version 257 is already newer than 254 2024-09-21T18:07:29.753 INFO:tasks.rados.rados.0.smithi160.stdout:302: oid 302 version 257 is already newer than 255 2024-09-21T18:07:29.754 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 302 v 257 (ObjNum 301 snap 0 seq_num 301) dirty exists 2024-09-21T18:07:29.754 INFO:tasks.rados.rados.0.smithi160.stdout:302: left oid 302 (ObjNum 301 snap 0 seq_num 301) 2024-09-21T18:07:29.754 INFO:tasks.rados.rados.0.smithi160.stdout:302: done (15 left) 2024-09-21T18:07:29.754 INFO:tasks.rados.rados.0.smithi160.stdout:321: write initial oid 321 2024-09-21T18:07:29.754 INFO:tasks.rados.rados.0.smithi160.stdout:321: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-21T18:07:29.760 INFO:tasks.rados.rados.0.smithi160.stdout:321: writing smithi16030472-321 from 578240 to 1144746 tid 1 2024-09-21T18:07:29.763 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:29.760+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:29.765 INFO:tasks.rados.rados.0.smithi160.stdout:321: writing smithi16030472-321 from 1609179 to 2167277 tid 2 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:321: writing smithi16030472-321 from 2804887 to 3356637 tid 3 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:308: finishing write tid 1 to smithi16030472-308 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:308: finishing write tid 2 to smithi16030472-308 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: finishing write tid 1 to smithi16030472-312 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: finishing write tid 2 to smithi16030472-312 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: finishing write tid 3 to smithi16030472-312 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: finishing write tid 4 to smithi16030472-312 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: finishing write tid 5 to smithi16030472-312 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: oid 312 updating version 0 to 191 2024-09-21T18:07:29.773 INFO:tasks.rados.rados.0.smithi160.stdout:312: oid 312 updating version 191 to 193 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:312: oid 312 updating version 193 to 194 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:312: oid 312 version 194 is already newer than 192 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 312 v 194 (ObjNum 311 snap 0 seq_num 311) dirty exists 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:312: left oid 312 (ObjNum 311 snap 0 seq_num 311) 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:312: done (15 left) 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:322: write initial oid 322 2024-09-21T18:07:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:322: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-21T18:07:29.779 INFO:tasks.rados.rados.0.smithi160.stdout:322: writing smithi16030472-322 from 595047 to 1236802 tid 1 2024-09-21T18:07:29.784 INFO:tasks.rados.rados.0.smithi160.stdout:322: writing smithi16030472-322 from 1667661 to 2085770 tid 2 2024-09-21T18:07:29.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:29.791 INFO:tasks.rados.rados.0.smithi160.stdout:322: writing smithi16030472-322 from 2548663 to 3311685 tid 3 2024-09-21T18:07:29.791 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.859 INFO:tasks.rados.rados.0.smithi160.stdout:311: finishing write tid 1 to smithi16030472-311 2024-09-21T18:07:29.859 INFO:tasks.rados.rados.0.smithi160.stdout:311: finishing write tid 2 to smithi16030472-311 2024-09-21T18:07:29.863 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:29.858+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:29.869 INFO:tasks.rados.rados.0.smithi160.stdout:308: finishing write tid 3 to smithi16030472-308 2024-09-21T18:07:29.869 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:29.869 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:29.877 INFO:tasks.rados.rados.0.smithi160.stdout:313: finishing write tid 1 to smithi16030472-313 2024-09-21T18:07:29.877 INFO:tasks.rados.rados.0.smithi160.stdout:308: finishing write tid 4 to smithi16030472-308 2024-09-21T18:07:29.877 INFO:tasks.rados.rados.0.smithi160.stdout:313: finishing write tid 2 to smithi16030472-313 2024-09-21T18:07:29.877 INFO:tasks.rados.rados.0.smithi160.stdout:313: finishing write tid 3 to smithi16030472-313 2024-09-21T18:07:29.880 INFO:tasks.rados.rados.0.smithi160.stdout:308: finishing write tid 5 to smithi16030472-308 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: oid 308 updating version 0 to 190 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: oid 308 updating version 190 to 191 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: oid 308 version 191 is already newer than 189 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: oid 308 updating version 191 to 192 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 192 (ObjNum 307 snap 0 seq_num 307) dirty exists 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: left oid 308 (ObjNum 307 snap 0 seq_num 307) 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:308: done (15 left) 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:323: write initial oid 323 2024-09-21T18:07:29.881 INFO:tasks.rados.rados.0.smithi160.stdout:323: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-21T18:07:29.885 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:29.888 INFO:tasks.rados.rados.0.smithi160.stdout:323: writing smithi16030472-323 from 611854 to 1328858 tid 1 2024-09-21T18:07:29.895 INFO:tasks.rados.rados.0.smithi160.stdout:323: writing smithi16030472-323 from 2126143 to 2687910 tid 2 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:323: writing smithi16030472-323 from 3259733 to 3411854 tid 3 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: finishing write tid 1 to smithi16030472-307 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: finishing write tid 2 to smithi16030472-307 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: finishing write tid 3 to smithi16030472-307 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: finishing write tid 4 to smithi16030472-307 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: oid 307 updating version 0 to 205 2024-09-21T18:07:29.900 INFO:tasks.rados.rados.0.smithi160.stdout:307: oid 307 updating version 205 to 206 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:307: oid 307 updating version 206 to 207 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 207 (ObjNum 306 snap 0 seq_num 306) dirty exists 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:307: left oid 307 (ObjNum 306 snap 0 seq_num 306) 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:309: finishing write tid 1 to smithi16030472-309 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:307: done (15 left) 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:324: write initial oid 324 2024-09-21T18:07:29.901 INFO:tasks.rados.rados.0.smithi160.stdout:324: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-21T18:07:29.907 INFO:tasks.rados.rados.0.smithi160.stdout:324: writing smithi16030472-324 from 628661 to 1420914 tid 1 2024-09-21T18:07:29.914 INFO:tasks.rados.rados.0.smithi160.stdout:324: writing smithi16030472-324 from 2184625 to 2606403 tid 2 2024-09-21T18:07:29.919 INFO:tasks.rados.rados.0.smithi160.stdout:324: writing smithi16030472-324 from 3403509 to 3428661 tid 3 2024-09-21T18:07:29.919 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:29.919 INFO:tasks.rados.rados.0.smithi160.stdout:309: finishing write tid 2 to smithi16030472-309 2024-09-21T18:07:29.919 INFO:tasks.rados.rados.0.smithi160.stdout:309: finishing write tid 3 to smithi16030472-309 2024-09-21T18:07:29.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:29.957 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:29.958 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:29.964 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:29.958+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:29.978 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:310: finishing write tid 1 to smithi16030472-310 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: finishing write tid 4 to smithi16030472-309 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: finishing write tid 5 to smithi16030472-309 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: oid 309 updating version 0 to 209 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: oid 309 updating version 209 to 210 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: oid 309 updating version 210 to 211 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: oid 309 version 211 is already newer than 208 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 309 v 211 (ObjNum 308 snap 0 seq_num 308) dirty exists 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: left oid 309 (ObjNum 308 snap 0 seq_num 308) 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:309: done (15 left) 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:325: write initial oid 325 2024-09-21T18:07:29.998 INFO:tasks.rados.rados.0.smithi160.stdout:325: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-21T18:07:29.999 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:29.999 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:30.004 INFO:tasks.rados.rados.0.smithi160.stdout:325: writing smithi16030472-325 from 645468 to 1112970 tid 1 2024-09-21T18:07:30.011 INFO:tasks.rados.rados.0.smithi160.stdout:325: writing smithi16030472-325 from 1726754 to 2292190 tid 2 2024-09-21T18:07:30.014 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:30.014 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout:325: writing smithi16030472-325 from 2798226 to 3395064 tid 3 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout:316: finishing write tid 2 to smithi16030472-316 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout:316: finishing write tid 3 to smithi16030472-316 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout:316: finishing write tid 4 to smithi16030472-316 2024-09-21T18:07:30.019 INFO:tasks.rados.rados.0.smithi160.stdout:316: finishing write tid 5 to smithi16030472-316 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: oid 316 updating version 0 to 213 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: oid 316 updating version 213 to 216 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: oid 316 version 216 is already newer than 215 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: oid 316 version 216 is already newer than 214 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 316 v 216 (ObjNum 315 snap 0 seq_num 315) dirty exists 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: left oid 316 (ObjNum 315 snap 0 seq_num 315) 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:311: finishing write tid 3 to smithi16030472-311 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:311: finishing write tid 4 to smithi16030472-311 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:316: done (15 left) 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:326: write initial oid 326 2024-09-21T18:07:30.020 INFO:tasks.rados.rados.0.smithi160.stdout:326: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-21T18:07:30.026 INFO:tasks.rados.rados.0.smithi160.stdout:326: writing smithi16030472-326 from 662275 to 1205026 tid 1 2024-09-21T18:07:30.031 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:30.032 INFO:tasks.rados.rados.0.smithi160.stdout:326: writing smithi16030472-326 from 1785236 to 2210683 tid 2 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout:326: writing smithi16030472-326 from 2942002 to 3462275 tid 3 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout:300: finishing write tid 5 to smithi16030472-300 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout:300: oid 300 updating version 0 to 186 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout:300: oid 300 updating version 186 to 187 2024-09-21T18:07:30.040 INFO:tasks.rados.rados.0.smithi160.stdout:300: oid 300 version 187 is already newer than 185 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:300: oid 300 updating version 187 to 188 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 300 v 188 (ObjNum 299 snap 0 seq_num 299) dirty exists 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:300: left oid 300 (ObjNum 299 snap 0 seq_num 299) 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: finishing write tid 4 to smithi16030472-313 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: finishing write tid 5 to smithi16030472-313 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: oid 313 updating version 0 to 197 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: oid 313 version 197 is already newer than 196 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: oid 313 version 197 is already newer than 195 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: oid 313 updating version 197 to 198 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 313 v 198 (ObjNum 312 snap 0 seq_num 312) dirty exists 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: left oid 313 (ObjNum 312 snap 0 seq_num 312) 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:300: done (15 left) 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:313: done (14 left) 2024-09-21T18:07:30.041 INFO:tasks.rados.rados.0.smithi160.stdout:327: write initial oid 327 2024-09-21T18:07:30.042 INFO:tasks.rados.rados.0.smithi160.stdout:327: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-21T18:07:30.049 INFO:tasks.rados.rados.0.smithi160.stdout:327: writing smithi16030472-327 from 679082 to 1297082 tid 1 2024-09-21T18:07:30.056 INFO:tasks.rados.rados.0.smithi160.stdout:327: writing smithi16030472-327 from 1843718 to 2412823 tid 2 2024-09-21T18:07:30.063 INFO:tasks.rados.rados.0.smithi160.stdout:327: writing smithi16030472-327 from 2853072 to 3479082 tid 3 2024-09-21T18:07:30.063 INFO:tasks.rados.rados.0.smithi160.stdout:328: write initial oid 328 2024-09-21T18:07:30.063 INFO:tasks.rados.rados.0.smithi160.stdout:328: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-21T18:07:30.067 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:30.062+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:30.069 INFO:tasks.rados.rados.0.smithi160.stdout:328: writing smithi16030472-328 from 695889 to 1105491 tid 1 2024-09-21T18:07:30.076 INFO:tasks.rados.rados.0.smithi160.stdout:328: writing smithi16030472-328 from 1618553 to 2331316 tid 2 2024-09-21T18:07:30.082 INFO:tasks.rados.rados.0.smithi160.stdout:328: writing smithi16030472-328 from 2996848 to 3495889 tid 3 2024-09-21T18:07:30.082 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.123 INFO:tasks.rados.rados.0.smithi160.stdout:311: finishing write tid 5 to smithi16030472-311 2024-09-21T18:07:30.123 INFO:tasks.rados.rados.0.smithi160.stdout:311: oid 311 updating version 0 to 258 2024-09-21T18:07:30.123 INFO:tasks.rados.rados.0.smithi160.stdout:311: oid 311 updating version 258 to 260 2024-09-21T18:07:30.123 INFO:tasks.rados.rados.0.smithi160.stdout:311: oid 311 updating version 260 to 261 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:311: oid 311 version 261 is already newer than 259 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 311 v 261 (ObjNum 310 snap 0 seq_num 310) dirty exists 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:311: left oid 311 (ObjNum 310 snap 0 seq_num 310) 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: finishing write tid 1 to smithi16030472-314 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: finishing write tid 2 to smithi16030472-314 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: finishing write tid 3 to smithi16030472-314 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: finishing write tid 4 to smithi16030472-314 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: finishing write tid 5 to smithi16030472-314 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: oid 314 updating version 0 to 262 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: oid 314 updating version 262 to 263 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: oid 314 updating version 263 to 265 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: oid 314 version 265 is already newer than 264 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 314 v 265 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: left oid 314 (ObjNum 313 snap 0 seq_num 313) 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:311: done (15 left) 2024-09-21T18:07:30.124 INFO:tasks.rados.rados.0.smithi160.stdout:314: done (14 left) 2024-09-21T18:07:30.125 INFO:tasks.rados.rados.0.smithi160.stdout:329: write initial oid 329 2024-09-21T18:07:30.125 INFO:tasks.rados.rados.0.smithi160.stdout:329: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-21T18:07:30.130 INFO:tasks.rados.rados.0.smithi160.stdout:329: writing smithi16030472-329 from 712696 to 1197547 tid 1 2024-09-21T18:07:30.139 INFO:tasks.rados.rados.0.smithi160.stdout:329: writing smithi16030472-329 from 1960682 to 2533456 tid 2 2024-09-21T18:07:30.145 INFO:tasks.rados.rados.0.smithi160.stdout:329: writing smithi16030472-329 from 3307918 to 3512696 tid 3 2024-09-21T18:07:30.145 INFO:tasks.rados.rados.0.smithi160.stdout:330: write initial oid 330 2024-09-21T18:07:30.146 INFO:tasks.rados.rados.0.smithi160.stdout:330: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-21T18:07:30.154 INFO:tasks.rados.rados.0.smithi160.stdout:330: writing smithi16030472-330 from 729503 to 1289603 tid 1 2024-09-21T18:07:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:330: writing smithi16030472-330 from 2019164 to 2735596 tid 2 2024-09-21T18:07:30.167 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:30.162+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:30.168 INFO:tasks.rados.rados.0.smithi160.stdout:330: writing smithi16030472-330 from 3335341 to 3529503 tid 3 2024-09-21T18:07:30.168 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.178 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:30.178 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:30.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:30.202 INFO:tasks.rados.rados.0.smithi160.stdout:310: finishing write tid 2 to smithi16030472-310 2024-09-21T18:07:30.202 INFO:tasks.rados.rados.0.smithi160.stdout:310: finishing write tid 3 to smithi16030472-310 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:310: finishing write tid 4 to smithi16030472-310 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:310: oid 310 updating version 0 to 211 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:310: oid 310 updating version 211 to 213 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:310: oid 310 version 213 is already newer than 212 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 213 (ObjNum 309 snap 0 seq_num 309) dirty exists 2024-09-21T18:07:30.203 INFO:tasks.rados.rados.0.smithi160.stdout:310: left oid 310 (ObjNum 309 snap 0 seq_num 309) 2024-09-21T18:07:30.204 INFO:tasks.rados.rados.0.smithi160.stdout:318: finishing write tid 1 to smithi16030472-318 2024-09-21T18:07:30.204 INFO:tasks.rados.rados.0.smithi160.stdout:310: done (15 left) 2024-09-21T18:07:30.204 INFO:tasks.rados.rados.0.smithi160.stdout:331: write initial oid 331 2024-09-21T18:07:30.204 INFO:tasks.rados.rados.0.smithi160.stdout:331: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-21T18:07:30.211 INFO:tasks.rados.rados.0.smithi160.stdout:331: writing smithi16030472-331 from 746310 to 1381659 tid 1 2024-09-21T18:07:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:331: writing smithi16030472-331 from 2077646 to 2654089 tid 2 2024-09-21T18:07:30.223 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:30.223 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:30.223 INFO:tasks.rados.rados.0.smithi160.stdout:331: writing smithi16030472-331 from 3362764 to 3546310 tid 3 2024-09-21T18:07:30.223 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.223 INFO:tasks.rados.rados.0.smithi160.stdout:323: finishing write tid 1 to smithi16030472-323 2024-09-21T18:07:30.224 INFO:tasks.rados.rados.0.smithi160.stdout:323: finishing write tid 2 to smithi16030472-323 2024-09-21T18:07:30.224 INFO:tasks.rados.rados.0.smithi160.stdout:323: finishing write tid 3 to smithi16030472-323 2024-09-21T18:07:30.224 INFO:tasks.rados.rados.0.smithi160.stdout:323: finishing write tid 4 to smithi16030472-323 2024-09-21T18:07:30.226 INFO:tasks.rados.rados.0.smithi160.stdout:315: finishing write tid 1 to smithi16030472-315 2024-09-21T18:07:30.227 INFO:tasks.rados.rados.0.smithi160.stdout:315: finishing write tid 2 to smithi16030472-315 2024-09-21T18:07:30.227 INFO:tasks.rados.rados.0.smithi160.stdout:315: finishing write tid 3 to smithi16030472-315 2024-09-21T18:07:30.227 INFO:tasks.rados.rados.0.smithi160.stdout:315: finishing write tid 4 to smithi16030472-315 2024-09-21T18:07:30.232 INFO:tasks.rados.rados.0.smithi160.stdout:317: finishing write tid 1 to smithi16030472-317 2024-09-21T18:07:30.232 INFO:tasks.rados.rados.0.smithi160.stdout:317: finishing write tid 2 to smithi16030472-317 2024-09-21T18:07:30.242 INFO:tasks.rados.rados.0.smithi160.stdout:319: finishing write tid 1 to smithi16030472-319 2024-09-21T18:07:30.254 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:30.268 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:30.265+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:30.288 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: finishing write tid 5 to smithi16030472-315 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: oid 315 updating version 0 to 269 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: oid 315 version 269 is already newer than 266 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: oid 315 version 269 is already newer than 267 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: oid 315 version 269 is already newer than 268 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 315 v 269 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: left oid 315 (ObjNum 314 snap 0 seq_num 314) 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:315: done (15 left) 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:332: write initial oid 332 2024-09-21T18:07:30.302 INFO:tasks.rados.rados.0.smithi160.stdout:332: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-21T18:07:30.310 INFO:tasks.rados.rados.0.smithi160.stdout:332: writing smithi16030472-332 from 763117 to 1473715 tid 1 2024-09-21T18:07:30.318 INFO:tasks.rados.rados.0.smithi160.stdout:332: writing smithi16030472-332 from 2136128 to 2856229 tid 2 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:332: writing smithi16030472-332 from 3390187 to 3563117 tid 3 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:321: finishing write tid 1 to smithi16030472-321 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: finishing write tid 5 to smithi16030472-323 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: oid 323 updating version 0 to 216 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: oid 323 version 216 is already newer than 215 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: oid 323 updating version 216 to 217 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: oid 323 updating version 217 to 218 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 323 v 218 (ObjNum 322 snap 0 seq_num 322) dirty exists 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: left oid 323 (ObjNum 322 snap 0 seq_num 322) 2024-09-21T18:07:30.322 INFO:tasks.rados.rados.0.smithi160.stdout:323: done (15 left) 2024-09-21T18:07:30.323 INFO:tasks.rados.rados.0.smithi160.stdout:333: write initial oid 333 2024-09-21T18:07:30.323 INFO:tasks.rados.rados.0.smithi160.stdout:333: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-21T18:07:30.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:30.330 INFO:tasks.rados.rados.0.smithi160.stdout:333: writing smithi16030472-333 from 779924 to 1565771 tid 1 2024-09-21T18:07:30.332 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:30.332 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:07:30.337 INFO:tasks.rados.rados.0.smithi160.stdout:333: writing smithi16030472-333 from 2078257 to 2658369 tid 2 2024-09-21T18:07:30.343 INFO:tasks.rados.rados.0.smithi160.stdout:333: writing smithi16030472-333 from 3301257 to 3579924 tid 3 2024-09-21T18:07:30.343 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.343 INFO:tasks.rados.rados.0.smithi160.stdout:317: finishing write tid 3 to smithi16030472-317 2024-09-21T18:07:30.343 INFO:tasks.rados.rados.0.smithi160.stdout:317: finishing write tid 4 to smithi16030472-317 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: finishing write tid 5 to smithi16030472-317 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: oid 317 updating version 0 to 214 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: oid 317 updating version 214 to 217 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: oid 317 version 217 is already newer than 215 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: oid 317 version 217 is already newer than 216 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 217 (ObjNum 316 snap 0 seq_num 316) dirty exists 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:317: left oid 317 (ObjNum 316 snap 0 seq_num 316) 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: finishing write tid 1 to smithi16030472-320 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: finishing write tid 2 to smithi16030472-320 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: finishing write tid 3 to smithi16030472-320 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: finishing write tid 4 to smithi16030472-320 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: finishing write tid 5 to smithi16030472-320 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: oid 320 updating version 0 to 220 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: oid 320 version 220 is already newer than 219 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: oid 320 updating version 220 to 221 2024-09-21T18:07:30.344 INFO:tasks.rados.rados.0.smithi160.stdout:320: oid 320 version 221 is already newer than 218 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 320 v 221 (ObjNum 319 snap 0 seq_num 319) dirty exists 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:320: left oid 320 (ObjNum 319 snap 0 seq_num 319) 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:317: done (15 left) 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:320: done (14 left) 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:334: write initial oid 334 2024-09-21T18:07:30.345 INFO:tasks.rados.rados.0.smithi160.stdout:334: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-21T18:07:30.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:30.349 INFO:tasks.rados.rados.0.smithi160.stdout:334: writing smithi16030472-334 from 796731 to 1257827 tid 1 2024-09-21T18:07:30.356 INFO:tasks.rados.rados.0.smithi160.stdout:334: writing smithi16030472-334 from 1736739 to 2460509 tid 2 2024-09-21T18:07:30.362 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:30.362 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:30.363 INFO:tasks.rados.rados.0.smithi160.stdout:334: writing smithi16030472-334 from 2928680 to 3596731 tid 3 2024-09-21T18:07:30.363 INFO:tasks.rados.rados.0.smithi160.stdout:335: write initial oid 335 2024-09-21T18:07:30.363 INFO:tasks.rados.rados.0.smithi160.stdout:335: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2024-09-21T18:07:30.368 INFO:tasks.rados.rados.0.smithi160.stdout:335: writing smithi16030472-335 from 413538 to 949883 tid 1 2024-09-21T18:07:30.374 INFO:tasks.rados.rados.0.smithi160.stdout:335: writing smithi16030472-335 from 1395221 to 1979002 tid 2 2024-09-21T18:07:30.380 INFO:tasks.rados.rados.0.smithi160.stdout:335: writing smithi16030472-335 from 2556103 to 3214720 tid 3 2024-09-21T18:07:30.380 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.381 INFO:tasks.rados.rados.0.smithi160.stdout:319: finishing write tid 2 to smithi16030472-319 2024-09-21T18:07:30.381 INFO:tasks.rados.rados.0.smithi160.stdout:319: finishing write tid 3 to smithi16030472-319 2024-09-21T18:07:30.382 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:30.382 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:30.382 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:30.380+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:30.419 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:30.420 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:30.444 INFO:tasks.rados.rados.0.smithi160.stdout:321: finishing write tid 2 to smithi16030472-321 2024-09-21T18:07:30.444 INFO:tasks.rados.rados.0.smithi160.stdout:322: finishing write tid 1 to smithi16030472-322 2024-09-21T18:07:30.463 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:30.464 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:30.469 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:30.464+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:30.483 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:30.483 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:30.488 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:30.488 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:30.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:30.508 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:30.508 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:30.521 INFO:tasks.rados.rados.0.smithi160.stdout:318: finishing write tid 2 to smithi16030472-318 2024-09-21T18:07:30.521 INFO:tasks.rados.rados.0.smithi160.stdout:318: finishing write tid 3 to smithi16030472-318 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: finishing write tid 4 to smithi16030472-318 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: finishing write tid 5 to smithi16030472-318 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: oid 318 updating version 0 to 216 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: oid 318 version 216 is already newer than 215 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: oid 318 updating version 216 to 217 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: oid 318 version 217 is already newer than 214 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 318 v 217 (ObjNum 317 snap 0 seq_num 317) dirty exists 2024-09-21T18:07:30.524 INFO:tasks.rados.rados.0.smithi160.stdout:318: left oid 318 (ObjNum 317 snap 0 seq_num 317) 2024-09-21T18:07:30.525 INFO:tasks.rados.rados.0.smithi160.stdout:332: finishing write tid 1 to smithi16030472-332 2024-09-21T18:07:30.525 INFO:tasks.rados.rados.0.smithi160.stdout:332: finishing write tid 2 to smithi16030472-332 2024-09-21T18:07:30.525 INFO:tasks.rados.rados.0.smithi160.stdout:318: done (15 left) 2024-09-21T18:07:30.525 INFO:tasks.rados.rados.0.smithi160.stdout:336: write initial oid 336 2024-09-21T18:07:30.525 INFO:tasks.rados.rados.0.smithi160.stdout:336: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-21T18:07:30.529 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:30.530 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:30.530 INFO:tasks.rados.rados.0.smithi160.stdout:336: writing smithi16030472-336 from 430345 to 1158292 tid 1 2024-09-21T18:07:30.537 INFO:tasks.rados.rados.0.smithi160.stdout:336: writing smithi16030472-336 from 1570056 to 2297495 tid 2 2024-09-21T18:07:30.542 INFO:tasks.rados.rados.0.smithi160.stdout:336: writing smithi16030472-336 from 2699879 to 3169768 tid 3 2024-09-21T18:07:30.542 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: finishing write tid 3 to smithi16030472-321 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: finishing write tid 4 to smithi16030472-321 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: finishing write tid 5 to smithi16030472-321 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: oid 321 updating version 0 to 272 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: oid 321 updating version 272 to 273 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: oid 321 version 273 is already newer than 270 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: oid 321 version 273 is already newer than 271 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 321 v 273 (ObjNum 320 snap 0 seq_num 320) dirty exists 2024-09-21T18:07:30.544 INFO:tasks.rados.rados.0.smithi160.stdout:321: left oid 321 (ObjNum 320 snap 0 seq_num 320) 2024-09-21T18:07:30.545 INFO:tasks.rados.rados.0.smithi160.stdout:325: finishing write tid 1 to smithi16030472-325 2024-09-21T18:07:30.545 INFO:tasks.rados.rados.0.smithi160.stdout:321: done (15 left) 2024-09-21T18:07:30.545 INFO:tasks.rados.rados.0.smithi160.stdout:337: write initial oid 337 2024-09-21T18:07:30.545 INFO:tasks.rados.rados.0.smithi160.stdout:337: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-21T18:07:30.549 INFO:tasks.rados.rados.0.smithi160.stdout:337: writing smithi16030472-337 from 447152 to 850348 tid 1 2024-09-21T18:07:30.554 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:30.552+0000 7f094d1de640 -1 osd.0 50 handle_osd_map injecting map crc failure 2024-09-21T18:07:30.557 INFO:tasks.rados.rados.0.smithi160.stdout:337: writing smithi16030472-337 from 1512185 to 2099635 tid 2 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:337: writing smithi16030472-337 from 2727302 to 3408463 tid 3 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:325: finishing write tid 2 to smithi16030472-325 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:325: finishing write tid 3 to smithi16030472-325 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:325: finishing write tid 4 to smithi16030472-325 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:325: finishing write tid 5 to smithi16030472-325 2024-09-21T18:07:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:325: oid 325 updating version 0 to 274 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:325: oid 325 updating version 274 to 276 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:325: oid 325 updating version 276 to 277 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:325: oid 325 version 277 is already newer than 275 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 325 v 277 (ObjNum 324 snap 0 seq_num 324) dirty exists 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:325: left oid 325 (ObjNum 324 snap 0 seq_num 324) 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:333: finishing write tid 1 to smithi16030472-333 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:322: finishing write tid 2 to smithi16030472-322 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:325: done (15 left) 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:338: write initial oid 338 2024-09-21T18:07:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:338: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-21T18:07:30.570 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:30.565+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:30.570 INFO:tasks.rados.rados.0.smithi160.stdout:338: writing smithi16030472-338 from 463959 to 942404 tid 1 2024-09-21T18:07:30.579 INFO:tasks.rados.rados.0.smithi160.stdout:338: writing smithi16030472-338 from 1570667 to 2301775 tid 2 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:338: writing smithi16030472-338 from 3038372 to 3530805 tid 3 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: finishing write tid 3 to smithi16030472-322 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: finishing write tid 4 to smithi16030472-322 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: finishing write tid 5 to smithi16030472-322 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: oid 322 updating version 0 to 223 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: oid 322 version 223 is already newer than 222 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: oid 322 updating version 223 to 225 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: oid 322 version 225 is already newer than 224 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 322 v 225 (ObjNum 321 snap 0 seq_num 321) dirty exists 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: left oid 322 (ObjNum 321 snap 0 seq_num 321) 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:327: finishing write tid 1 to smithi16030472-327 2024-09-21T18:07:30.589 INFO:tasks.rados.rados.0.smithi160.stdout:322: done (15 left) 2024-09-21T18:07:30.590 INFO:tasks.rados.rados.0.smithi160.stdout:339: write initial oid 339 2024-09-21T18:07:30.590 INFO:tasks.rados.rados.0.smithi160.stdout:339: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-21T18:07:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:339: writing smithi16030472-339 from 480766 to 1034460 tid 1 2024-09-21T18:07:30.601 INFO:tasks.rados.rados.0.smithi160.stdout:339: writing smithi16030472-339 from 1629149 to 2220268 tid 2 2024-09-21T18:07:30.603 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:30.604 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:30.607 INFO:tasks.rados.rados.0.smithi160.stdout:339: writing smithi16030472-339 from 2782148 to 3202206 tid 3 2024-09-21T18:07:30.607 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:333: finishing write tid 2 to smithi16030472-333 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: finishing write tid 1 to smithi16030472-324 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: finishing write tid 2 to smithi16030472-324 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: finishing write tid 3 to smithi16030472-324 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: finishing write tid 4 to smithi16030472-324 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: finishing write tid 5 to smithi16030472-324 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: oid 324 updating version 0 to 202 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: oid 324 updating version 202 to 205 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: oid 324 version 205 is already newer than 203 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: oid 324 version 205 is already newer than 204 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 324 v 205 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:324: left oid 324 (ObjNum 323 snap 0 seq_num 323) 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:326: finishing write tid 1 to smithi16030472-326 2024-09-21T18:07:30.608 INFO:tasks.rados.rados.0.smithi160.stdout:326: finishing write tid 2 to smithi16030472-326 2024-09-21T18:07:30.609 INFO:tasks.rados.rados.0.smithi160.stdout:324: done (15 left) 2024-09-21T18:07:30.609 INFO:tasks.rados.rados.0.smithi160.stdout:340: write initial oid 340 2024-09-21T18:07:30.609 INFO:tasks.rados.rados.0.smithi160.stdout:340: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-21T18:07:30.613 INFO:tasks.rados.rados.0.smithi160.stdout:340: writing smithi16030472-340 from 497573 to 1126516 tid 1 2024-09-21T18:07:30.621 INFO:tasks.rados.rados.0.smithi160.stdout:340: writing smithi16030472-340 from 1687631 to 2422408 tid 2 2024-09-21T18:07:30.626 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:30.626 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:30.626 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:30.626 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:30.626 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:30.628 INFO:tasks.rados.rados.0.smithi160.stdout:340: writing smithi16030472-340 from 3093218 to 3697573 tid 3 2024-09-21T18:07:30.628 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.631 INFO:tasks.rados.rados.0.smithi160.stdout:332: finishing write tid 3 to smithi16030472-332 2024-09-21T18:07:30.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:30.682 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:30.680+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:30.735 INFO:tasks.rados.rados.0.smithi160.stdout:332: finishing write tid 4 to smithi16030472-332 2024-09-21T18:07:30.735 INFO:tasks.rados.rados.0.smithi160.stdout:332: finishing write tid 5 to smithi16030472-332 2024-09-21T18:07:30.735 INFO:tasks.rados.rados.0.smithi160.stdout:332: oid 332 updating version 0 to 222 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:332: oid 332 version 222 is already newer than 219 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:332: oid 332 version 222 is already newer than 221 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:332: oid 332 version 222 is already newer than 220 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 222 (ObjNum 331 snap 0 seq_num 331) dirty exists 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:332: left oid 332 (ObjNum 331 snap 0 seq_num 331) 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:334: finishing write tid 1 to smithi16030472-334 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:332: done (15 left) 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:341: write initial oid 341 2024-09-21T18:07:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:341: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-21T18:07:30.742 INFO:tasks.rados.rados.0.smithi160.stdout:341: writing smithi16030472-341 from 514380 to 1218572 tid 1 2024-09-21T18:07:30.747 INFO:tasks.rados.rados.0.smithi160.stdout:341: writing smithi16030472-341 from 1629760 to 2108195 tid 2 2024-09-21T18:07:30.752 INFO:tasks.rados.rados.0.smithi160.stdout:341: writing smithi16030472-341 from 2604288 to 3046890 tid 3 2024-09-21T18:07:30.752 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.753 INFO:tasks.rados.rados.0.smithi160.stdout:334: finishing write tid 2 to smithi16030472-334 2024-09-21T18:07:30.771 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:30.769+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:30.781 INFO:tasks.rados.rados.0.smithi160.stdout:333: finishing write tid 3 to smithi16030472-333 2024-09-21T18:07:30.781 INFO:tasks.rados.rados.0.smithi160.stdout:327: finishing write tid 2 to smithi16030472-327 2024-09-21T18:07:30.781 INFO:tasks.rados.rados.0.smithi160.stdout:327: finishing write tid 3 to smithi16030472-327 2024-09-21T18:07:30.781 INFO:tasks.rados.rados.0.smithi160.stdout:336: finishing write tid 1 to smithi16030472-336 2024-09-21T18:07:30.781 INFO:tasks.rados.rados.0.smithi160.stdout:327: finishing write tid 4 to smithi16030472-327 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: finishing write tid 4 to smithi16030472-319 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: finishing write tid 5 to smithi16030472-319 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: oid 319 updating version 0 to 200 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: oid 319 updating version 200 to 201 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: oid 319 version 201 is already newer than 199 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: oid 319 updating version 201 to 202 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 319 v 202 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: left oid 319 (ObjNum 318 snap 0 seq_num 318) 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:319: done (15 left) 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:342: write initial oid 342 2024-09-21T18:07:30.789 INFO:tasks.rados.rados.0.smithi160.stdout:342: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-21T18:07:30.799 INFO:tasks.rados.rados.0.smithi160.stdout:342: writing smithi16030472-342 from 531187 to 1310628 tid 1 2024-09-21T18:07:30.808 INFO:tasks.rados.rados.0.smithi160.stdout:342: writing smithi16030472-342 from 2088242 to 2826688 tid 2 2024-09-21T18:07:30.808 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:30.808 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:30.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:30.808 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:30.808 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:30.813 INFO:tasks.rados.rados.0.smithi160.stdout:342: writing smithi16030472-342 from 3431711 to 3731187 tid 3 2024-09-21T18:07:30.813 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:30.849 INFO:tasks.rados.rados.0.smithi160.stdout:326: finishing write tid 3 to smithi16030472-326 2024-09-21T18:07:30.850 INFO:tasks.rados.rados.0.smithi160.stdout:326: finishing write tid 4 to smithi16030472-326 2024-09-21T18:07:30.872 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:30.866+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:326: finishing write tid 5 to smithi16030472-326 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:326: oid 326 updating version 0 to 207 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:326: oid 326 updating version 207 to 208 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:326: oid 326 version 208 is already newer than 206 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:326: oid 326 updating version 208 to 209 2024-09-21T18:07:30.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 326 v 209 (ObjNum 325 snap 0 seq_num 325) dirty exists 2024-09-21T18:07:30.883 INFO:tasks.rados.rados.0.smithi160.stdout:326: left oid 326 (ObjNum 325 snap 0 seq_num 325) 2024-09-21T18:07:30.883 INFO:tasks.rados.rados.0.smithi160.stdout:326: done (15 left) 2024-09-21T18:07:30.883 INFO:tasks.rados.rados.0.smithi160.stdout:343: write initial oid 343 2024-09-21T18:07:30.883 INFO:tasks.rados.rados.0.smithi160.stdout:329: finishing write tid 1 to smithi16030472-329 2024-09-21T18:07:30.883 INFO:tasks.rados.rados.0.smithi160.stdout:343: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-21T18:07:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:343: writing smithi16030472-343 from 547994 to 1002684 tid 1 2024-09-21T18:07:30.902 INFO:tasks.rados.rados.0.smithi160.stdout:343: writing smithi16030472-343 from 1746724 to 2228828 tid 2 2024-09-21T18:07:30.909 INFO:tasks.rados.rados.0.smithi160.stdout:343: writing smithi16030472-343 from 2659134 to 3124280 tid 3 2024-09-21T18:07:30.914 INFO:tasks.rados.rados.0.smithi160.stdout:343: writing smithi16030472-343 from 3605976 to 3747994 tid 4 2024-09-21T18:07:30.914 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.914 INFO:tasks.rados.rados.0.smithi160.stdout:328: finishing write tid 1 to smithi16030472-328 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: finishing write tid 2 to smithi16030472-328 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: finishing write tid 3 to smithi16030472-328 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: finishing write tid 4 to smithi16030472-328 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: finishing write tid 5 to smithi16030472-328 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: oid 328 updating version 0 to 232 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: oid 328 version 232 is already newer than 231 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: oid 328 version 232 is already newer than 230 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: oid 328 updating version 232 to 233 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 233 (ObjNum 327 snap 0 seq_num 327) dirty exists 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: left oid 328 (ObjNum 327 snap 0 seq_num 327) 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:333: finishing write tid 4 to smithi16030472-333 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:328: done (15 left) 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:344: write initial oid 344 2024-09-21T18:07:30.915 INFO:tasks.rados.rados.0.smithi160.stdout:344: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-21T18:07:30.924 INFO:tasks.rados.rados.0.smithi160.stdout:344: writing smithi16030472-344 from 564801 to 1211093 tid 1 2024-09-21T18:07:30.937 INFO:tasks.rados.rados.0.smithi160.stdout:344: writing smithi16030472-344 from 1921559 to 2663674 tid 2 2024-09-21T18:07:30.945 INFO:tasks.rados.rados.0.smithi160.stdout:344: writing smithi16030472-344 from 3202910 to 3764801 tid 3 2024-09-21T18:07:30.946 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1504 3.1 e51/31)", 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.841024+0000", 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.111476194, 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.11150906200000001, 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841024+0000", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841024+0000", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841026+0000", 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1100000000000001e-06 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841033+0000", 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6850000000000001e-06 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841046+0000", 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2874000000000001e-05 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.841054+0000", 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0290000000000005e-06 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1505 3.1 e51/31)", 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.845064+0000", 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.10743599500000001, 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10754099, 2024-09-21T18:07:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845064+0000", 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845064+0000", 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845065+0000", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.249e-06 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845069+0000", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845079+0000", 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0404000000000001e-05 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845085+0000", 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6439999999999997e-06 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1504 3.1 3:9963577d:::smithi16030472-336:head [write 1570056~727439 in=727439b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.545483+0000", 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.407016867, 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40720438799999997, 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.545483+0000", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.545483+0000", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.545484+0000", 2024-09-21T18:07:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6169999999999999e-06 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546389+0000", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00090469699999999999 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546410+0000", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0415e-05 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546421+0000", 2024-09-21T18:07:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1636e-05 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.647872+0000", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.101450789 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.648009+0000", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013726999999999999 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.648221+0000", 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021191 2024-09-21T18:07:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.651726+0000", 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035052859999999998 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1506 3.1 e51/31)", 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.845094+0000", 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.107405498, 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.107688011, 2024-09-21T18:07:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845094+0000", 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845094+0000", 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845095+0000", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7599999999999998e-07 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845099+0000", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8509999999999998e-06 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845103+0000", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0979999999999998e-06 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845106+0000", 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5549999999999998e-06 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1505 3.1 3:9963577d:::smithi16030472-336:head [write 2699879~469889 in=469889b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.546443+0000", 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.406056533, 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40641160300000001, 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546443+0000", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546443+0000", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546445+0000", 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.497e-06 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546684+0000", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023885200000000001 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546690+0000", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546695+0000", 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0669999999999999e-06 2024-09-21T18:07:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.650315+0000", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.103620352 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.650382+0000", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6600000000000006e-05 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.650459+0000", 2024-09-21T18:07:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6977999999999994e-05 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.654149+0000", 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036899060000000002 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1513 3.1 e51/31)", 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.845110+0000", 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.10738973, 2024-09-21T18:07:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.107837499, 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845110+0000", 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845110+0000", 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845110+0000", 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4900000000000002e-07 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845111+0000", 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1599999999999999e-06 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845114+0000", 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.429e-06 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845116+0000", 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5100000000000001e-06 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1506 3.1 3:9963577d:::smithi16030472-336:head [setxattr _header (57) in=64b,truncate 3630345] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.546702+0000", 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.40579732600000001, 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40631946800000002, 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546702+0000", 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546702+0000", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546703+0000", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0299999999999999e-07 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546706+0000", 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8619999999999999e-06 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546710+0000", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8940000000000003e-06 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546714+0000", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7560000000000001e-06 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.653824+0000", 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.107110068 2024-09-21T18:07:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.653866+0000", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2722999999999997e-05 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.653936+0000", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9969999999999996e-05 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.654598+0000", 2024-09-21T18:07:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00066183699999999997 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1514 3.1 e51/31)", 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.845767+0000", 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.106732469, 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.107343353, 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845767+0000", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845767+0000", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845768+0000", 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1629999999999999e-06 2024-09-21T18:07:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845772+0000", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5499999999999999e-06 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845782+0000", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.043e-05 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.845788+0000", 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.519e-06 2024-09-21T18:07:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:738 3.1 3:9963577d:::smithi16030472-336:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.546719+0000", 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.40578097000000002, 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40646518100000001, 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546719+0000", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546719+0000", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546719+0000", 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:07:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546723+0000", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5540000000000002e-06 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546729+0000", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6800000000000004e-06 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.546734+0000", 2024-09-21T18:07:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0779999999999997e-06 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.654182+0000", 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.107448611 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.654279+0000", 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6664999999999997e-05 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1515 3.1 e51/31)", 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.867411+0000", 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.085088618000000005, 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.085848124999999997, 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867411+0000", 2024-09-21T18:07:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867411+0000", 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867413+0000", 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1459999999999999e-06 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867420+0000", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4649999999999999e-06 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867436+0000", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5883e-05 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.867447+0000", 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0879999999999999e-05 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1508 4.0 4:0382d963:::smithi16030472-337:head [write 447152~403196 in=403196b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.631789+0000", 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.32071069000000002, 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32154102000000001, 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1508 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.631789+0000", 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.631789+0000", 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.631791+0000", 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.765e-06 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632031+0000", 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024064999999999999 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632055+0000", 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3643999999999999e-05 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632068+0000", 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2619000000000001e-05 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.726884+0000", 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094816536000000007 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.4876.0:1516 3.1 e51/31)", 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.868043+0000", 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.084456619999999996, 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.08536966, 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868043+0000", 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868043+0000", 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868046+0000", 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5629999999999999e-06 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868054+0000", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1040000000000007e-06 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868078+0000", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4515000000000002e-05 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.868094+0000", 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5463000000000001e-05 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1509 4.0 4:0382d963:::smithi16030472-337:head [write 1512185~587450 in=587450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.632091+0000", 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.32040855299999998, 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.321390069, 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1509 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632091+0000", 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632091+0000", 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632093+0000", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4890000000000001e-06 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632450+0000", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035693299999999998 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632464+0000", 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4469e-05 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632471+0000", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.934e-06 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727167+0000", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094695710000000002 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1538 4.0 4:1baed0db:::smithi16030472-343:head [write 547994~454690 in=454690b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.892555+0000", 2024-09-21T18:07:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.059944929000000001, 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.061012724999999997, 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1538 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892555+0000", 2024-09-21T18:07:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892555+0000", 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892560+0000", 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1429999999999997e-06 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892870+0000", 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031039899999999999 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892897+0000", 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7084999999999999e-05 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.892915+0000", 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7846000000000002e-05 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935688+0000", 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.042773130999999999 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1510 4.0 4:0382d963:::smithi16030472-337:head [write 2727302~681161 in=681161b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.632482+0000", 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.32001749099999999, 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.321180986, 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1510 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632482+0000", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632482+0000", 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632483+0000", 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3500000000000005e-07 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632782+0000", 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029926099999999999 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632792+0000", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6290000000000005e-06 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632797+0000", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2429999999999996e-06 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727324+0000", 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094526537999999993 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1539 4.0 4:1baed0db:::smithi16030472-343:head [write 1746724~482104 in=482104b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.902456+0000", 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.050044125000000002, 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.051292702000000003, 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1539 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902456+0000", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902456+0000", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902460+0000", 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0629999999999999e-06 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902816+0000", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000356487 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902838+0000", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1335000000000002e-05 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.902849+0000", 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1534e-05 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936023+0000", 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.033173566000000002 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1511 4.0 4:0382d963:::smithi16030472-337:head [setxattr _header (57) in=64b,truncate 3647152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.632805+0000", 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.31969447499999998, 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.321074309, 2024-09-21T18:07:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1511 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632805+0000", 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632805+0000", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632806+0000", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8700000000000005e-07 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632809+0000", 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0649999999999999e-06 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632813+0000", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2910000000000001e-06 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632817+0000", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0130000000000004e-06 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727644+0000", 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094826251 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727683+0000", 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9088000000000003e-05 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1540 4.0 4:1baed0db:::smithi16030472-343:head [write 2659134~465146 in=465146b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.909629+0000", 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.042870845999999997, 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.044346918999999999, 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1540 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.909629+0000", 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.909629+0000", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.909631+0000", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6610000000000001e-06 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.909991+0000", 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036059999999999998 2024-09-21T18:07:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.910006+0000", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4854e-05 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.910013+0000", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7009999999999999e-06 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936269+0000", 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.026256218000000001 2024-09-21T18:07:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1512 4.0 4:0382d963:::smithi16030472-337:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.632822+0000", 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.319677339, 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32124219399999998, 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1512 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632822+0000", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632822+0000", 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632823+0000", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6499999999999999e-07 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632825+0000", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2199999999999999e-06 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632831+0000", 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0780000000000001e-06 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632838+0000", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6880000000000002e-06 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727735+0000", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094897640000000005 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.727774+0000", 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8680999999999998e-05 2024-09-21T18:07:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.037840749, 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.039520345999999998, 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:07:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1523 4.0 4:3d0b4c2b:::smithi16030472-340:head [write 497573~628943 in=628943b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.632845+0000", 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.31965506100000002, 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32142880600000001, 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1523 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632845+0000", 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632845+0000", 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.632845+0000", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5600000000000001e-07 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633093+0000", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000248096 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633102+0000", 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3699999999999995e-06 2024-09-21T18:07:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633106+0000", 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8709999999999996e-06 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733643+0000", 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100536059 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.037166154, 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.039028668000000002, 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:07:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:07:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1524 4.0 4:3d0b4c2b:::smithi16030472-340:head [write 1687631~734777 in=734777b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.633114+0000", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.319386051, 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32133678100000002, 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1524 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633114+0000", 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633114+0000", 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633114+0000", 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6199999999999998e-07 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633410+0000", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000296032 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.633419+0000", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4989999999999994e-06 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733501+0000", 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10008175599999999 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.827820+0000", 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.094319900999999998 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.827869+0000", 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8146000000000003e-05 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.037139616, 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.039186291999999998, 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:07:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1525 4.0 4:3d0b4c2b:::smithi16030472-340:head [write 3093218~604355 in=604355b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.733528+0000", 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.21897165900000001, 2024-09-21T18:07:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22110321399999999, 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1525 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733528+0000", 2024-09-21T18:07:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733528+0000", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733530+0000", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3350000000000001e-06 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733858+0000", 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00032792800000000001 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733872+0000", 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733896+0000", 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4073000000000001e-05 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834215+0000", 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100319256 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834265+0000", 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9378000000000001e-05 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.027305874000000001, 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.029543881000000001, 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:07:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1526 4.0 4:3d0b4c2b:::smithi16030472-340:head [setxattr _header (57) in=64b,truncate 3697573] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.733912+0000", 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.218587316, 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22091319200000001, 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1526 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733912+0000", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733912+0000", 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733914+0000", 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.192e-06 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733919+0000", 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1159999999999998e-06 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733928+0000", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2410000000000001e-06 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733935+0000", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5509999999999996e-06 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834294+0000", 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10035928600000001 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834324+0000", 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0672e-05 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.014894086000000001, 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.017318066999999999, 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:07:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:07:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1527 4.0 4:3d0b4c2b:::smithi16030472-340:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.733945+0000", 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.218555108, 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22107117800000001, 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1527 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733945+0000", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733945+0000", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733945+0000", 2024-09-21T18:07:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0500000000000002e-07 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733949+0000", 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1709999999999998e-06 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733957+0000", 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6479999999999992e-06 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.733968+0000", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0866000000000001e-05 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834347+0000", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100379179 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834376+0000", 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8793999999999999e-05 2024-09-21T18:07:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.006004016, 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086202980000000002, 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:07:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4.c98f36f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0051919990000000001, 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0080209510000000001, 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1528 4.0 4:2aa3e9e6:::smithi16030472-341:head [write 514380~704192 in=704192b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.742785+0000", 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.209714491, 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.212625112, 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1528 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.742785+0000", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.742785+0000", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.742788+0000", 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6560000000000001e-06 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.743398+0000", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00060959399999999998 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.743417+0000", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9117e-05 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.743427+0000", 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0067e-05 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.834399+0000", 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.090972440000000002 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4.c98f36f4 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0050837870000000002, 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0080804039999999994, 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "no flag points reached", 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1529 4.0 4:2aa3e9e6:::smithi16030472-341:head [write 1629760~478435 in=478435b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.747744+0000", 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.20475618000000001, 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.207823707, 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1529 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.747744+0000", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.747744+0000", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.747746+0000", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.492e-06 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.748135+0000", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000389078 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.748152+0000", 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7156e-05 2024-09-21T18:07:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.748163+0000", 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0872e-05 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.934773+0000", 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18661009100000001 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1530 4.0 4:2aa3e9e6:::smithi16030472-341:head [write 2604288~442602 in=442602b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.752967+0000", 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19953247399999999, 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.202688437, 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1530 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.752967+0000", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.752967+0000", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.752970+0000", 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3250000000000002e-06 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753656+0000", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00068594900000000004 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753674+0000", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8130000000000001e-05 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753685+0000", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1728e-05 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935207+0000", 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18152182 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935313+0000", 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105635 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1531 4.0 4:2aa3e9e6:::smithi16030472-341:head [setxattr _header (57) in=64b,truncate 3714380] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753706+0000", 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19879381199999999, 2024-09-21T18:07:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20204502999999999, 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1531 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753706+0000", 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753706+0000", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753707+0000", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.159e-06 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753712+0000", 2024-09-21T18:07:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.164e-06 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753720+0000", 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8909999999999995e-06 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753726+0000", 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1809999999999998e-06 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935406+0000", 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.181679176 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935487+0000", 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1539999999999995e-05 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.198764213, 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.202110917, 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:07:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:07:31.094 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1513 3.1 3:b3a70d5e:::smithi16030472-338:head [write 463959~478445 in=478445b,stat out=16b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.758792+0000", 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19370796300000001, 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.19715042599999999, 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.758792+0000", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.758792+0000", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.758795+0000", 2024-09-21T18:07:31.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3270000000000001e-06 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759237+0000", 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044238900000000001 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759255+0000", 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7008e-05 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759263+0000", 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8950000000000004e-06 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759354+0000", 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0273999999999997e-05 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759511+0000", 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000157384 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759741+0000", 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023042200000000001 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1514 3.1 3:b3a70d5e:::smithi16030472-338:head [write 1570667~731108 in=731108b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.759291+0000", 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19320849100000001, 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.196747159, 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759291+0000", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759291+0000", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.759292+0000", 2024-09-21T18:07:31.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.063e-06 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760022+0000", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00072935600000000001 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760032+0000", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0507e-05 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760038+0000", 2024-09-21T18:07:31.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9599999999999997e-06 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761710+0000", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001671659 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761833+0000", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000122946 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761997+0000", 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016377899999999999 2024-09-21T18:07:31.102 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1515 3.1 3:b3a70d5e:::smithi16030472-338:head [write 3038372~492433 in=492433b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.760050+0000", 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19244956699999999, 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.19608362400000001, 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.103 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760050+0000", 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760050+0000", 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760051+0000", 2024-09-21T18:07:31.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0999999999999997e-07 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760459+0000", 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00040835200000000001 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760470+0000", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0291e-05 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760475+0000", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.801e-06 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.863621+0000", 2024-09-21T18:07:31.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.103145792 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.863752+0000", 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131144 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.863931+0000", 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000178932 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1495 4.0 4:0bfeb78a:::smithi16030472-334:head [write 2928680~668051 in=668051b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.420454+0000", 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.53204586899999995, 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.535770792, 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1495 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.420454+0000", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.420454+0000", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.420455+0000", 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.54e-07 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.420792+0000", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00033731800000000003 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.420804+0000", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2542e-05 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520887+0000", 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100082937 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520977+0000", 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9293e-05 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1516 3.1 3:b3a70d5e:::smithi16030472-338:head [setxattr _header (57) in=64b,truncate 3663959] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.760485+0000", 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.192014938, 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.195833956, 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760485+0000", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760485+0000", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760486+0000", 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9599999999999998e-07 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760489+0000", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1379999999999999e-06 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760495+0000", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2600000000000002e-06 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760499+0000", 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8970000000000001e-06 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.865809+0000", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.105309966 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.865895+0000", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6144999999999996e-05 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.866021+0000", 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012594 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1496 4.0 4:0bfeb78a:::smithi16030472-334:head [setxattr _header (57) in=64b,truncate 3596731] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.520920+0000", 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.43157973700000002, 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43549086799999998, 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1496 2024-09-21T18:07:31.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520920+0000", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520920+0000", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520923+0000", 2024-09-21T18:07:31.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1549999999999999e-06 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520941+0000", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7362000000000001e-05 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520956+0000", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5313000000000001e-05 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520967+0000", 2024-09-21T18:07:31.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0672e-05 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.521194+0000", 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000227612 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:743 3.1 3.7ab0e5cd (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.760504+0000", 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19199556400000001, 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.19599434399999999, 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760504+0000", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760504+0000", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760505+0000", 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6699999999999999e-07 2024-09-21T18:07:31.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760507+0000", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0109999999999999e-06 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760511+0000", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7330000000000003e-06 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760515+0000", 2024-09-21T18:07:31.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9720000000000003e-06 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1497 4.0 4:0bfeb78a:::smithi16030472-334:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e50)", 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.520980+0000", 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.43151968299999999, 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43558555900000001, 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.121 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1497 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520980+0000", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520980+0000", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520981+0000", 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.006e-06 2024-09-21T18:07:31.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520986+0000", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6550000000000003e-06 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.520995+0000", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1560000000000001e-06 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.521003+0000", 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0160000000000008e-06 2024-09-21T18:07:31.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.521395+0000", 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039180299999999998 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1518 3.1 3.65f6a549 (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.760731+0000", 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.19176827099999999, 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.195919603, 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760731+0000", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760731+0000", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.760732+0000", 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1699999999999998e-07 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761536+0000", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00080365600000000003 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761545+0000", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5449999999999992e-06 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761551+0000", 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6960000000000002e-06 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.126 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1519 3.1 3.65f6a549 (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.761596+0000", 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.190903347, 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.195118716, 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761596+0000", 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761596+0000", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.761598+0000", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4789999999999999e-06 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.762436+0000", 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083803899999999995 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.762459+0000", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3136e-05 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.762477+0000", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7900000000000001e-05 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1520 3.1 3.65f6a549 (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.772444+0000", 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.180056199, 2024-09-21T18:07:31.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.184338998, 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772444+0000", 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772444+0000", 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772446+0000", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7690000000000001e-06 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772807+0000", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036022799999999998 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772823+0000", 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6138999999999999e-05 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772832+0000", 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0110000000000006e-06 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1521 3.1 3.65f6a549 (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.772982+0000", 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.179517279, 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18386448499999999, 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772982+0000", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772982+0000", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772984+0000", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772988+0000", 2024-09-21T18:07:31.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1409999999999998e-06 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.772996+0000", 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5609999999999999e-06 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773001+0000", 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0869999999999997e-06 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:748 3.1 3.65f6a549 (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.773445+0000", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.17905481600000001, 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18346552899999999, 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773445+0000", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773445+0000", 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773446+0000", 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.362e-06 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773450+0000", 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7670000000000001e-06 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773460+0000", 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0288e-05 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.773466+0000", 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5010000000000004e-06 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 50 2024-09-21T18:07:31.137 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:31.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:31.140 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:31.136+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: finishing write tid 2 to smithi16030472-336 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: finishing write tid 3 to smithi16030472-336 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: finishing write tid 4 to smithi16030472-336 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: finishing write tid 5 to smithi16030472-336 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: oid 336 updating version 0 to 284 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: oid 336 version 284 is already newer than 281 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: oid 336 version 284 is already newer than 282 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: oid 336 version 284 is already newer than 283 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 284 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: left oid 336 (ObjNum 335 snap 0 seq_num 335) 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:336: done (15 left) 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:345: write initial oid 345 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:345: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-21T18:07:31.142 INFO:tasks.rados.rados.0.smithi160.stdout:345: writing smithi16030472-345 from 581608 to 1303149 tid 1 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:345: writing smithi16030472-345 from 1863688 to 2349461 tid 2 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:345: writing smithi16030472-345 from 3113980 to 3718023 tid 3 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:338: finishing write tid 1 to smithi16030472-338 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:338: finishing write tid 2 to smithi16030472-338 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: finishing write tid 5 to smithi16030472-333 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: oid 333 updating version 0 to 279 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: oid 333 updating version 279 to 281 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: oid 333 version 281 is already newer than 280 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: oid 333 version 281 is already newer than 278 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 281 (ObjNum 332 snap 0 seq_num 332) dirty exists 2024-09-21T18:07:31.143 INFO:tasks.rados.rados.0.smithi160.stdout:333: left oid 333 (ObjNum 332 snap 0 seq_num 332) 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:329: finishing write tid 2 to smithi16030472-329 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:333: done (15 left) 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:346: write initial oid 346 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:346: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:346: writing smithi16030472-346 from 598415 to 1395205 tid 1 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:346: writing smithi16030472-346 from 1922170 to 2667954 tid 2 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:346: writing smithi16030472-346 from 3141403 to 3556718 tid 3 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: finishing write tid 3 to smithi16030472-338 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: finishing write tid 4 to smithi16030472-338 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: finishing write tid 5 to smithi16030472-338 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: oid 338 updating version 0 to 286 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: oid 338 version 286 is already newer than 285 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: oid 338 updating version 286 to 288 2024-09-21T18:07:31.144 INFO:tasks.rados.rados.0.smithi160.stdout:338: oid 338 version 288 is already newer than 287 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 338 v 288 (ObjNum 337 snap 0 seq_num 337) dirty exists 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:338: left oid 338 (ObjNum 337 snap 0 seq_num 337) 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:338: done (15 left) 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:347: write initial oid 347 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:347: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:347: writing smithi16030472-347 from 615222 to 1087261 tid 1 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:347: writing smithi16030472-347 from 1580652 to 2070094 tid 2 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout:347: writing smithi16030472-347 from 2768826 to 3395413 tid 3 2024-09-21T18:07:31.145 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: finishing write tid 3 to smithi16030472-334 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: finishing write tid 4 to smithi16030472-334 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: finishing write tid 5 to smithi16030472-334 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: oid 334 updating version 0 to 223 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: oid 334 updating version 223 to 224 2024-09-21T18:07:31.151 INFO:tasks.rados.rados.0.smithi160.stdout:334: oid 334 updating version 224 to 226 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:334: oid 334 version 226 is already newer than 225 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 334 v 226 (ObjNum 333 snap 0 seq_num 333) dirty exists 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:334: left oid 334 (ObjNum 333 snap 0 seq_num 333) 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:334: done (15 left) 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:348: write initial oid 348 2024-09-21T18:07:31.152 INFO:tasks.rados.rados.0.smithi160.stdout:348: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-21T18:07:31.160 INFO:tasks.rados.rados.0.smithi160.stdout:348: writing smithi16030472-348 from 632029 to 1179317 tid 1 2024-09-21T18:07:31.168 INFO:tasks.rados.rados.0.smithi160.stdout:348: writing smithi16030472-348 from 1639134 to 2388587 tid 2 2024-09-21T18:07:31.175 INFO:tasks.rados.rados.0.smithi160.stdout:348: writing smithi16030472-348 from 2796249 to 3234108 tid 3 2024-09-21T18:07:31.179 INFO:tasks.rados.rados.0.smithi160.stdout:348: writing smithi16030472-348 from 3653524 to 3832029 tid 4 2024-09-21T18:07:31.179 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.195 INFO:tasks.rados.rados.0.smithi160.stdout:346: finishing write tid 1 to smithi16030472-346 2024-09-21T18:07:31.239 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:31.236+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:31.262 INFO:tasks.rados.rados.0.smithi160.stdout:337: finishing write tid 1 to smithi16030472-337 2024-09-21T18:07:31.263 INFO:tasks.rados.rados.0.smithi160.stdout:337: finishing write tid 2 to smithi16030472-337 2024-09-21T18:07:31.263 INFO:tasks.rados.rados.0.smithi160.stdout:337: finishing write tid 3 to smithi16030472-337 2024-09-21T18:07:31.263 INFO:tasks.rados.rados.0.smithi160.stdout:340: finishing write tid 1 to smithi16030472-340 2024-09-21T18:07:31.297 INFO:tasks.rados.rados.0.smithi160.stdout:339: finishing write tid 1 to smithi16030472-339 2024-09-21T18:07:31.297 INFO:tasks.rados.rados.0.smithi160.stdout:339: finishing write tid 2 to smithi16030472-339 2024-09-21T18:07:31.297 INFO:tasks.rados.rados.0.smithi160.stdout:339: finishing write tid 3 to smithi16030472-339 2024-09-21T18:07:31.297 INFO:tasks.rados.rados.0.smithi160.stdout:339: finishing write tid 4 to smithi16030472-339 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: finishing write tid 5 to smithi16030472-339 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: oid 339 updating version 0 to 292 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: oid 339 version 292 is already newer than 290 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: oid 339 version 292 is already newer than 291 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: oid 339 version 292 is already newer than 289 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 339 v 292 (ObjNum 338 snap 0 seq_num 338) dirty exists 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: left oid 339 (ObjNum 338 snap 0 seq_num 338) 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:339: done (15 left) 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:349: write initial oid 349 2024-09-21T18:07:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:349: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-21T18:07:31.307 INFO:tasks.rados.rados.0.smithi160.stdout:349: writing smithi16030472-349 from 648836 to 1271373 tid 1 2024-09-21T18:07:31.316 INFO:tasks.rados.rados.0.smithi160.stdout:349: writing smithi16030472-349 from 1981263 to 2474374 tid 2 2024-09-21T18:07:31.319 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:31.319 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:31.326 INFO:tasks.rados.rados.0.smithi160.stdout:349: writing smithi16030472-349 from 3107319 to 3848836 tid 3 2024-09-21T18:07:31.326 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:31.342 INFO:tasks.rados.rados.0.smithi160.stdout:329: finishing write tid 3 to smithi16030472-329 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: finishing write tid 4 to smithi16030472-329 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: finishing write tid 5 to smithi16030472-329 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: oid 329 updating version 0 to 210 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: oid 329 updating version 210 to 212 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: oid 329 updating version 212 to 213 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: oid 329 version 213 is already newer than 211 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 329 v 213 (ObjNum 328 snap 0 seq_num 328) dirty exists 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:329: left oid 329 (ObjNum 328 snap 0 seq_num 328) 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:330: finishing write tid 1 to smithi16030472-330 2024-09-21T18:07:31.343 INFO:tasks.rados.rados.0.smithi160.stdout:330: finishing write tid 2 to smithi16030472-330 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: finishing write tid 3 to smithi16030472-330 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: finishing write tid 4 to smithi16030472-330 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: finishing write tid 5 to smithi16030472-330 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: oid 330 updating version 0 to 214 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: oid 330 updating version 214 to 216 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: oid 330 updating version 216 to 217 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: oid 330 version 217 is already newer than 215 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 330 v 217 (ObjNum 329 snap 0 seq_num 329) dirty exists 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: left oid 330 (ObjNum 329 snap 0 seq_num 329) 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:331: finishing write tid 1 to smithi16030472-331 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:331: finishing write tid 2 to smithi16030472-331 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:329: done (15 left) 2024-09-21T18:07:31.344 INFO:tasks.rados.rados.0.smithi160.stdout:330: done (14 left) 2024-09-21T18:07:31.345 INFO:tasks.rados.rados.0.smithi160.stdout:350: write initial oid 350 2024-09-21T18:07:31.345 INFO:tasks.rados.rados.0.smithi160.stdout:350: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2024-09-21T18:07:31.353 INFO:tasks.rados.rados.0.smithi160.stdout:350: writing smithi16030472-350 from 665643 to 1363429 tid 1 2024-09-21T18:07:31.365 INFO:tasks.rados.rados.0.smithi160.stdout:350: writing smithi16030472-350 from 2039745 to 2792867 tid 2 2024-09-21T18:07:31.373 INFO:tasks.rados.rados.0.smithi160.stdout:350: writing smithi16030472-350 from 3534742 to 3865643 tid 3 2024-09-21T18:07:31.374 INFO:tasks.rados.rados.0.smithi160.stdout:351: write initial oid 351 2024-09-21T18:07:31.374 INFO:tasks.rados.rados.0.smithi160.stdout:351: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-21T18:07:31.378 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:31.376+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:31.382 INFO:tasks.rados.rados.0.smithi160.stdout:351: writing smithi16030472-351 from 682450 to 1171838 tid 1 2024-09-21T18:07:31.391 INFO:tasks.rados.rados.0.smithi160.stdout:351: writing smithi16030472-351 from 1814580 to 2311360 tid 2 2024-09-21T18:07:31.400 INFO:tasks.rados.rados.0.smithi160.stdout:351: writing smithi16030472-351 from 2878518 to 3666546 tid 3 2024-09-21T18:07:31.400 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.400 INFO:tasks.rados.rados.0.smithi160.stdout:341: finishing write tid 1 to smithi16030472-341 2024-09-21T18:07:31.441 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:31.436+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:31.460 INFO:tasks.rados.rados.0.smithi160.stdout:341: finishing write tid 2 to smithi16030472-341 2024-09-21T18:07:31.485 INFO:tasks.rados.rados.0.smithi160.stdout:340: finishing write tid 2 to smithi16030472-340 2024-09-21T18:07:31.485 INFO:tasks.rados.rados.0.smithi160.stdout:340: finishing write tid 3 to smithi16030472-340 2024-09-21T18:07:31.486 INFO:tasks.rados.rados.0.smithi160.stdout:337: finishing write tid 4 to smithi16030472-337 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:158 4.0 4:1192aa23:::smithi16030472-40:head [write 2099033~556440] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.513 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209276+0000", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.299589527999998, 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.075556736, 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 158 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209276+0000", 2024-09-21T18:07:31.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209276+0000", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209282+0000", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7880000000000003e-06 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209730+0000", 2024-09-21T18:07:31.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044851500000000002 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209756+0000", 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5454999999999999e-05 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209770+0000", 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209881+0000", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011105800000000001 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209984+0000", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000103191 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.731260+0000", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.52127637000000004 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.731399+0000", 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000138622 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.731597+0000", 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019844599999999999 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.741456+0000", 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0098585359999999993 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.284596+0000", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.54314011799999995 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.284699+0000", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000103212 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.284833+0000", 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013379 2024-09-21T18:07:31.519 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.299067705999999, 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:07:31.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:07:31.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:160 4.0 4:1192aa23:::smithi16030472-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.309995+0000", 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.198870803999998, 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.075469027, 2024-09-21T18:07:31.526 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 160 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309995+0000", 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309995+0000", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309999+0000", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310012+0000", 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3437e-05 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310031+0000", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8913e-05 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310045+0000", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4175999999999999e-05 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410295+0000", 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10024971100000001 2024-09-21T18:07:31.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410348+0000", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3044999999999998e-05 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742693+0000", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33234512599999999 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742825+0000", 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013188699999999999 2024-09-21T18:07:31.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385207+0000", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64238246600000004 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385341+0000", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013395400000000001 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385464+0000", 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000122662 2024-09-21T18:07:31.531 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:167 4.0 4:01b9dd46:::smithi16030472-42:head [setxattr _header (56) in=7b,truncate 2689087] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.311065+0000", 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.197800955000002, 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1748382470000001, 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.532 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 167 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311065+0000", 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311065+0000", 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311066+0000", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2899999999999999e-06 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311072+0000", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7529999999999998e-06 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311080+0000", 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2239999999999995e-06 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311087+0000", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.25e-06 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410731+0000", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.099643727000000001 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410763+0000", 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1612999999999997e-05 2024-09-21T18:07:31.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.741562+0000", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33079892 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.741695+0000", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013313800000000001 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.741885+0000", 2024-09-21T18:07:31.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000190834 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742923+0000", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010372490000000001 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.485721+0000", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.74279785399999998 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.485809+0000", 2024-09-21T18:07:31.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8075e-05 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.485903+0000", 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4320000000000005e-05 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:168 4.0 4:01b9dd46:::smithi16030472-42:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.311100+0000", 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.197765945, 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175159587, 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 168 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311100+0000", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311100+0000", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311101+0000", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311108+0000", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3499999999999999e-06 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311118+0000", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0343000000000001e-05 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.311126+0000", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1999999999999997e-06 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410790+0000", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.099664224999999995 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.410814+0000", 2024-09-21T18:07:31.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3794999999999998e-05 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742438+0000", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33162438700000002 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742576+0000", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013807399999999999 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.485992+0000", 2024-09-21T18:07:31.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.74341578100000005 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486121+0000", 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012921800000000001 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486259+0000", 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013829099999999999 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:176 4.0 4:364cb6d5:::smithi16030472-44:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.312615+0000", 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.196251106999998, 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1740791310000001, 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 176 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312615+0000", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312615+0000", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312616+0000", 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3599999999999999e-06 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312621+0000", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0459999999999996e-06 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312634+0000", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.289e-05 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.312645+0000", 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0813e-05 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.411355+0000", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.098710592 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.411382+0000", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6574999999999999e-05 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.754158+0000", 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.34277646299999998 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.754289+0000", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013040400000000001 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486478+0000", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.732189278 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486577+0000", 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9426000000000007e-05 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486694+0000", 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000116284 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:178 4.0 4:16d9fc1d:::smithi16030472-45:head [write 1875090~648012] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.364648+0000", 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.144217941000001, 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1223188020000001, 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 178 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.364648+0000", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.364648+0000", 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.364651+0000", 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5599999999999998e-06 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365436+0000", 2024-09-21T18:07:31.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00078491699999999999 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365469+0000", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2509999999999999e-05 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365485+0000", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6212999999999999e-05 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.511820+0000", 2024-09-21T18:07:31.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.14633468699999999 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.511917+0000", 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6953999999999994e-05 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.952384+0000", 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44046693100000001 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.952522+0000", 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000138064 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.952707+0000", 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000185327 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.955897+0000", 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031903000000000001 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486874+0000", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.53097709800000004 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486924+0000", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9972000000000002e-05 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.486967+0000", 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2268999999999998e-05 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.555 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:179 4.0 4:16d9fc1d:::smithi16030472-45:head [setxattr _header (56) in=7b,truncate 2739508] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.365516+0000", 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.143349407999999, 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1215869009999999, 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 179 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.556 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365516+0000", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365516+0000", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365518+0000", 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8059999999999999e-06 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365525+0000", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6440000000000003e-06 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.365537+0000", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2583e-05 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465664+0000", 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100126732 2024-09-21T18:07:31.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.511978+0000", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.046313528999999999 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.512040+0000", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2532000000000004e-05 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.955121+0000", 2024-09-21T18:07:31.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.443080847 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.955281+0000", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016033399999999999 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.955469+0000", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000187474 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.956523+0000", 2024-09-21T18:07:31.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001053802 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.487006+0000", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.53048328300000003 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.487052+0000", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5736999999999999e-05 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.487103+0000", 2024-09-21T18:07:31.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1598000000000003e-05 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:180 4.0 4:16d9fc1d:::smithi16030472-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.465697+0000", 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.043168550000001, 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.121657667, 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.562 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 180 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465697+0000", 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465697+0000", 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465700+0000", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3800000000000001e-06 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465710+0000", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0083e-05 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465727+0000", 2024-09-21T18:07:31.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6956000000000002e-05 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.465738+0000", 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0957e-05 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.512133+0000", 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.046395618 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.512216+0000", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3281000000000004e-05 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.955973+0000", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44375697600000003 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.956106+0000", 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132891 2024-09-21T18:07:31.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587221+0000", 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.631114799 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587289+0000", 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7569000000000003e-05 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587355+0000", 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6156999999999997e-05 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:183 4.0 4:15fed93e:::smithi16030472-46:head [setxattr _header (56) in=7b,truncate 2756315] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.467263+0000", 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.041603013, 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1202395860000001, 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.568 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 183 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467263+0000", 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467263+0000", 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467264+0000", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.24e-06 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467269+0000", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3329999999999996e-06 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467277+0000", 2024-09-21T18:07:31.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1130000000000001e-06 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.467284+0000", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9999999999999999e-06 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.567588+0000", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100304123 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.567672+0000", 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3715999999999996e-05 2024-09-21T18:07:31.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.067567+0000", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.49989492800000002 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.067728+0000", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016095399999999999 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.067944+0000", 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000215727 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.177430+0000", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.109486613 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587442+0000", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41001168900000001 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587473+0000", 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0531000000000003e-05 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587502+0000", 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9618999999999999e-05 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:191 4.0 4:1f69a690:::smithi16030472-48:head [setxattr _header (56) in=7b,truncate 2789929] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.574 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.569358+0000", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.939508086, 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.018506575, 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 191 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569358+0000", 2024-09-21T18:07:31.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569358+0000", 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569359+0000", 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3659999999999999e-06 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569367+0000", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.977e-06 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569375+0000", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569383+0000", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3259999999999998e-06 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570120+0000", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073764499999999999 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570206+0000", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5817999999999997e-05 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.176308+0000", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.60610187199999999 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.176460+0000", 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015165799999999999 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.176668+0000", 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020844 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.284868+0000", 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10820031500000001 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587672+0000", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30280374100000002 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587760+0000", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7842000000000004e-05 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587864+0000", 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000104102 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:192 4.0 4:1f69a690:::smithi16030472-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.569393+0000", 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.939473014000001, 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0186350369999999, 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 192 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.581 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569393+0000", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569393+0000", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569394+0000", 2024-09-21T18:07:31.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569397+0000", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6339999999999999e-06 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569405+0000", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.569412+0000", 2024-09-21T18:07:31.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7499999999999997e-06 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570599+0000", 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0011873229999999999 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570681+0000", 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2242000000000003e-05 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.177245+0000", 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.606563615 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.177408+0000", 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016306099999999999 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587903+0000", 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41049494199999997 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.587968+0000", 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5041000000000003e-05 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.588028+0000", 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9565999999999999e-05 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.586 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:198 4.0 4:11edfedf:::smithi16030472-50:head [write 1767500~799008] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.570483+0000", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.938383072000001, 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.017913176, 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 198 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570483+0000", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570483+0000", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.570484+0000", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7129999999999999e-06 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.571409+0000", 2024-09-21T18:07:31.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00092475100000000002 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.571424+0000", 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4843e-05 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.571433+0000", 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6969999999999999e-06 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.612246+0000", 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.040813470999999997 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.612327+0000", 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0605999999999994e-05 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.181313+0000", 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.56898665400000004 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.181461+0000", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000147637 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.181665+0000", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020376300000000001 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385147+0000", 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.203482407 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.588255+0000", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.203107959 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.588337+0000", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2101999999999994e-05 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.588396+0000", 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8573000000000003e-05 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.592 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1001 3.1 3:837e1005:::smithi16030472-224:head [write 547961~554764,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.738088+0000", 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.7707779959999996, 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.054345699, 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.738088+0000", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.738088+0000", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.738092+0000", 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1300000000000003e-06 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.738702+0000", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00061000900000000001 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.738730+0000", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8615e-05 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.838857+0000", 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100126547 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.249333+0000", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.410475963 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.249519+0000", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000186193 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.249753+0000", 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023372700000000001 2024-09-21T18:07:31.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.260829+0000", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011075814 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.792268+0000", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.53143947999999996 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.792357+0000", 2024-09-21T18:07:31.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8974999999999994e-05 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.792433+0000", 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6246000000000001e-05 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:522 3.1 3:8894617b:::smithi16030472-225:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.945623+0000", 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.5632431709999999, 2024-09-21T18:07:31.598 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1512844719999999, 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945623+0000", 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945623+0000", 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945623+0000", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7899999999999999e-07 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945626+0000", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945630+0000", 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.972e-06 2024-09-21T18:07:31.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.945634+0000", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.264155+0000", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.31852090100000002 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.264281+0000", 2024-09-21T18:07:31.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000126193 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:26.095477+0000", 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.83119573000000002 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:26.095599+0000", 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012214300000000001 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.602 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:26.096907+0000", 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0013081480000000001 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1037 3.1 3:93f3a785:::smithi16030472-231:head [write 1909329~458405] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.988551+0000", 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.5203149089999997, 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0066204910000001, 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.603 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988551+0000", 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988551+0000", 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988553+0000", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7e-06 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988869+0000", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031538999999999997 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988892+0000", 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3289e-05 2024-09-21T18:07:31.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988911+0000", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.861e-05 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.267654+0000", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.27874330899999999 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.267778+0000", 2024-09-21T18:07:31.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012385599999999999 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.267971+0000", 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019313699999999999 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.272989+0000", 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.005018008 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.607 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.994955+0000", 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.72196633399999999 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995044+0000", 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8294999999999994e-05 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995171+0000", 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127563 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1038 3.1 3:93f3a785:::smithi16030472-231:head [write 2816700~410210] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.988938+0000", 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.519927397, 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0065484920000001, 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988938+0000", 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988938+0000", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.988941+0000", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3499999999999999e-06 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989506+0000", 2024-09-21T18:07:31.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00056499899999999995 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989528+0000", 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.215e-05 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989546+0000", 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7978999999999998e-05 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.611 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.269904+0000", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.280358528 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.270031+0000", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012680999999999999 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.270238+0000", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020697200000000001 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.375416+0000", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.105177984 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995323+0000", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.61990674899999998 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995414+0000", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1192000000000004e-05 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995487+0000", 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2779000000000002e-05 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1039 3.1 3:93f3a785:::smithi16030472-231:head [setxattr _header (57) in=7b,truncate 3865610] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.989572+0000", 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.5192937830000002, 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.006141969, 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.614 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989572+0000", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989572+0000", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989574+0000", 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1900000000000002e-06 2024-09-21T18:07:31.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989585+0000", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0917000000000001e-05 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989606+0000", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.107e-05 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989622+0000", 2024-09-21T18:07:31.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6373000000000001e-05 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.272089+0000", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.28246675199999999 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.272241+0000", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000151679 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.272489+0000", 2024-09-21T18:07:31.617 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000247693 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.377408+0000", 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10491948500000001 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995559+0000", 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.61815052199999998 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995633+0000", 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4785999999999999e-05 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995714+0000", 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0501999999999996e-05 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:531 3.1 3:93f3a785:::smithi16030472-231:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.989871+0000", 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.5189951300000004, 2024-09-21T18:07:31.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0061865409999999, 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989871+0000", 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989871+0000", 2024-09-21T18:07:31.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989873+0000", 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4770000000000002e-06 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989880+0000", 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5699999999999998e-06 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989900+0000", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0758999999999999e-05 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.989919+0000", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8219999999999998e-05 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.364268+0000", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.37434990600000001 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:31.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.364408+0000", 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013904600000000001 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995782+0000", 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.63137483100000003 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.995906+0000", 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012327599999999999 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.996057+0000", 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015145600000000001 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1046 3.1 3:956c7121:::smithi16030472-233:head [write 699224~548358,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:24.990270+0000", 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 6.5185961749999999, 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0059730950000001, 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:31.624 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.990270+0000", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.990270+0000", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.990272+0000", 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.323e-06 2024-09-21T18:07:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.990991+0000", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071901400000000002 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.991015+0000", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4422000000000001e-05 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:24.991035+0000", 2024-09-21T18:07:31.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9230000000000001e-05 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.364468+0000", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.37343390999999998 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.364649+0000", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018016700000000001 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 14", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.364885+0000", 2024-09-21T18:07:31.627 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023639399999999999 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.377416+0000", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012530800999999999 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.996113+0000", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.61869760799999995 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.996186+0000", 2024-09-21T18:07:31.628 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2442999999999997e-05 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:25.996243+0000", 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6783000000000002e-05 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:31.629 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:31.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: finishing write tid 5 to smithi16030472-327 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: oid 327 updating version 0 to 226 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: oid 327 updating version 226 to 227 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: oid 327 updating version 227 to 228 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: oid 327 updating version 228 to 229 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 327 v 229 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:07:31.633 INFO:tasks.rados.rados.0.smithi160.stdout:327: left oid 327 (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: finishing write tid 2 to smithi16030472-346 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: finishing write tid 3 to smithi16030472-346 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: finishing write tid 4 to smithi16030472-346 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:327: done (15 left) 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:352: write initial oid 352 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:352: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:352: writing smithi16030472-352 from 699257 to 1263894 tid 1 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:352: writing smithi16030472-352 from 1873062 to 2513500 tid 2 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:352: writing smithi16030472-352 from 3305941 to 3899257 tid 3 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: finishing write tid 5 to smithi16030472-346 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: oid 346 updating version 0 to 237 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: oid 346 version 237 is already newer than 234 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: oid 346 version 237 is already newer than 236 2024-09-21T18:07:31.634 INFO:tasks.rados.rados.0.smithi160.stdout:346: oid 346 version 237 is already newer than 235 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 346 v 237 (ObjNum 345 snap 0 seq_num 345) dirty exists 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:346: left oid 346 (ObjNum 345 snap 0 seq_num 345) 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:346: done (15 left) 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:353: write initial oid 353 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:353: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:353: writing smithi16030472-353 from 716064 to 1355950 tid 1 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:353: writing smithi16030472-353 from 1815191 to 2315640 tid 2 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:353: writing smithi16030472-353 from 2817011 to 3343936 tid 3 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:337: finishing write tid 5 to smithi16030472-337 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:337: oid 337 updating version 0 to 233 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:337: oid 337 version 233 is already newer than 228 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:337: oid 337 version 233 is already newer than 229 2024-09-21T18:07:31.635 INFO:tasks.rados.rados.0.smithi160.stdout:337: oid 337 version 233 is already newer than 227 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 337 v 233 (ObjNum 336 snap 0 seq_num 336) dirty exists 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:337: left oid 337 (ObjNum 336 snap 0 seq_num 336) 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:337: done (15 left) 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:354: write initial oid 354 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:354: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:354: writing smithi16030472-354 from 732871 to 1448006 tid 1 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:354: writing smithi16030472-354 from 1873673 to 2517780 tid 2 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:354: writing smithi16030472-354 from 3244434 to 3932871 tid 3 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: finishing write tid 4 to smithi16030472-340 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: finishing write tid 5 to smithi16030472-340 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: oid 340 updating version 0 to 232 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: oid 340 version 232 is already newer than 230 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: oid 340 updating version 232 to 234 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:340: oid 340 version 234 is already newer than 231 2024-09-21T18:07:31.636 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 340 v 234 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:340: left oid 340 (ObjNum 339 snap 0 seq_num 339) 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 1 to smithi16030472-343 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:340: done (15 left) 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:355: write initial oid 355 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:355: seq_num 354 ranges {749678=790384,2332155=504118,3271857=549469} 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:355: writing smithi16030472-355 from 749678 to 1540062 tid 1 2024-09-21T18:07:31.637 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:31.626+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:31.637 INFO:tasks.rados.rados.0.smithi160.stdout:355: writing smithi16030472-355 from 2332155 to 2836273 tid 2 2024-09-21T18:07:31.642 INFO:tasks.rados.rados.0.smithi160.stdout:355: writing smithi16030472-355 from 3271857 to 3821326 tid 3 2024-09-21T18:07:31.642 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.642 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 2 to smithi16030472-343 2024-09-21T18:07:31.642 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 3 to smithi16030472-343 2024-09-21T18:07:31.642 INFO:tasks.rados.rados.0.smithi160.stdout:350: finishing write tid 1 to smithi16030472-350 2024-09-21T18:07:31.706 INFO:tasks.rados.rados.0.smithi160.stdout:331: finishing write tid 3 to smithi16030472-331 2024-09-21T18:07:31.706 INFO:tasks.rados.rados.0.smithi160.stdout:331: finishing write tid 4 to smithi16030472-331 2024-09-21T18:07:31.706 INFO:tasks.rados.rados.0.smithi160.stdout:331: finishing write tid 5 to smithi16030472-331 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: oid 331 updating version 0 to 220 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: oid 331 version 220 is already newer than 218 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: oid 331 updating version 220 to 221 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: oid 331 version 221 is already newer than 219 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 331 v 221 (ObjNum 330 snap 0 seq_num 330) dirty exists 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: left oid 331 (ObjNum 330 snap 0 seq_num 330) 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:331: done (15 left) 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:356: write initial oid 356 2024-09-21T18:07:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:356: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-21T18:07:31.714 INFO:tasks.rados.rados.0.smithi160.stdout:356: writing smithi16030472-356 from 766485 to 1232118 tid 1 2024-09-21T18:07:31.722 INFO:tasks.rados.rados.0.smithi160.stdout:356: writing smithi16030472-356 from 1990637 to 2638413 tid 2 2024-09-21T18:07:31.732 INFO:tasks.rados.rados.0.smithi160.stdout:356: writing smithi16030472-356 from 3299280 to 3966485 tid 3 2024-09-21T18:07:31.732 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:335: finishing write tid 1 to smithi16030472-335 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:335: finishing write tid 2 to smithi16030472-335 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:335: finishing write tid 3 to smithi16030472-335 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:341: finishing write tid 3 to smithi16030472-341 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: finishing write tid 1 to smithi16030472-349 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: finishing write tid 2 to smithi16030472-349 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: finishing write tid 3 to smithi16030472-349 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: finishing write tid 4 to smithi16030472-349 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: finishing write tid 5 to smithi16030472-349 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: oid 349 updating version 0 to 240 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: oid 349 version 240 is already newer than 238 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: oid 349 updating version 240 to 241 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: oid 349 version 241 is already newer than 239 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 349 v 241 (ObjNum 348 snap 0 seq_num 348) dirty exists 2024-09-21T18:07:31.733 INFO:tasks.rados.rados.0.smithi160.stdout:349: left oid 349 (ObjNum 348 snap 0 seq_num 348) 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:354: finishing write tid 1 to smithi16030472-354 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:350: finishing write tid 2 to smithi16030472-350 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:350: finishing write tid 3 to smithi16030472-350 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: finishing write tid 4 to smithi16030472-335 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: finishing write tid 5 to smithi16030472-335 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: oid 335 updating version 0 to 223 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: oid 335 updating version 223 to 225 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: oid 335 version 225 is already newer than 222 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: oid 335 version 225 is already newer than 224 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 335 v 225 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:335: left oid 335 (ObjNum 334 snap 0 seq_num 334) 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:342: finishing write tid 1 to smithi16030472-342 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:342: finishing write tid 2 to smithi16030472-342 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:347: finishing write tid 1 to smithi16030472-347 2024-09-21T18:07:31.734 INFO:tasks.rados.rados.0.smithi160.stdout:347: finishing write tid 2 to smithi16030472-347 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: finishing write tid 3 to smithi16030472-347 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: finishing write tid 4 to smithi16030472-347 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: finishing write tid 5 to smithi16030472-347 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: oid 347 updating version 0 to 230 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: oid 347 updating version 230 to 231 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: oid 347 version 231 is already newer than 229 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: oid 347 version 231 is already newer than 228 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 231 (ObjNum 346 snap 0 seq_num 346) dirty exists 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: left oid 347 (ObjNum 346 snap 0 seq_num 346) 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:335: done (15 left) 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:347: done (14 left) 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:349: done (13 left) 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:357: write initial oid 357 2024-09-21T18:07:31.735 INFO:tasks.rados.rados.0.smithi160.stdout:357: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2024-09-21T18:07:31.736 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:31.729+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:31.741 INFO:tasks.rados.rados.0.smithi160.stdout:357: writing smithi16030472-357 from 783292 to 1324174 tid 1 2024-09-21T18:07:31.749 INFO:tasks.rados.rados.0.smithi160.stdout:357: writing smithi16030472-357 from 1932766 to 2440553 tid 2 2024-09-21T18:07:31.758 INFO:tasks.rados.rados.0.smithi160.stdout:357: writing smithi16030472-357 from 3210350 to 3782363 tid 3 2024-09-21T18:07:31.758 INFO:tasks.rados.rados.0.smithi160.stdout:358: write initial oid 358 2024-09-21T18:07:31.758 INFO:tasks.rados.rados.0.smithi160.stdout:358: seq_num 357 ranges {400099=616131,1591248=408851} 2024-09-21T18:07:31.764 INFO:tasks.rados.rados.0.smithi160.stdout:358: writing smithi16030472-358 from 400099 to 1016230 tid 1 2024-09-21T18:07:31.770 INFO:tasks.rados.rados.0.smithi160.stdout:358: writing smithi16030472-358 from 1591248 to 2000099 tid 2 2024-09-21T18:07:31.770 INFO:tasks.rados.rados.0.smithi160.stdout:359: write initial oid 359 2024-09-21T18:07:31.771 INFO:tasks.rados.rados.0.smithi160.stdout:359: seq_num 358 ranges {416906=407733,1366083=511456} 2024-09-21T18:07:31.775 INFO:tasks.rados.rados.0.smithi160.stdout:359: writing smithi16030472-359 from 416906 to 824639 tid 1 2024-09-21T18:07:31.781 INFO:tasks.rados.rados.0.smithi160.stdout:359: writing smithi16030472-359 from 1366083 to 1877539 tid 2 2024-09-21T18:07:31.781 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.812 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:31.812 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:31.812 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:31.812 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:31.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: finishing write tid 4 to smithi16030472-350 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: finishing write tid 5 to smithi16030472-350 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: oid 350 updating version 0 to 294 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: oid 350 version 294 is already newer than 293 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: oid 350 updating version 294 to 295 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: oid 350 updating version 295 to 296 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 296 (ObjNum 349 snap 0 seq_num 349) dirty exists 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: left oid 350 (ObjNum 349 snap 0 seq_num 349) 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:354: finishing write tid 2 to smithi16030472-354 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:350: done (15 left) 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:360: write initial oid 360 2024-09-21T18:07:31.831 INFO:tasks.rados.rados.0.smithi160.stdout:360: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-21T18:07:31.833 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:31.829+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:31.836 INFO:tasks.rados.rados.0.smithi160.stdout:360: writing smithi16030472-360 from 433713 to 916695 tid 1 2024-09-21T18:07:31.842 INFO:tasks.rados.rados.0.smithi160.stdout:360: writing smithi16030472-360 from 1424565 to 2033713 tid 2 2024-09-21T18:07:31.842 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.843 INFO:tasks.rados.rados.0.smithi160.stdout:351: finishing write tid 1 to smithi16030472-351 2024-09-21T18:07:31.843 INFO:tasks.rados.rados.0.smithi160.stdout:351: finishing write tid 2 to smithi16030472-351 2024-09-21T18:07:31.843 INFO:tasks.rados.rados.0.smithi160.stdout:351: finishing write tid 3 to smithi16030472-351 2024-09-21T18:07:31.843 INFO:tasks.rados.rados.0.smithi160.stdout:351: finishing write tid 4 to smithi16030472-351 2024-09-21T18:07:31.896 INFO:tasks.rados.rados.0.smithi160.stdout:341: finishing write tid 4 to smithi16030472-341 2024-09-21T18:07:31.933 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:31.928+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: finishing write tid 5 to smithi16030472-351 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: oid 351 updating version 0 to 234 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: oid 351 updating version 234 to 235 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: oid 351 version 235 is already newer than 232 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: oid 351 version 235 is already newer than 233 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 351 v 235 (ObjNum 350 snap 0 seq_num 350) dirty exists 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:351: left oid 351 (ObjNum 350 snap 0 seq_num 350) 2024-09-21T18:07:31.946 INFO:tasks.rados.rados.0.smithi160.stdout:353: finishing write tid 1 to smithi16030472-353 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: finishing write tid 3 to smithi16030472-342 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: finishing write tid 4 to smithi16030472-342 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: finishing write tid 5 to smithi16030472-342 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: oid 342 updating version 0 to 229 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: oid 342 version 229 is already newer than 227 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: oid 342 version 229 is already newer than 226 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: oid 342 version 229 is already newer than 228 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 229 (ObjNum 341 snap 0 seq_num 341) dirty exists 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: left oid 342 (ObjNum 341 snap 0 seq_num 341) 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:342: done (15 left) 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:351: done (14 left) 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:361: write initial oid 361 2024-09-21T18:07:31.947 INFO:tasks.rados.rados.0.smithi160.stdout:361: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-21T18:07:31.952 INFO:tasks.rados.rados.0.smithi160.stdout:361: writing smithi16030472-361 from 450520 to 1008751 tid 1 2024-09-21T18:07:31.957 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:31.957 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:31.957 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:31.958 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:31.958 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:31.958 INFO:tasks.rados.rados.0.smithi160.stdout:361: writing smithi16030472-361 from 1483047 to 1998172 tid 2 2024-09-21T18:07:31.959 INFO:tasks.rados.rados.0.smithi160.stdout:362: write initial oid 362 2024-09-21T18:07:31.959 INFO:tasks.rados.rados.0.smithi160.stdout:362: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-21T18:07:31.965 INFO:tasks.rados.rados.0.smithi160.stdout:362: writing smithi16030472-362 from 467327 to 1100807 tid 1 2024-09-21T18:07:31.971 INFO:tasks.rados.rados.0.smithi160.stdout:362: writing smithi16030472-362 from 1825176 to 2067327 tid 2 2024-09-21T18:07:31.971 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:31.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:31.997 INFO:tasks.rados.rados.0.smithi160.stdout:341: finishing write tid 5 to smithi16030472-341 2024-09-21T18:07:31.997 INFO:tasks.rados.rados.0.smithi160.stdout:341: oid 341 updating version 0 to 232 2024-09-21T18:07:31.997 INFO:tasks.rados.rados.0.smithi160.stdout:341: oid 341 updating version 232 to 236 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:341: oid 341 version 236 is already newer than 231 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:341: oid 341 version 236 is already newer than 235 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 236 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:341: left oid 341 (ObjNum 340 snap 0 seq_num 340) 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:341: done (15 left) 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:363: write initial oid 363 2024-09-21T18:07:31.998 INFO:tasks.rados.rados.0.smithi160.stdout:363: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-21T18:07:32.005 INFO:tasks.rados.rados.0.smithi160.stdout:363: writing smithi16030472-363 from 484134 to 1192863 tid 1 2024-09-21T18:07:32.011 INFO:tasks.rados.rados.0.smithi160.stdout:363: writing smithi16030472-363 from 1883658 to 2084134 tid 2 2024-09-21T18:07:32.011 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.032 INFO:tasks.rados.rados.0.smithi160.stdout:354: finishing write tid 3 to smithi16030472-354 2024-09-21T18:07:32.034 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:32.030+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:32.060 INFO:tasks.rados.rados.0.smithi160.stdout:357: finishing write tid 1 to smithi16030472-357 2024-09-21T18:07:32.113 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 4 to smithi16030472-343 2024-09-21T18:07:32.113 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 5 to smithi16030472-343 2024-09-21T18:07:32.113 INFO:tasks.rados.rados.0.smithi160.stdout:343: finishing write tid 6 to smithi16030472-343 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: oid 343 updating version 0 to 234 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: oid 343 updating version 234 to 237 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: oid 343 version 237 is already newer than 236 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: oid 343 version 237 is already newer than 233 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: oid 343 version 237 is already newer than 235 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 237 (ObjNum 342 snap 0 seq_num 342) dirty exists 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: left oid 343 (ObjNum 342 snap 0 seq_num 342) 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:343: done (15 left) 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:364: write initial oid 364 2024-09-21T18:07:32.114 INFO:tasks.rados.rados.0.smithi160.stdout:364: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-21T18:07:32.123 INFO:tasks.rados.rados.0.smithi160.stdout:364: writing smithi16030472-364 from 500941 to 1284919 tid 1 2024-09-21T18:07:32.129 INFO:tasks.rados.rados.0.smithi160.stdout:364: writing smithi16030472-364 from 1942140 to 2100941 tid 2 2024-09-21T18:07:32.129 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: finishing write tid 2 to smithi16030472-353 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: finishing write tid 3 to smithi16030472-353 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: finishing write tid 4 to smithi16030472-353 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: finishing write tid 5 to smithi16030472-353 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: oid 353 updating version 0 to 236 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: oid 353 updating version 236 to 239 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: oid 353 version 239 is already newer than 238 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: oid 353 version 239 is already newer than 237 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 353 v 239 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: left oid 353 (ObjNum 352 snap 0 seq_num 352) 2024-09-21T18:07:32.135 INFO:tasks.rados.rados.0.smithi160.stdout:353: done (15 left) 2024-09-21T18:07:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:365: write initial oid 365 2024-09-21T18:07:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:365: seq_num 364 ranges {517748=459227,1600622=406110} 2024-09-21T18:07:32.136 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:32.132+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:32.141 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:32.141 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:32.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:32.141 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:32.141 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:32.143 INFO:tasks.rados.rados.0.smithi160.stdout:365: writing smithi16030472-365 from 517748 to 976975 tid 1 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:365: writing smithi16030472-365 from 1600622 to 2006732 tid 2 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: finishing write tid 4 to smithi16030472-354 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: finishing write tid 5 to smithi16030472-354 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: oid 354 updating version 0 to 245 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: oid 354 version 245 is already newer than 244 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: oid 354 version 245 is already newer than 243 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: oid 354 version 245 is already newer than 242 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 354 v 245 (ObjNum 353 snap 0 seq_num 353) dirty exists 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:354: left oid 354 (ObjNum 353 snap 0 seq_num 353) 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:356: finishing write tid 1 to smithi16030472-356 2024-09-21T18:07:32.152 INFO:tasks.rados.rados.0.smithi160.stdout:344: finishing write tid 1 to smithi16030472-344 2024-09-21T18:07:32.153 INFO:tasks.rados.rados.0.smithi160.stdout:354: done (15 left) 2024-09-21T18:07:32.153 INFO:tasks.rados.rados.0.smithi160.stdout:366: write initial oid 366 2024-09-21T18:07:32.153 INFO:tasks.rados.rados.0.smithi160.stdout:366: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-21T18:07:32.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:32.162 INFO:tasks.rados.rados.0.smithi160.stdout:366: writing smithi16030472-366 from 534555 to 1185384 tid 1 2024-09-21T18:07:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:366: writing smithi16030472-366 from 1659104 to 2134555 tid 2 2024-09-21T18:07:32.170 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:357: finishing write tid 2 to smithi16030472-357 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: finishing write tid 3 to smithi16030472-357 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: finishing write tid 4 to smithi16030472-357 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: finishing write tid 5 to smithi16030472-357 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: oid 357 updating version 0 to 299 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: oid 357 version 299 is already newer than 297 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: oid 357 version 299 is already newer than 298 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: oid 357 updating version 299 to 300 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 357 v 300 (ObjNum 356 snap 0 seq_num 356) dirty exists 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: left oid 357 (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:359: finishing write tid 1 to smithi16030472-359 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:357: done (15 left) 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:367: write initial oid 367 2024-09-21T18:07:32.171 INFO:tasks.rados.rados.0.smithi160.stdout:367: seq_num 366 ranges {551362=726078,1717586=409779} 2024-09-21T18:07:32.181 INFO:tasks.rados.rados.0.smithi160.stdout:367: writing smithi16030472-367 from 551362 to 1277440 tid 1 2024-09-21T18:07:32.188 INFO:tasks.rados.rados.0.smithi160.stdout:367: writing smithi16030472-367 from 1717586 to 2127365 tid 2 2024-09-21T18:07:32.188 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.235 INFO:tasks.rados.rados.0.smithi160.stdout:356: finishing write tid 2 to smithi16030472-356 2024-09-21T18:07:32.236 INFO:tasks.rados.rados.0.smithi160.stdout:356: finishing write tid 3 to smithi16030472-356 2024-09-21T18:07:32.236 INFO:tasks.rados.rados.0.smithi160.stdout:356: finishing write tid 4 to smithi16030472-356 2024-09-21T18:07:32.236 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:32.233+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: finishing write tid 5 to smithi16030472-356 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: oid 356 updating version 0 to 247 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: oid 356 updating version 247 to 249 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: oid 356 version 249 is already newer than 246 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: oid 356 version 249 is already newer than 248 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 356 v 249 (ObjNum 355 snap 0 seq_num 355) dirty exists 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: left oid 356 (ObjNum 355 snap 0 seq_num 355) 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:356: done (15 left) 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:368: write initial oid 368 2024-09-21T18:07:32.237 INFO:tasks.rados.rados.0.smithi160.stdout:368: seq_num 367 ranges {568169=401327,1376068=669790} 2024-09-21T18:07:32.244 INFO:tasks.rados.rados.0.smithi160.stdout:368: writing smithi16030472-368 from 568169 to 969496 tid 1 2024-09-21T18:07:32.252 INFO:tasks.rados.rados.0.smithi160.stdout:368: writing smithi16030472-368 from 1376068 to 2045858 tid 2 2024-09-21T18:07:32.252 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.252 INFO:tasks.rados.rados.0.smithi160.stdout:345: finishing write tid 1 to smithi16030472-345 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: finishing write tid 2 to smithi16030472-345 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: finishing write tid 3 to smithi16030472-345 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: finishing write tid 4 to smithi16030472-345 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: finishing write tid 5 to smithi16030472-345 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: oid 345 updating version 0 to 242 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: oid 345 version 242 is already newer than 240 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: oid 345 version 242 is already newer than 239 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: oid 345 version 242 is already newer than 241 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 345 v 242 (ObjNum 344 snap 0 seq_num 344) dirty exists 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: left oid 345 (ObjNum 344 snap 0 seq_num 344) 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 1 to smithi16030472-348 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 2 to smithi16030472-348 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:352: finishing write tid 1 to smithi16030472-352 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:352: finishing write tid 2 to smithi16030472-352 2024-09-21T18:07:32.253 INFO:tasks.rados.rados.0.smithi160.stdout:345: done (15 left) 2024-09-21T18:07:32.254 INFO:tasks.rados.rados.0.smithi160.stdout:369: write initial oid 369 2024-09-21T18:07:32.254 INFO:tasks.rados.rados.0.smithi160.stdout:369: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-21T18:07:32.260 INFO:tasks.rados.rados.0.smithi160.stdout:369: writing smithi16030472-369 from 584976 to 1061552 tid 1 2024-09-21T18:07:32.269 INFO:tasks.rados.rados.0.smithi160.stdout:369: writing smithi16030472-369 from 1834550 to 2184976 tid 2 2024-09-21T18:07:32.269 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: finishing write tid 2 to smithi16030472-359 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: finishing write tid 3 to smithi16030472-359 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: finishing write tid 4 to smithi16030472-359 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: oid 359 updating version 0 to 303 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: oid 359 version 303 is already newer than 302 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: oid 359 version 303 is already newer than 301 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 303 (ObjNum 358 snap 0 seq_num 358) dirty exists 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: left oid 359 (ObjNum 358 snap 0 seq_num 358) 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:368: finishing write tid 1 to smithi16030472-368 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:368: finishing write tid 2 to smithi16030472-368 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:368: finishing write tid 3 to smithi16030472-368 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:359: done (15 left) 2024-09-21T18:07:32.270 INFO:tasks.rados.rados.0.smithi160.stdout:370: write initial oid 370 2024-09-21T18:07:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:370: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-21T18:07:32.278 INFO:tasks.rados.rados.0.smithi160.stdout:370: writing smithi16030472-370 from 601783 to 1153608 tid 1 2024-09-21T18:07:32.286 INFO:tasks.rados.rados.0.smithi160.stdout:370: writing smithi16030472-370 from 1776679 to 2201783 tid 2 2024-09-21T18:07:32.286 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.327 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:32.327 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:32.327 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:32.327 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:32.336 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:32.330+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:32.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: finishing write tid 4 to smithi16030472-368 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: oid 368 updating version 0 to 250 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: oid 368 updating version 250 to 251 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: oid 368 updating version 251 to 252 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 368 v 252 (ObjNum 367 snap 0 seq_num 367) dirty exists 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: left oid 368 (ObjNum 367 snap 0 seq_num 367) 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:368: done (15 left) 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:371: write initial oid 371 2024-09-21T18:07:32.347 INFO:tasks.rados.rados.0.smithi160.stdout:371: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-21T18:07:32.356 INFO:tasks.rados.rados.0.smithi160.stdout:371: writing smithi16030472-371 from 618590 to 1245664 tid 1 2024-09-21T18:07:32.364 INFO:tasks.rados.rados.0.smithi160.stdout:371: writing smithi16030472-371 from 1835161 to 2218590 tid 2 2024-09-21T18:07:32.364 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:07:32.371 INFO:tasks.rados.rados.0.smithi160.stdout:364: finishing write tid 1 to smithi16030472-364 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: finishing write tid 2 to smithi16030472-364 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: finishing write tid 3 to smithi16030472-364 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: finishing write tid 4 to smithi16030472-364 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: oid 364 updating version 0 to 238 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: oid 364 version 238 is already newer than 237 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: oid 364 updating version 238 to 239 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 239 (ObjNum 363 snap 0 seq_num 363) dirty exists 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: left oid 364 (ObjNum 363 snap 0 seq_num 363) 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:364: done (15 left) 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:372: write initial oid 372 2024-09-21T18:07:32.373 INFO:tasks.rados.rados.0.smithi160.stdout:372: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-21T18:07:32.385 INFO:tasks.rados.rados.0.smithi160.stdout:372: writing smithi16030472-372 from 635397 to 1337720 tid 1 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:372: writing smithi16030472-372 from 1893643 to 2235397 tid 2 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:365: finishing write tid 1 to smithi16030472-365 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:365: finishing write tid 2 to smithi16030472-365 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:371: finishing write tid 1 to smithi16030472-371 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:371: finishing write tid 2 to smithi16030472-371 2024-09-21T18:07:32.394 INFO:tasks.rados.rados.0.smithi160.stdout:352: finishing write tid 3 to smithi16030472-352 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:352: finishing write tid 4 to smithi16030472-352 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: finishing write tid 1 to smithi16030472-355 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: finishing write tid 2 to smithi16030472-355 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: finishing write tid 3 to smithi16030472-355 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: finishing write tid 4 to smithi16030472-355 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: finishing write tid 5 to smithi16030472-355 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: oid 355 updating version 0 to 250 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: oid 355 version 250 is already newer than 249 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: oid 355 updating version 250 to 251 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: oid 355 updating version 251 to 252 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 252 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:355: left oid 355 (ObjNum 354 snap 0 seq_num 354) 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:358: finishing write tid 1 to smithi16030472-358 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 3 to smithi16030472-348 2024-09-21T18:07:32.395 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 4 to smithi16030472-348 2024-09-21T18:07:32.396 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 5 to smithi16030472-348 2024-09-21T18:07:32.396 INFO:tasks.rados.rados.0.smithi160.stdout:355: done (15 left) 2024-09-21T18:07:32.396 INFO:tasks.rados.rados.0.smithi160.stdout:373: write initial oid 373 2024-09-21T18:07:32.396 INFO:tasks.rados.rados.0.smithi160.stdout:373: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-21T18:07:32.407 INFO:tasks.rados.rados.0.smithi160.stdout:373: writing smithi16030472-373 from 652204 to 1429776 tid 1 2024-09-21T18:07:32.414 INFO:tasks.rados.rados.0.smithi160.stdout:373: writing smithi16030472-373 from 1952125 to 2252204 tid 2 2024-09-21T18:07:32.415 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.437 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:32.430+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:32.456 INFO:tasks.rados.rados.0.smithi160.stdout:365: finishing write tid 3 to smithi16030472-365 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: finishing write tid 4 to smithi16030472-365 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: oid 365 updating version 0 to 242 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: oid 365 version 242 is already newer than 241 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: oid 365 version 242 is already newer than 240 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 365 v 242 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: left oid 365 (ObjNum 364 snap 0 seq_num 364) 2024-09-21T18:07:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:365: done (15 left) 2024-09-21T18:07:32.458 INFO:tasks.rados.rados.0.smithi160.stdout:374: write initial oid 374 2024-09-21T18:07:32.458 INFO:tasks.rados.rados.0.smithi160.stdout:374: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-21T18:07:32.467 INFO:tasks.rados.rados.0.smithi160.stdout:374: writing smithi16030472-374 from 669011 to 1238185 tid 1 2024-09-21T18:07:32.477 INFO:tasks.rados.rados.0.smithi160.stdout:374: writing smithi16030472-374 from 2010607 to 2269011 tid 2 2024-09-21T18:07:32.477 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.477 INFO:tasks.rados.rados.0.smithi160.stdout:366: finishing write tid 1 to smithi16030472-366 2024-09-21T18:07:32.477 INFO:tasks.rados.rados.0.smithi160.stdout:366: finishing write tid 2 to smithi16030472-366 2024-09-21T18:07:32.483 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:32.483 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:32.483 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:32.483 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:32.483 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:32.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:32.538 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:32.534+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:32.567 INFO:tasks.rados.rados.0.smithi160.stdout:371: finishing write tid 3 to smithi16030472-371 2024-09-21T18:07:32.567 INFO:tasks.rados.rados.0.smithi160.stdout:360: finishing write tid 1 to smithi16030472-360 2024-09-21T18:07:32.567 INFO:tasks.rados.rados.0.smithi160.stdout:360: finishing write tid 2 to smithi16030472-360 2024-09-21T18:07:32.567 INFO:tasks.rados.rados.0.smithi160.stdout:360: finishing write tid 3 to smithi16030472-360 2024-09-21T18:07:32.567 INFO:tasks.rados.rados.0.smithi160.stdout:360: finishing write tid 4 to smithi16030472-360 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:360: oid 360 updating version 0 to 306 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:360: oid 360 version 306 is already newer than 304 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:360: oid 360 version 306 is already newer than 305 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 306 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:360: left oid 360 (ObjNum 359 snap 0 seq_num 359) 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:361: finishing write tid 1 to smithi16030472-361 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:360: done (15 left) 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:375: write initial oid 375 2024-09-21T18:07:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:375: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-21T18:07:32.579 INFO:tasks.rados.rados.0.smithi160.stdout:375: writing smithi16030472-375 from 685818 to 1330241 tid 1 2024-09-21T18:07:32.588 INFO:tasks.rados.rados.0.smithi160.stdout:375: writing smithi16030472-375 from 2069089 to 2285818 tid 2 2024-09-21T18:07:32.588 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:362: finishing write tid 1 to smithi16030472-362 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:366: finishing write tid 3 to smithi16030472-366 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:366: finishing write tid 4 to smithi16030472-366 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:366: oid 366 updating version 0 to 244 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:366: oid 366 updating version 244 to 245 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:366: oid 366 version 245 is already newer than 243 2024-09-21T18:07:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 245 (ObjNum 365 snap 0 seq_num 365) dirty exists 2024-09-21T18:07:32.590 INFO:tasks.rados.rados.0.smithi160.stdout:366: left oid 366 (ObjNum 365 snap 0 seq_num 365) 2024-09-21T18:07:32.590 INFO:tasks.rados.rados.0.smithi160.stdout:366: done (15 left) 2024-09-21T18:07:32.590 INFO:tasks.rados.rados.0.smithi160.stdout:376: write initial oid 376 2024-09-21T18:07:32.590 INFO:tasks.rados.rados.0.smithi160.stdout:376: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-21T18:07:32.600 INFO:tasks.rados.rados.0.smithi160.stdout:376: writing smithi16030472-376 from 702625 to 1422297 tid 1 2024-09-21T18:07:32.608 INFO:tasks.rados.rados.0.smithi160.stdout:376: writing smithi16030472-376 from 2127571 to 2302625 tid 2 2024-09-21T18:07:32.608 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.612 INFO:tasks.rados.rados.0.smithi160.stdout:348: finishing write tid 6 to smithi16030472-348 2024-09-21T18:07:32.612 INFO:tasks.rados.rados.0.smithi160.stdout:348: oid 348 updating version 0 to 247 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: oid 348 version 247 is already newer than 244 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: oid 348 version 247 is already newer than 245 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: oid 348 version 247 is already newer than 243 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: oid 348 version 247 is already newer than 246 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 348 v 247 (ObjNum 347 snap 0 seq_num 347) dirty exists 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: left oid 348 (ObjNum 347 snap 0 seq_num 347) 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:344: finishing write tid 2 to smithi16030472-344 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:348: done (15 left) 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:377: write initial oid 377 2024-09-21T18:07:32.613 INFO:tasks.rados.rados.0.smithi160.stdout:377: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.721911183000003, 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:32.619 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:07:32.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:07:32.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:07:32.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 45.636518793999997, 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:07:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.624 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:07:32.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:32.627 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:32.628 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:32.628 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:32.628 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:32.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:32.629 INFO:tasks.rados.rados.0.smithi160.stdout:377: writing smithi16030472-377 from 719432 to 1514353 tid 1 2024-09-21T18:07:32.633 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:32.634 INFO:tasks.rados.rados.0.smithi160.stdout:377: writing smithi16030472-377 from 2186053 to 2319432 tid 2 2024-09-21T18:07:32.635 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.639 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:32.634+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:367: finishing write tid 1 to smithi16030472-367 2024-09-21T18:07:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:367: finishing write tid 2 to smithi16030472-367 2024-09-21T18:07:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:367: finishing write tid 3 to smithi16030472-367 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: finishing write tid 4 to smithi16030472-367 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: oid 367 updating version 0 to 248 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: oid 367 version 248 is already newer than 247 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: oid 367 version 248 is already newer than 246 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 248 (ObjNum 366 snap 0 seq_num 366) dirty exists 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: left oid 367 (ObjNum 366 snap 0 seq_num 366) 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:367: done (15 left) 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:378: write initial oid 378 2024-09-21T18:07:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:378: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-21T18:07:32.684 INFO:tasks.rados.rados.0.smithi160.stdout:378: writing smithi16030472-378 from 736239 to 1206409 tid 1 2024-09-21T18:07:32.694 INFO:tasks.rados.rados.0.smithi160.stdout:378: writing smithi16030472-378 from 1728182 to 2299964 tid 2 2024-09-21T18:07:32.694 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.694 INFO:tasks.rados.rados.0.smithi160.stdout:369: finishing write tid 1 to smithi16030472-369 2024-09-21T18:07:32.694 INFO:tasks.rados.rados.0.smithi160.stdout:369: finishing write tid 2 to smithi16030472-369 2024-09-21T18:07:32.694 INFO:tasks.rados.rados.0.smithi160.stdout:369: finishing write tid 3 to smithi16030472-369 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: finishing write tid 3 to smithi16030472-344 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: finishing write tid 4 to smithi16030472-344 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:358: finishing write tid 2 to smithi16030472-358 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: finishing write tid 5 to smithi16030472-344 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: oid 344 updating version 0 to 248 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: oid 344 updating version 248 to 249 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: oid 344 version 249 is already newer than 238 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: oid 344 updating version 249 to 250 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 344 v 250 (ObjNum 343 snap 0 seq_num 343) dirty exists 2024-09-21T18:07:32.715 INFO:tasks.rados.rados.0.smithi160.stdout:344: left oid 344 (ObjNum 343 snap 0 seq_num 343) 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:358: finishing write tid 3 to smithi16030472-358 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: finishing write tid 5 to smithi16030472-352 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: oid 352 updating version 0 to 248 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: oid 352 version 248 is already newer than 245 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: oid 352 version 248 is already newer than 246 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: oid 352 version 248 is already newer than 247 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 352 v 248 (ObjNum 351 snap 0 seq_num 351) dirty exists 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: left oid 352 (ObjNum 351 snap 0 seq_num 351) 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:363: finishing write tid 1 to smithi16030472-363 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:344: done (15 left) 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:352: done (14 left) 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:379: write initial oid 379 2024-09-21T18:07:32.716 INFO:tasks.rados.rados.0.smithi160.stdout:379: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-21T18:07:32.730 INFO:tasks.rados.rados.0.smithi160.stdout:379: writing smithi16030472-379 from 753046 to 1298465 tid 1 2024-09-21T18:07:32.740 INFO:tasks.rados.rados.0.smithi160.stdout:379: writing smithi16030472-379 from 1786664 to 2218457 tid 2 2024-09-21T18:07:32.740 INFO:tasks.rados.rados.0.smithi160.stdout:380: write initial oid 380 2024-09-21T18:07:32.740 INFO:tasks.rados.rados.0.smithi160.stdout:380: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-21T18:07:32.740 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:32.737+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:32.754 INFO:tasks.rados.rados.0.smithi160.stdout:380: writing smithi16030472-380 from 769853 to 1390521 tid 1 2024-09-21T18:07:32.762 INFO:tasks.rados.rados.0.smithi160.stdout:380: writing smithi16030472-380 from 1845146 to 2369853 tid 2 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: finishing write tid 2 to smithi16030472-361 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: finishing write tid 3 to smithi16030472-361 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: finishing write tid 4 to smithi16030472-361 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: oid 361 updating version 0 to 309 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: oid 361 version 309 is already newer than 308 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: oid 361 version 309 is already newer than 307 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 361 v 309 (ObjNum 360 snap 0 seq_num 360) dirty exists 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:361: left oid 361 (ObjNum 360 snap 0 seq_num 360) 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:369: finishing write tid 4 to smithi16030472-369 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:369: oid 369 updating version 0 to 250 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:369: oid 369 version 250 is already newer than 249 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:369: oid 369 updating version 250 to 251 2024-09-21T18:07:32.763 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 369 v 251 (ObjNum 368 snap 0 seq_num 368) dirty exists 2024-09-21T18:07:32.764 INFO:tasks.rados.rados.0.smithi160.stdout:369: left oid 369 (ObjNum 368 snap 0 seq_num 368) 2024-09-21T18:07:32.764 INFO:tasks.rados.rados.0.smithi160.stdout:361: done (15 left) 2024-09-21T18:07:32.764 INFO:tasks.rados.rados.0.smithi160.stdout:369: done (14 left) 2024-09-21T18:07:32.764 INFO:tasks.rados.rados.0.smithi160.stdout:381: write initial oid 381 2024-09-21T18:07:32.764 INFO:tasks.rados.rados.0.smithi160.stdout:381: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-21T18:07:32.772 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:32.772 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:32.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1664 3.0 3.c5ee34ac (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:32.673215+0000", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.098138800999999998, 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.098182683000000007, 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673215+0000", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:32.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673215+0000", 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673217+0000", 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.858e-06 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673524+0000", 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030686199999999998 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673542+0000", 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7569000000000001e-05 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673554+0000", 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2425999999999999e-05 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:32.775 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.83726293500000004, 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.83739710999999994, 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.778 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1663 3.0 3.c5ee34ac (undecoded) ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:32.672376+0000", 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.098977812999999998, 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.099194738000000005, 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:32.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.672376+0000", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.672376+0000", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.672381+0000", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7509999999999999e-06 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673127+0000", 2024-09-21T18:07:32.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000745599 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673159+0000", 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2351999999999998e-05 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.673177+0000", 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8272999999999999e-05 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:07:32.781 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 3 2024-09-21T18:07:32.782 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:32.782 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:32.782 INFO:tasks.rados.rados.0.smithi160.stdout:381: writing smithi16030472-381 from 786660 to 1482577 tid 1 2024-09-21T18:07:32.783 INFO:tasks.rados.rados.0.smithi160.stdout:381: writing smithi16030472-381 from 1903628 to 2339090 tid 2 2024-09-21T18:07:32.783 INFO:tasks.rados.rados.0.smithi160.stdout:382: write initial oid 382 2024-09-21T18:07:32.783 INFO:tasks.rados.rados.0.smithi160.stdout:382: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-21T18:07:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:32.788 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:32.789 INFO:tasks.rados.rados.0.smithi160.stdout:382: writing smithi16030472-382 from 403467 to 890986 tid 1 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:382: writing smithi16030472-382 from 1562110 to 2141230 tid 2 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: finishing write tid 4 to smithi16030472-371 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: oid 371 updating version 0 to 253 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: oid 371 updating version 253 to 255 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: oid 371 version 255 is already newer than 254 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 255 (ObjNum 370 snap 0 seq_num 370) dirty exists 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: left oid 371 (ObjNum 370 snap 0 seq_num 370) 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:371: done (15 left) 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:383: write initial oid 383 2024-09-21T18:07:32.799 INFO:tasks.rados.rados.0.smithi160.stdout:383: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-21T18:07:32.805 INFO:tasks.rados.rados.0.smithi160.stdout:383: writing smithi16030472-383 from 420274 to 983042 tid 1 2024-09-21T18:07:32.813 INFO:tasks.rados.rados.0.smithi160.stdout:383: writing smithi16030472-383 from 1620592 to 2059723 tid 2 2024-09-21T18:07:32.813 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.813 INFO:tasks.rados.rados.0.smithi160.stdout:374: finishing write tid 1 to smithi16030472-374 2024-09-21T18:07:32.813 INFO:tasks.rados.rados.0.smithi160.stdout:379: finishing write tid 1 to smithi16030472-379 2024-09-21T18:07:32.813 INFO:tasks.rados.rados.0.smithi160.stdout:379: finishing write tid 2 to smithi16030472-379 2024-09-21T18:07:32.841 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:32.836+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:32.876 INFO:tasks.rados.rados.0.smithi160.stdout:363: finishing write tid 2 to smithi16030472-363 2024-09-21T18:07:32.876 INFO:tasks.rados.rados.0.smithi160.stdout:363: finishing write tid 3 to smithi16030472-363 2024-09-21T18:07:32.876 INFO:tasks.rados.rados.0.smithi160.stdout:363: finishing write tid 4 to smithi16030472-363 2024-09-21T18:07:32.876 INFO:tasks.rados.rados.0.smithi160.stdout:363: oid 363 updating version 0 to 256 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:363: oid 363 updating version 256 to 257 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:363: oid 363 updating version 257 to 258 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 258 (ObjNum 362 snap 0 seq_num 362) dirty exists 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:363: left oid 363 (ObjNum 362 snap 0 seq_num 362) 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:363: done (15 left) 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:384: write initial oid 384 2024-09-21T18:07:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:384: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-21T18:07:32.883 INFO:tasks.rados.rados.0.smithi160.stdout:384: writing smithi16030472-384 from 437081 to 1075098 tid 1 2024-09-21T18:07:32.892 INFO:tasks.rados.rados.0.smithi160.stdout:384: writing smithi16030472-384 from 1679074 to 2261863 tid 2 2024-09-21T18:07:32.892 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.901 INFO:tasks.rados.rados.0.smithi160.stdout:362: finishing write tid 2 to smithi16030472-362 2024-09-21T18:07:32.901 INFO:tasks.rados.rados.0.smithi160.stdout:362: finishing write tid 3 to smithi16030472-362 2024-09-21T18:07:32.901 INFO:tasks.rados.rados.0.smithi160.stdout:362: finishing write tid 4 to smithi16030472-362 2024-09-21T18:07:32.901 INFO:tasks.rados.rados.0.smithi160.stdout:362: oid 362 updating version 0 to 309 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:362: oid 362 version 309 is already newer than 308 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:362: oid 362 updating version 309 to 310 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 362 v 310 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:362: left oid 362 (ObjNum 361 snap 0 seq_num 361) 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:362: done (15 left) 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:385: write initial oid 385 2024-09-21T18:07:32.902 INFO:tasks.rados.rados.0.smithi160.stdout:385: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-21T18:07:32.910 INFO:tasks.rados.rados.0.smithi160.stdout:385: writing smithi16030472-385 from 453888 to 1167154 tid 1 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:385: writing smithi16030472-385 from 1737556 to 2180356 tid 2 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:379: finishing write tid 3 to smithi16030472-379 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:379: finishing write tid 4 to smithi16030472-379 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:379: oid 379 updating version 0 to 254 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:379: oid 379 version 254 is already newer than 253 2024-09-21T18:07:32.918 INFO:tasks.rados.rados.0.smithi160.stdout:379: oid 379 version 254 is already newer than 252 2024-09-21T18:07:32.919 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 379 v 254 (ObjNum 378 snap 0 seq_num 378) dirty exists 2024-09-21T18:07:32.919 INFO:tasks.rados.rados.0.smithi160.stdout:379: left oid 379 (ObjNum 378 snap 0 seq_num 378) 2024-09-21T18:07:32.919 INFO:tasks.rados.rados.0.smithi160.stdout:379: done (15 left) 2024-09-21T18:07:32.919 INFO:tasks.rados.rados.0.smithi160.stdout:386: write initial oid 386 2024-09-21T18:07:32.919 INFO:tasks.rados.rados.0.smithi160.stdout:386: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-21T18:07:32.928 INFO:tasks.rados.rados.0.smithi160.stdout:386: writing smithi16030472-386 from 470695 to 1259210 tid 1 2024-09-21T18:07:32.936 INFO:tasks.rados.rados.0.smithi160.stdout:386: writing smithi16030472-386 from 1679685 to 2266143 tid 2 2024-09-21T18:07:32.936 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:32.942 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:32.936+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:32.960 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:32.960 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:32.960 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:32.960 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:32.960 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:32.975 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:33.002 INFO:tasks.rados.rados.0.smithi160.stdout:375: finishing write tid 1 to smithi16030472-375 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: finishing write tid 2 to smithi16030472-375 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: finishing write tid 3 to smithi16030472-375 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: finishing write tid 4 to smithi16030472-375 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: oid 375 updating version 0 to 312 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: oid 375 version 312 is already newer than 311 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: oid 375 updating version 312 to 313 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 375 v 313 (ObjNum 374 snap 0 seq_num 374) dirty exists 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: left oid 375 (ObjNum 374 snap 0 seq_num 374) 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:375: done (15 left) 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:387: write initial oid 387 2024-09-21T18:07:33.003 INFO:tasks.rados.rados.0.smithi160.stdout:387: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-21T18:07:33.010 INFO:tasks.rados.rados.0.smithi160.stdout:387: writing smithi16030472-387 from 487502 to 951266 tid 1 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:387: writing smithi16030472-387 from 1738167 to 2184636 tid 2 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: finishing write tid 2 to smithi16030472-374 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: finishing write tid 3 to smithi16030472-374 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: finishing write tid 4 to smithi16030472-374 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: oid 374 updating version 0 to 258 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: oid 374 version 258 is already newer than 256 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: oid 374 version 258 is already newer than 257 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 374 v 258 (ObjNum 373 snap 0 seq_num 373) dirty exists 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:374: left oid 374 (ObjNum 373 snap 0 seq_num 373) 2024-09-21T18:07:33.020 INFO:tasks.rados.rados.0.smithi160.stdout:377: finishing write tid 1 to smithi16030472-377 2024-09-21T18:07:33.021 INFO:tasks.rados.rados.0.smithi160.stdout:377: finishing write tid 2 to smithi16030472-377 2024-09-21T18:07:33.021 INFO:tasks.rados.rados.0.smithi160.stdout:374: done (15 left) 2024-09-21T18:07:33.021 INFO:tasks.rados.rados.0.smithi160.stdout:388: write initial oid 388 2024-09-21T18:07:33.021 INFO:tasks.rados.rados.0.smithi160.stdout:388: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-21T18:07:33.028 INFO:tasks.rados.rados.0.smithi160.stdout:388: writing smithi16030472-388 from 504309 to 1043322 tid 1 2024-09-21T18:07:33.040 INFO:tasks.rados.rados.0.smithi160.stdout:388: writing smithi16030472-388 from 1796649 to 2386776 tid 2 2024-09-21T18:07:33.040 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.040 INFO:tasks.rados.rados.0.smithi160.stdout:377: finishing write tid 3 to smithi16030472-377 2024-09-21T18:07:33.040 INFO:tasks.rados.rados.0.smithi160.stdout:377: finishing write tid 4 to smithi16030472-377 2024-09-21T18:07:33.040 INFO:tasks.rados.rados.0.smithi160.stdout:377: oid 377 updating version 0 to 261 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:377: oid 377 version 261 is already newer than 260 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:377: oid 377 version 261 is already newer than 259 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 377 v 261 (ObjNum 376 snap 0 seq_num 376) dirty exists 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:377: left oid 377 (ObjNum 376 snap 0 seq_num 376) 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:380: finishing write tid 1 to smithi16030472-380 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:380: finishing write tid 2 to smithi16030472-380 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:377: done (15 left) 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:389: write initial oid 389 2024-09-21T18:07:33.041 INFO:tasks.rados.rados.0.smithi160.stdout:389: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-21T18:07:33.042 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:33.037+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:33.051 INFO:tasks.rados.rados.0.smithi160.stdout:389: writing smithi16030472-389 from 521116 to 1251731 tid 1 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:389: writing smithi16030472-389 from 1971484 to 2521116 tid 2 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: finishing write tid 3 to smithi16030472-380 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: finishing write tid 4 to smithi16030472-380 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: oid 380 updating version 0 to 264 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: oid 380 version 264 is already newer than 263 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: oid 380 version 264 is already newer than 262 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 380 v 264 (ObjNum 379 snap 0 seq_num 379) dirty exists 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: left oid 380 (ObjNum 379 snap 0 seq_num 379) 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:384: finishing write tid 1 to smithi16030472-384 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:384: finishing write tid 2 to smithi16030472-384 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:382: finishing write tid 1 to smithi16030472-382 2024-09-21T18:07:33.062 INFO:tasks.rados.rados.0.smithi160.stdout:380: done (15 left) 2024-09-21T18:07:33.063 INFO:tasks.rados.rados.0.smithi160.stdout:390: write initial oid 390 2024-09-21T18:07:33.063 INFO:tasks.rados.rados.0.smithi160.stdout:390: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-21T18:07:33.069 INFO:tasks.rados.rados.0.smithi160.stdout:390: writing smithi16030472-390 from 537923 to 943787 tid 1 2024-09-21T18:07:33.077 INFO:tasks.rados.rados.0.smithi160.stdout:390: writing smithi16030472-390 from 1513613 to 2107409 tid 2 2024-09-21T18:07:33.077 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.091 INFO:tasks.rados.rados.0.smithi160.stdout:370: finishing write tid 1 to smithi16030472-370 2024-09-21T18:07:33.091 INFO:tasks.rados.rados.0.smithi160.stdout:370: finishing write tid 2 to smithi16030472-370 2024-09-21T18:07:33.091 INFO:tasks.rados.rados.0.smithi160.stdout:370: finishing write tid 3 to smithi16030472-370 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:370: finishing write tid 4 to smithi16030472-370 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:370: oid 370 updating version 0 to 257 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:370: oid 370 updating version 257 to 259 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:370: oid 370 version 259 is already newer than 258 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 370 v 259 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-21T18:07:33.092 INFO:tasks.rados.rados.0.smithi160.stdout:370: left oid 370 (ObjNum 369 snap 0 seq_num 369) 2024-09-21T18:07:33.093 INFO:tasks.rados.rados.0.smithi160.stdout:370: done (15 left) 2024-09-21T18:07:33.093 INFO:tasks.rados.rados.0.smithi160.stdout:391: write initial oid 391 2024-09-21T18:07:33.093 INFO:tasks.rados.rados.0.smithi160.stdout:391: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-21T18:07:33.099 INFO:tasks.rados.rados.0.smithi160.stdout:391: writing smithi16030472-391 from 554730 to 1035843 tid 1 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0629543180000001, 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.104 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:07:33.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:07:33.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:07:33.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0627146939999998, 2024-09-21T18:07:33.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:07:33.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:07:33.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:07:33.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0626792869999999, 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:07:33.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:33.119 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0626461140000001, 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:07:33.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0626165539999999, 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.124 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:07:33.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:07:33.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0625868980000002, 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.129 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:07:33.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:07:33.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0621006419999999, 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:07:33.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 2.0620670489999999, 2024-09-21T18:07:33.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.961849357, 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:07:33.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.9612504529999999, 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.960535836, 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.154 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.159 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.951283104, 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:07:33.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:07:33.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.951018403, 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.165 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:07:33.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:07:33.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.950923714, 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:07:33.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:07:33.174 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.950882424, 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:07:33.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.179 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.9396125209999999, 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:33.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:07:33.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:07:33.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:07:33.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.184 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.9304962880000001, 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.189 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.9294329210000001, 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.9293926109999999, 2024-09-21T18:07:33.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:07:33.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 1.166754769, 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:33.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:07:33.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:07:33.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.203 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.204 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:33.204 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:33.204 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:33.205 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:33.206 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:33.206 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:33.207 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:33.207 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:33.208 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:33.208 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:33.209 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:33.200+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:391: writing smithi16030472-391 from 1572095 to 2309549 tid 2 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: finishing write tid 2 to smithi16030472-382 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: finishing write tid 3 to smithi16030472-382 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: finishing write tid 4 to smithi16030472-382 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: oid 382 updating version 0 to 267 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: oid 382 version 267 is already newer than 265 2024-09-21T18:07:33.209 INFO:tasks.rados.rados.0.smithi160.stdout:382: oid 382 version 267 is already newer than 266 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 382 v 267 (ObjNum 381 snap 0 seq_num 381) dirty exists 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:382: left oid 382 (ObjNum 381 snap 0 seq_num 381) 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:382: done (15 left) 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:392: write initial oid 392 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:392: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:392: writing smithi16030472-392 from 571537 to 1127899 tid 1 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:392: writing smithi16030472-392 from 1630577 to 2228042 tid 2 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:392: finishing write tid 1 to smithi16030472-392 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:383: finishing write tid 1 to smithi16030472-383 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:383: finishing write tid 2 to smithi16030472-383 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:383: finishing write tid 3 to smithi16030472-383 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:383: finishing write tid 4 to smithi16030472-383 2024-09-21T18:07:33.210 INFO:tasks.rados.rados.0.smithi160.stdout:383: oid 383 updating version 0 to 314 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:383: oid 383 updating version 314 to 316 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:383: oid 383 version 316 is already newer than 315 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 316 (ObjNum 382 snap 0 seq_num 382) dirty exists 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:383: left oid 383 (ObjNum 382 snap 0 seq_num 382) 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:383: done (15 left) 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:393: write initial oid 393 2024-09-21T18:07:33.211 INFO:tasks.rados.rados.0.smithi160.stdout:393: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-21T18:07:33.218 INFO:tasks.rados.rados.0.smithi160.stdout:393: writing smithi16030472-393 from 588344 to 1219955 tid 1 2024-09-21T18:07:33.224 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:33.228 INFO:tasks.rados.rados.0.smithi160.stdout:393: writing smithi16030472-393 from 1689059 to 2430182 tid 2 2024-09-21T18:07:33.228 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: finishing write tid 4 to smithi16030472-358 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: oid 358 updating version 0 to 255 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: oid 358 version 255 is already newer than 254 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: oid 358 version 255 is already newer than 253 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 358 v 255 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: left oid 358 (ObjNum 357 snap 0 seq_num 357) 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:372: finishing write tid 1 to smithi16030472-372 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:372: finishing write tid 2 to smithi16030472-372 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:372: finishing write tid 3 to smithi16030472-372 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:358: done (15 left) 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:394: write initial oid 394 2024-09-21T18:07:33.229 INFO:tasks.rados.rados.0.smithi160.stdout:394: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-21T18:07:33.238 INFO:tasks.rados.rados.0.smithi160.stdout:394: writing smithi16030472-394 from 605151 to 1312011 tid 1 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:394: writing smithi16030472-394 from 2031188 to 2605151 tid 2 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:372: finishing write tid 4 to smithi16030472-372 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:372: oid 372 updating version 0 to 262 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:372: oid 372 version 262 is already newer than 261 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:372: oid 372 version 262 is already newer than 260 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 262 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-09-21T18:07:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:372: left oid 372 (ObjNum 371 snap 0 seq_num 371) 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: finishing write tid 1 to smithi16030472-373 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: finishing write tid 2 to smithi16030472-373 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: finishing write tid 3 to smithi16030472-373 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: finishing write tid 4 to smithi16030472-373 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: oid 373 updating version 0 to 265 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: oid 373 version 265 is already newer than 264 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: oid 373 version 265 is already newer than 263 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 373 v 265 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: left oid 373 (ObjNum 372 snap 0 seq_num 372) 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:376: finishing write tid 1 to smithi16030472-376 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:376: finishing write tid 2 to smithi16030472-376 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:376: finishing write tid 3 to smithi16030472-376 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:372: done (15 left) 2024-09-21T18:07:33.250 INFO:tasks.rados.rados.0.smithi160.stdout:373: done (14 left) 2024-09-21T18:07:33.251 INFO:tasks.rados.rados.0.smithi160.stdout:395: write initial oid 395 2024-09-21T18:07:33.251 INFO:tasks.rados.rados.0.smithi160.stdout:395: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-21T18:07:33.253 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:33.260 INFO:tasks.rados.rados.0.smithi160.stdout:395: writing smithi16030472-395 from 621958 to 1404067 tid 1 2024-09-21T18:07:33.270 INFO:tasks.rados.rados.0.smithi160.stdout:395: writing smithi16030472-395 from 2089670 to 2621958 tid 2 2024-09-21T18:07:33.270 INFO:tasks.rados.rados.0.smithi160.stdout:396: write initial oid 396 2024-09-21T18:07:33.270 INFO:tasks.rados.rados.0.smithi160.stdout:396: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-21T18:07:33.277 INFO:tasks.rados.rados.0.smithi160.stdout:396: writing smithi16030472-396 from 638765 to 1096123 tid 1 2024-09-21T18:07:33.283 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:396: writing smithi16030472-396 from 1748152 to 2352955 tid 2 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: finishing write tid 4 to smithi16030472-376 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: oid 376 updating version 0 to 267 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: oid 376 version 267 is already newer than 266 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: oid 376 updating version 267 to 268 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 376 v 268 (ObjNum 375 snap 0 seq_num 375) dirty exists 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: left oid 376 (ObjNum 375 snap 0 seq_num 375) 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:376: done (15 left) 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:397: write initial oid 397 2024-09-21T18:07:33.287 INFO:tasks.rados.rados.0.smithi160.stdout:397: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-21T18:07:33.293 INFO:tasks.rados.rados.0.smithi160.stdout:397: writing smithi16030472-397 from 655572 to 1304532 tid 1 2024-09-21T18:07:33.301 INFO:tasks.rados.rados.0.smithi160.stdout:397: writing smithi16030472-397 from 1922987 to 2655572 tid 2 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:378: finishing write tid 1 to smithi16030472-378 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:378: finishing write tid 2 to smithi16030472-378 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:378: finishing write tid 3 to smithi16030472-378 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:389: finishing write tid 1 to smithi16030472-389 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:389: finishing write tid 2 to smithi16030472-389 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:389: finishing write tid 3 to smithi16030472-389 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: finishing write tid 2 to smithi16030472-392 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: finishing write tid 3 to smithi16030472-392 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: finishing write tid 4 to smithi16030472-392 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: oid 392 updating version 0 to 258 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: oid 392 version 258 is already newer than 257 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:392: oid 392 updating version 258 to 259 2024-09-21T18:07:33.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 392 v 259 (ObjNum 391 snap 0 seq_num 391) dirty exists 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:392: left oid 392 (ObjNum 391 snap 0 seq_num 391) 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: finishing write tid 4 to smithi16030472-378 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: oid 378 updating version 0 to 271 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: oid 378 version 271 is already newer than 270 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: oid 378 version 271 is already newer than 269 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 378 v 271 (ObjNum 377 snap 0 seq_num 377) dirty exists 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: left oid 378 (ObjNum 377 snap 0 seq_num 377) 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:381: finishing write tid 1 to smithi16030472-381 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:378: done (15 left) 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:392: done (14 left) 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:398: write initial oid 398 2024-09-21T18:07:33.303 INFO:tasks.rados.rados.0.smithi160.stdout:398: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-21T18:07:33.306 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:33.301+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:33.310 INFO:tasks.rados.rados.0.smithi160.stdout:398: writing smithi16030472-398 from 672379 to 1396588 tid 1 2024-09-21T18:07:33.312 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:33.316 INFO:tasks.rados.rados.0.smithi160.stdout:398: writing smithi16030472-398 from 1865116 to 2473588 tid 2 2024-09-21T18:07:33.316 INFO:tasks.rados.rados.0.smithi160.stdout:399: write initial oid 399 2024-09-21T18:07:33.316 INFO:tasks.rados.rados.0.smithi160.stdout:381: finishing write tid 2 to smithi16030472-381 2024-09-21T18:07:33.316 INFO:tasks.rados.rados.0.smithi160.stdout:399: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-21T18:07:33.324 INFO:tasks.rados.rados.0.smithi160.stdout:399: writing smithi16030472-399 from 689186 to 1488644 tid 1 2024-09-21T18:07:33.331 INFO:tasks.rados.rados.0.smithi160.stdout:399: writing smithi16030472-399 from 1923598 to 2675728 tid 2 2024-09-21T18:07:33.331 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.331 INFO:tasks.rados.rados.0.smithi160.stdout:381: finishing write tid 3 to smithi16030472-381 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: finishing write tid 4 to smithi16030472-381 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: oid 381 updating version 0 to 272 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: oid 381 updating version 272 to 274 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: oid 381 version 274 is already newer than 273 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 274 (ObjNum 380 snap 0 seq_num 380) dirty exists 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: left oid 381 (ObjNum 380 snap 0 seq_num 380) 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:381: done (15 left) 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:400: write initial oid 400 2024-09-21T18:07:33.332 INFO:tasks.rados.rados.0.smithi160.stdout:400: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-21T18:07:33.337 INFO:tasks.rados.rados.0.smithi160.stdout:400: writing smithi16030472-400 from 705993 to 1180700 tid 1 2024-09-21T18:07:33.338 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:33.343 INFO:tasks.rados.rados.0.smithi160.stdout:400: writing smithi16030472-400 from 1582080 to 2194221 tid 2 2024-09-21T18:07:33.343 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:07:33.351 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:33.348+0000 7ff40b999640 -1 osd.4 51 handle_osd_map injecting map crc failure 2024-09-21T18:07:33.387 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:33.387 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:33.401 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: finishing write tid 4 to smithi16030472-389 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: oid 389 updating version 0 to 268 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: oid 389 updating version 268 to 269 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: oid 389 updating version 269 to 270 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 389 v 270 (ObjNum 388 snap 0 seq_num 388) dirty exists 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: left oid 389 (ObjNum 388 snap 0 seq_num 388) 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:389: done (15 left) 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:401: write initial oid 401 2024-09-21T18:07:33.405 INFO:tasks.rados.rados.0.smithi160.stdout:401: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-21T18:07:33.412 INFO:tasks.rados.rados.0.smithi160.stdout:401: writing smithi16030472-401 from 722800 to 1272756 tid 1 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:401: writing smithi16030472-401 from 2040562 to 2722800 tid 2 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:387: finishing write tid 1 to smithi16030472-387 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:397: finishing write tid 1 to smithi16030472-397 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:397: finishing write tid 2 to smithi16030472-397 2024-09-21T18:07:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:397: finishing write tid 3 to smithi16030472-397 2024-09-21T18:07:33.428 INFO:tasks.rados.rados.0.smithi160.stdout:384: finishing write tid 3 to smithi16030472-384 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: finishing write tid 4 to smithi16030472-384 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: oid 384 updating version 0 to 255 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: oid 384 updating version 255 to 257 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: oid 384 version 257 is already newer than 256 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 384 v 257 (ObjNum 383 snap 0 seq_num 383) dirty exists 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: left oid 384 (ObjNum 383 snap 0 seq_num 383) 2024-09-21T18:07:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:384: done (15 left) 2024-09-21T18:07:33.430 INFO:tasks.rados.rados.0.smithi160.stdout:402: write initial oid 402 2024-09-21T18:07:33.430 INFO:tasks.rados.rados.0.smithi160.stdout:402: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-21T18:07:33.436 INFO:tasks.rados.rados.0.smithi160.stdout:402: writing smithi16030472-402 from 739607 to 1364812 tid 1 2024-09-21T18:07:33.442 INFO:tasks.rados.rados.0.smithi160.stdout:402: writing smithi16030472-402 from 1982691 to 2482148 tid 2 2024-09-21T18:07:33.443 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.443 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:33.441+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:33.507 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:33.502+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:33.516 INFO:tasks.rados.rados.0.smithi160.stdout:385: finishing write tid 1 to smithi16030472-385 2024-09-21T18:07:33.538 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:33.572 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:33.575 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:33.575 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:33.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:33.605 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:33.601+0000 7fd79a4fd640 -1 osd.1 51 handle_osd_map injecting map crc failure 2024-09-21T18:07:33.608 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:33.608 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:33.602+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:33.623 INFO:tasks.rados.rados.0.smithi160.stdout:387: finishing write tid 2 to smithi16030472-387 2024-09-21T18:07:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:387: finishing write tid 3 to smithi16030472-387 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: finishing write tid 4 to smithi16030472-387 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: oid 387 updating version 0 to 275 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: oid 387 updating version 275 to 276 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: oid 387 updating version 276 to 277 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 387 v 277 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: left oid 387 (ObjNum 386 snap 0 seq_num 386) 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:387: done (15 left) 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:403: write initial oid 403 2024-09-21T18:07:33.626 INFO:tasks.rados.rados.0.smithi160.stdout:403: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-21T18:07:33.634 INFO:tasks.rados.rados.0.smithi160.stdout:403: writing smithi16030472-403 from 756414 to 1456868 tid 1 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:403: writing smithi16030472-403 from 2041173 to 2756414 tid 2 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: finishing write tid 2 to smithi16030472-385 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: finishing write tid 3 to smithi16030472-385 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:388: finishing write tid 1 to smithi16030472-388 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: finishing write tid 4 to smithi16030472-385 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: oid 385 updating version 0 to 319 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: oid 385 version 319 is already newer than 317 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: oid 385 version 319 is already newer than 318 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 319 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:385: left oid 385 (ObjNum 384 snap 0 seq_num 384) 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:386: finishing write tid 1 to smithi16030472-386 2024-09-21T18:07:33.643 INFO:tasks.rados.rados.0.smithi160.stdout:386: finishing write tid 2 to smithi16030472-386 2024-09-21T18:07:33.644 INFO:tasks.rados.rados.0.smithi160.stdout:386: finishing write tid 3 to smithi16030472-386 2024-09-21T18:07:33.644 INFO:tasks.rados.rados.0.smithi160.stdout:385: done (15 left) 2024-09-21T18:07:33.644 INFO:tasks.rados.rados.0.smithi160.stdout:404: write initial oid 404 2024-09-21T18:07:33.644 INFO:tasks.rados.rados.0.smithi160.stdout:404: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-21T18:07:33.644 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:33.644 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:33.649 INFO:tasks.rados.rados.0.smithi160.stdout:404: writing smithi16030472-404 from 773221 to 1265277 tid 1 2024-09-21T18:07:33.656 INFO:tasks.rados.rados.0.smithi160.stdout:404: writing smithi16030472-404 from 1816008 to 2319134 tid 2 2024-09-21T18:07:33.656 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: finishing write tid 4 to smithi16030472-386 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: oid 386 updating version 0 to 321 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: oid 386 updating version 321 to 322 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: oid 386 version 322 is already newer than 320 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 386 v 322 (ObjNum 385 snap 0 seq_num 385) dirty exists 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: left oid 386 (ObjNum 385 snap 0 seq_num 385) 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:395: finishing write tid 1 to smithi16030472-395 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:386: done (15 left) 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:405: write initial oid 405 2024-09-21T18:07:33.657 INFO:tasks.rados.rados.0.smithi160.stdout:405: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-21T18:07:33.667 INFO:tasks.rados.rados.0.smithi160.stdout:405: writing smithi16030472-405 from 790028 to 1357333 tid 1 2024-09-21T18:07:33.671 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:33.671 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:405: writing smithi16030472-405 from 1874490 to 2637627 tid 2 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:397: finishing write tid 4 to smithi16030472-397 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:397: oid 397 updating version 0 to 260 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:397: oid 397 updating version 260 to 261 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:397: oid 397 updating version 261 to 262 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 397 v 262 (ObjNum 396 snap 0 seq_num 396) dirty exists 2024-09-21T18:07:33.678 INFO:tasks.rados.rados.0.smithi160.stdout:397: left oid 397 (ObjNum 396 snap 0 seq_num 396) 2024-09-21T18:07:33.679 INFO:tasks.rados.rados.0.smithi160.stdout:397: done (15 left) 2024-09-21T18:07:33.679 INFO:tasks.rados.rados.0.smithi160.stdout:406: write initial oid 406 2024-09-21T18:07:33.679 INFO:tasks.rados.rados.0.smithi160.stdout:406: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-21T18:07:33.685 INFO:tasks.rados.rados.0.smithi160.stdout:406: writing smithi16030472-406 from 406835 to 1049389 tid 1 2024-09-21T18:07:33.692 INFO:tasks.rados.rados.0.smithi160.stdout:406: writing smithi16030472-406 from 1532972 to 2039767 tid 2 2024-09-21T18:07:33.697 INFO:tasks.rados.rados.0.smithi160.stdout:406: writing smithi16030472-406 from 2772312 to 2806835 tid 3 2024-09-21T18:07:33.697 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.700 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:33.709 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:33.702+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:33.720 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:33.720 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:33.733 INFO:tasks.rados.rados.0.smithi160.stdout:388: finishing write tid 2 to smithi16030472-388 2024-09-21T18:07:33.735 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:33.810 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:33.804+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:33.811 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:33.811 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:33.828 INFO:tasks.rados.rados.0.smithi160.stdout:388: finishing write tid 3 to smithi16030472-388 2024-09-21T18:07:33.828 INFO:tasks.rados.rados.0.smithi160.stdout:388: finishing write tid 4 to smithi16030472-388 2024-09-21T18:07:33.828 INFO:tasks.rados.rados.0.smithi160.stdout:388: oid 388 updating version 0 to 278 2024-09-21T18:07:33.828 INFO:tasks.rados.rados.0.smithi160.stdout:388: oid 388 updating version 278 to 280 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:388: oid 388 version 280 is already newer than 279 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 388 v 280 (ObjNum 387 snap 0 seq_num 387) dirty exists 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:388: left oid 388 (ObjNum 387 snap 0 seq_num 387) 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:388: done (15 left) 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:407: write initial oid 407 2024-09-21T18:07:33.829 INFO:tasks.rados.rados.0.smithi160.stdout:407: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-21T18:07:33.835 INFO:tasks.rados.rados.0.smithi160.stdout:407: writing smithi16030472-407 from 423642 to 1141445 tid 1 2024-09-21T18:07:33.846 INFO:tasks.rados.rados.0.smithi160.stdout:407: writing smithi16030472-407 from 1875101 to 2641907 tid 2 2024-09-21T18:07:33.846 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.846 INFO:tasks.rados.rados.0.smithi160.stdout:391: finishing write tid 1 to smithi16030472-391 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: finishing write tid 2 to smithi16030472-391 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: finishing write tid 3 to smithi16030472-391 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: finishing write tid 4 to smithi16030472-391 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: oid 391 updating version 0 to 281 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: oid 391 updating version 281 to 283 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: oid 391 version 283 is already newer than 282 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 391 v 283 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:391: left oid 391 (ObjNum 390 snap 0 seq_num 390) 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:393: finishing write tid 1 to smithi16030472-393 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:393: finishing write tid 2 to smithi16030472-393 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:393: finishing write tid 3 to smithi16030472-393 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:393: finishing write tid 4 to smithi16030472-393 2024-09-21T18:07:33.847 INFO:tasks.rados.rados.0.smithi160.stdout:393: oid 393 updating version 0 to 285 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:393: oid 393 version 285 is already newer than 284 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:393: oid 393 updating version 285 to 286 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 286 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:393: left oid 393 (ObjNum 392 snap 0 seq_num 392) 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:390: finishing write tid 1 to smithi16030472-390 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:391: done (15 left) 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:393: done (14 left) 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:408: write initial oid 408 2024-09-21T18:07:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:408: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-21T18:07:33.853 INFO:tasks.rados.rados.0.smithi160.stdout:408: writing smithi16030472-408 from 440449 to 1233501 tid 1 2024-09-21T18:07:33.862 INFO:tasks.rados.rados.0.smithi160.stdout:408: writing smithi16030472-408 from 1933583 to 2444047 tid 2 2024-09-21T18:07:33.862 INFO:tasks.rados.rados.0.smithi160.stdout:409: write initial oid 409 2024-09-21T18:07:33.862 INFO:tasks.rados.rados.0.smithi160.stdout:409: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-21T18:07:33.867 INFO:tasks.rados.rados.0.smithi160.stdout:409: writing smithi16030472-409 from 457256 to 925557 tid 1 2024-09-21T18:07:33.877 INFO:tasks.rados.rados.0.smithi160.stdout:409: writing smithi16030472-409 from 1592065 to 2362540 tid 2 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:409: writing smithi16030472-409 from 2854581 to 2857256 tid 3 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: finishing write tid 2 to smithi16030472-390 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: finishing write tid 3 to smithi16030472-390 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: finishing write tid 4 to smithi16030472-390 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: oid 390 updating version 0 to 269 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: oid 390 updating version 269 to 271 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: oid 390 version 271 is already newer than 270 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 390 v 271 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:390: left oid 390 (ObjNum 389 snap 0 seq_num 389) 2024-09-21T18:07:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:396: finishing write tid 1 to smithi16030472-396 2024-09-21T18:07:33.882 INFO:tasks.rados.rados.0.smithi160.stdout:396: finishing write tid 2 to smithi16030472-396 2024-09-21T18:07:33.882 INFO:tasks.rados.rados.0.smithi160.stdout:390: done (15 left) 2024-09-21T18:07:33.882 INFO:tasks.rados.rados.0.smithi160.stdout:410: write initial oid 410 2024-09-21T18:07:33.882 INFO:tasks.rados.rados.0.smithi160.stdout:410: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-21T18:07:33.887 INFO:tasks.rados.rados.0.smithi160.stdout:410: writing smithi16030472-410 from 474063 to 1017613 tid 1 2024-09-21T18:07:33.895 INFO:tasks.rados.rados.0.smithi160.stdout:410: writing smithi16030472-410 from 1650547 to 2164680 tid 2 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout:410: writing smithi16030472-410 from 2765651 to 2874063 tid 3 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout:396: finishing write tid 3 to smithi16030472-396 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout:396: finishing write tid 4 to smithi16030472-396 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout:396: oid 396 updating version 0 to 274 2024-09-21T18:07:33.900 INFO:tasks.rados.rados.0.smithi160.stdout:396: oid 396 version 274 is already newer than 272 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:396: oid 396 version 274 is already newer than 273 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 396 v 274 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:396: left oid 396 (ObjNum 395 snap 0 seq_num 395) 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:398: finishing write tid 1 to smithi16030472-398 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:396: done (15 left) 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:411: write initial oid 411 2024-09-21T18:07:33.901 INFO:tasks.rados.rados.0.smithi160.stdout:411: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2024-09-21T18:07:33.906 INFO:tasks.rados.rados.0.smithi160.stdout:411: writing smithi16030472-411 from 490870 to 1109669 tid 1 2024-09-21T18:07:33.907 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:33.908 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:33.910 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:33.905+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:33.914 INFO:tasks.rados.rados.0.smithi160.stdout:411: writing smithi16030472-411 from 1592676 to 2366820 tid 2 2024-09-21T18:07:33.917 INFO:tasks.rados.rados.0.smithi160.stdout:411: writing smithi16030472-411 from 2793074 to 2890870 tid 3 2024-09-21T18:07:33.917 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.922 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: finishing write tid 1 to smithi16030472-394 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: finishing write tid 2 to smithi16030472-394 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: finishing write tid 3 to smithi16030472-394 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: finishing write tid 4 to smithi16030472-394 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: oid 394 updating version 0 to 287 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: oid 394 updating version 287 to 288 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: oid 394 updating version 288 to 289 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 394 v 289 (ObjNum 393 snap 0 seq_num 393) dirty exists 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: left oid 394 (ObjNum 393 snap 0 seq_num 393) 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:394: done (15 left) 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:412: write initial oid 412 2024-09-21T18:07:33.931 INFO:tasks.rados.rados.0.smithi160.stdout:412: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-21T18:07:33.935 INFO:tasks.rados.rados.0.smithi160.stdout:412: writing smithi16030472-412 from 507677 to 918078 tid 1 2024-09-21T18:07:33.941 INFO:tasks.rados.rados.0.smithi160.stdout:412: writing smithi16030472-412 from 1367511 to 1885313 tid 2 2024-09-21T18:07:33.947 INFO:tasks.rados.rados.0.smithi160.stdout:412: writing smithi16030472-412 from 2420497 to 2907677 tid 3 2024-09-21T18:07:33.947 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: finishing write tid 2 to smithi16030472-395 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: finishing write tid 3 to smithi16030472-395 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: finishing write tid 4 to smithi16030472-395 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: oid 395 updating version 0 to 325 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: oid 395 version 325 is already newer than 324 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: oid 395 version 325 is already newer than 323 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 395 v 325 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: left oid 395 (ObjNum 394 snap 0 seq_num 394) 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:399: finishing write tid 1 to smithi16030472-399 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:395: done (15 left) 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:413: write initial oid 413 2024-09-21T18:07:33.948 INFO:tasks.rados.rados.0.smithi160.stdout:413: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-21T18:07:33.952 INFO:tasks.rados.rados.0.smithi160.stdout:413: writing smithi16030472-413 from 524484 to 1010134 tid 1 2024-09-21T18:07:33.958 INFO:tasks.rados.rados.0.smithi160.stdout:413: writing smithi16030472-413 from 1425993 to 2087453 tid 2 2024-09-21T18:07:33.963 INFO:tasks.rados.rados.0.smithi160.stdout:413: writing smithi16030472-413 from 2847920 to 2924484 tid 3 2024-09-21T18:07:33.964 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:33.964 INFO:tasks.rados.rados.0.smithi160.stdout:399: finishing write tid 2 to smithi16030472-399 2024-09-21T18:07:34.003 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:34.000+0000 7fb924fb6640 -1 osd.8 51 handle_osd_map injecting map crc failure 2024-09-21T18:07:34.010 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:34.008+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:34.031 INFO:tasks.rados.rados.0.smithi160.stdout:400: finishing write tid 1 to smithi16030472-400 2024-09-21T18:07:34.032 INFO:tasks.rados.rados.0.smithi160.stdout:400: finishing write tid 2 to smithi16030472-400 2024-09-21T18:07:34.089 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.089 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.104 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:34.111 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:34.108+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:34.136 INFO:tasks.rados.rados.0.smithi160.stdout:400: finishing write tid 3 to smithi16030472-400 2024-09-21T18:07:34.136 INFO:tasks.rados.rados.0.smithi160.stdout:400: finishing write tid 4 to smithi16030472-400 2024-09-21T18:07:34.136 INFO:tasks.rados.rados.0.smithi160.stdout:400: oid 400 updating version 0 to 290 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:400: oid 400 updating version 290 to 291 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:400: oid 400 updating version 291 to 292 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 400 v 292 (ObjNum 399 snap 0 seq_num 399) dirty exists 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:400: left oid 400 (ObjNum 399 snap 0 seq_num 399) 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:400: done (15 left) 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:414: write initial oid 414 2024-09-21T18:07:34.137 INFO:tasks.rados.rados.0.smithi160.stdout:414: seq_num 413 ranges {541291=560899,1884475=521471} 2024-09-21T18:07:34.142 INFO:tasks.rados.rados.0.smithi160.stdout:414: writing smithi16030472-414 from 541291 to 1102190 tid 1 2024-09-21T18:07:34.150 INFO:tasks.rados.rados.0.smithi160.stdout:414: writing smithi16030472-414 from 1884475 to 2405946 tid 2 2024-09-21T18:07:34.150 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:34.212 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:34.208+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:34.228 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.228 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.244 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:34.249 INFO:tasks.rados.rados.0.smithi160.stdout:398: finishing write tid 2 to smithi16030472-398 2024-09-21T18:07:34.249 INFO:tasks.rados.rados.0.smithi160.stdout:399: finishing write tid 3 to smithi16030472-399 2024-09-21T18:07:34.249 INFO:tasks.rados.rados.0.smithi160.stdout:399: finishing write tid 4 to smithi16030472-399 2024-09-21T18:07:34.249 INFO:tasks.rados.rados.0.smithi160.stdout:399: oid 399 updating version 0 to 327 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:399: oid 399 version 327 is already newer than 326 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:399: oid 399 updating version 327 to 328 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 399 v 328 (ObjNum 398 snap 0 seq_num 398) dirty exists 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:399: left oid 399 (ObjNum 398 snap 0 seq_num 398) 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:403: finishing write tid 1 to smithi16030472-403 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:403: finishing write tid 2 to smithi16030472-403 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: finishing write tid 3 to smithi16030472-398 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: finishing write tid 4 to smithi16030472-398 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: oid 398 updating version 0 to 276 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: oid 398 updating version 276 to 277 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: oid 398 version 277 is already newer than 275 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 398 v 277 (ObjNum 397 snap 0 seq_num 397) dirty exists 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:398: left oid 398 (ObjNum 397 snap 0 seq_num 397) 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:408: finishing write tid 1 to smithi16030472-408 2024-09-21T18:07:34.250 INFO:tasks.rados.rados.0.smithi160.stdout:408: finishing write tid 2 to smithi16030472-408 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: finishing write tid 3 to smithi16030472-408 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: finishing write tid 4 to smithi16030472-408 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: oid 408 updating version 0 to 280 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: oid 408 version 280 is already newer than 278 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: oid 408 version 280 is already newer than 279 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 280 (ObjNum 407 snap 0 seq_num 407) dirty exists 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: left oid 408 (ObjNum 407 snap 0 seq_num 407) 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:398: done (15 left) 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:399: done (14 left) 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:408: done (13 left) 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:415: write initial oid 415 2024-09-21T18:07:34.251 INFO:tasks.rados.rados.0.smithi160.stdout:415: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-21T18:07:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:415: writing smithi16030472-415 from 558098 to 1194246 tid 1 2024-09-21T18:07:34.270 INFO:tasks.rados.rados.0.smithi160.stdout:415: writing smithi16030472-415 from 1826604 to 2491733 tid 2 2024-09-21T18:07:34.271 INFO:tasks.rados.rados.0.smithi160.stdout:416: write initial oid 416 2024-09-21T18:07:34.271 INFO:tasks.rados.rados.0.smithi160.stdout:416: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-21T18:07:34.281 INFO:tasks.rados.rados.0.smithi160.stdout:416: writing smithi16030472-416 from 574905 to 1286302 tid 1 2024-09-21T18:07:34.291 INFO:tasks.rados.rados.0.smithi160.stdout:416: writing smithi16030472-416 from 1885086 to 2410226 tid 2 2024-09-21T18:07:34.296 INFO:tasks.rados.rados.0.smithi160.stdout:416: writing smithi16030472-416 from 2930189 to 2974905 tid 3 2024-09-21T18:07:34.296 INFO:tasks.rados.rados.0.smithi160.stdout:417: write initial oid 417 2024-09-21T18:07:34.296 INFO:tasks.rados.rados.0.smithi160.stdout:417: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-21T18:07:34.307 INFO:tasks.rados.rados.0.smithi160.stdout:417: writing smithi16030472-417 from 591712 to 1378358 tid 1 2024-09-21T18:07:34.313 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:34.309+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:34.316 INFO:tasks.rados.rados.0.smithi160.stdout:417: writing smithi16030472-417 from 1943568 to 2612366 tid 2 2024-09-21T18:07:34.316 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:34.316 INFO:tasks.rados.rados.0.smithi160.stdout:402: finishing write tid 1 to smithi16030472-402 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: finishing write tid 1 to smithi16030472-401 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: finishing write tid 2 to smithi16030472-401 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: finishing write tid 3 to smithi16030472-401 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: finishing write tid 4 to smithi16030472-401 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: oid 401 updating version 0 to 294 2024-09-21T18:07:34.345 INFO:tasks.rados.rados.0.smithi160.stdout:401: oid 401 version 294 is already newer than 293 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:401: oid 401 updating version 294 to 295 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 401 v 295 (ObjNum 400 snap 0 seq_num 400) dirty exists 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:401: left oid 401 (ObjNum 400 snap 0 seq_num 400) 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:414: finishing write tid 1 to smithi16030472-414 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:401: done (15 left) 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:418: write initial oid 418 2024-09-21T18:07:34.346 INFO:tasks.rados.rados.0.smithi160.stdout:418: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-21T18:07:34.353 INFO:tasks.rados.rados.0.smithi160.stdout:418: writing smithi16030472-418 from 608519 to 1070414 tid 1 2024-09-21T18:07:34.359 INFO:tasks.rados.rados.0.smithi160.stdout:418: writing smithi16030472-418 from 1602050 to 2130859 tid 2 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout:418: writing smithi16030472-418 from 2585035 to 3008519 tid 3 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout:414: finishing write tid 2 to smithi16030472-414 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout:414: finishing write tid 3 to smithi16030472-414 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout:414: finishing write tid 4 to smithi16030472-414 2024-09-21T18:07:34.366 INFO:tasks.rados.rados.0.smithi160.stdout:414: oid 414 updating version 0 to 297 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:414: oid 414 updating version 297 to 298 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:414: oid 414 version 298 is already newer than 296 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 298 (ObjNum 413 snap 0 seq_num 413) dirty exists 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:414: left oid 414 (ObjNum 413 snap 0 seq_num 413) 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:414: done (15 left) 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:419: write initial oid 419 2024-09-21T18:07:34.367 INFO:tasks.rados.rados.0.smithi160.stdout:419: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-21T18:07:34.367 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.367 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.374 INFO:tasks.rados.rados.0.smithi160.stdout:419: writing smithi16030472-419 from 625326 to 1162470 tid 1 2024-09-21T18:07:34.382 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:34.384 INFO:tasks.rados.rados.0.smithi160.stdout:419: writing smithi16030472-419 from 1944179 to 2616646 tid 2 2024-09-21T18:07:34.384 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:34.384 INFO:tasks.rados.rados.0.smithi160.stdout:418: finishing write tid 1 to smithi16030472-418 2024-09-21T18:07:34.423 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:34.421+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:34.515 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:34.510+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:34.543 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.544 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:34.569 INFO:tasks.rados.rados.0.smithi160.stdout:418: finishing write tid 2 to smithi16030472-418 2024-09-21T18:07:34.616 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:34.610+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:34.687 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.687 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.701 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:34.717 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:34.714+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:34.762 INFO:tasks.rados.rados.0.smithi160.stdout:418: finishing write tid 3 to smithi16030472-418 2024-09-21T18:07:34.818 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:34.814+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:34.832 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:34.832 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:34.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:34.862 INFO:tasks.rados.rados.0.smithi160.stdout:418: finishing write tid 4 to smithi16030472-418 2024-09-21T18:07:34.933 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:34.930+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:34.963 INFO:tasks.rados.rados.0.smithi160.stdout:418: finishing write tid 5 to smithi16030472-418 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: oid 418 updating version 0 to 301 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: oid 418 version 301 is already newer than 300 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: oid 418 updating version 301 to 302 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: oid 418 version 302 is already newer than 299 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 418 v 302 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: left oid 418 (ObjNum 417 snap 0 seq_num 417) 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:418: done (15 left) 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:420: write initial oid 420 2024-09-21T18:07:34.964 INFO:tasks.rados.rados.0.smithi160.stdout:420: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-21T18:07:34.974 INFO:tasks.rados.rados.0.smithi160.stdout:420: writing smithi16030472-420 from 642133 to 1370879 tid 1 2024-09-21T18:07:34.983 INFO:tasks.rados.rados.0.smithi160.stdout:420: writing smithi16030472-420 from 2119014 to 2651492 tid 2 2024-09-21T18:07:34.983 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:34.983 INFO:tasks.rados.rados.0.smithi160.stdout:409: finishing write tid 1 to smithi16030472-409 2024-09-21T18:07:34.983 INFO:tasks.rados.rados.0.smithi160.stdout:403: finishing write tid 3 to smithi16030472-403 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: finishing write tid 4 to smithi16030472-403 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: oid 403 updating version 0 to 329 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: oid 403 updating version 329 to 330 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: oid 403 updating version 330 to 331 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 403 v 331 (ObjNum 402 snap 0 seq_num 402) dirty exists 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: left oid 403 (ObjNum 402 snap 0 seq_num 402) 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:410: finishing write tid 1 to smithi16030472-410 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:403: done (15 left) 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:421: write initial oid 421 2024-09-21T18:07:34.984 INFO:tasks.rados.rados.0.smithi160.stdout:421: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-21T18:07:34.992 INFO:tasks.rados.rados.0.smithi160.stdout:421: writing smithi16030472-421 from 658940 to 1062935 tid 1 2024-09-21T18:07:35.003 INFO:tasks.rados.rados.0.smithi160.stdout:421: writing smithi16030472-421 from 1777496 to 2453632 tid 2 2024-09-21T18:07:35.008 INFO:tasks.rados.rados.0.smithi160.stdout:421: writing smithi16030472-421 from 2950951 to 3058940 tid 3 2024-09-21T18:07:35.008 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.008 INFO:tasks.rados.rados.0.smithi160.stdout:410: finishing write tid 2 to smithi16030472-410 2024-09-21T18:07:35.008 INFO:tasks.rados.rados.0.smithi160.stdout:410: finishing write tid 3 to smithi16030472-410 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: finishing write tid 4 to smithi16030472-410 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: finishing write tid 5 to smithi16030472-410 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: oid 410 updating version 0 to 334 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: oid 410 version 334 is already newer than 332 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: oid 410 updating version 334 to 335 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: oid 410 version 335 is already newer than 333 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 335 (ObjNum 409 snap 0 seq_num 409) dirty exists 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: left oid 410 (ObjNum 409 snap 0 seq_num 409) 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:413: finishing write tid 1 to smithi16030472-413 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:410: done (15 left) 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:422: write initial oid 422 2024-09-21T18:07:35.009 INFO:tasks.rados.rados.0.smithi160.stdout:422: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-21T18:07:35.011 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.012 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.018 INFO:tasks.rados.rados.0.smithi160.stdout:422: writing smithi16030472-422 from 675747 to 1154991 tid 1 2024-09-21T18:07:35.019 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:35.016+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:35.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout:422: writing smithi16030472-422 from 1835978 to 2372125 tid 2 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout:413: finishing write tid 2 to smithi16030472-413 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout:402: finishing write tid 2 to smithi16030472-402 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout:413: finishing write tid 3 to smithi16030472-413 2024-09-21T18:07:35.029 INFO:tasks.rados.rados.0.smithi160.stdout:413: finishing write tid 4 to smithi16030472-413 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: finishing write tid 3 to smithi16030472-402 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: finishing write tid 5 to smithi16030472-413 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: oid 413 updating version 0 to 338 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: oid 413 version 338 is already newer than 336 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: oid 413 version 338 is already newer than 337 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: oid 413 updating version 338 to 339 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 339 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:413: left oid 413 (ObjNum 412 snap 0 seq_num 412) 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: finishing write tid 4 to smithi16030472-402 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: oid 402 updating version 0 to 264 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: oid 402 version 264 is already newer than 263 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: oid 402 updating version 264 to 265 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 402 v 265 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-09-21T18:07:35.030 INFO:tasks.rados.rados.0.smithi160.stdout:402: left oid 402 (ObjNum 401 snap 0 seq_num 401) 2024-09-21T18:07:35.031 INFO:tasks.rados.rados.0.smithi160.stdout:402: done (15 left) 2024-09-21T18:07:35.031 INFO:tasks.rados.rados.0.smithi160.stdout:413: done (14 left) 2024-09-21T18:07:35.031 INFO:tasks.rados.rados.0.smithi160.stdout:423: write initial oid 423 2024-09-21T18:07:35.031 INFO:tasks.rados.rados.0.smithi160.stdout:423: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2024-09-21T18:07:35.038 INFO:tasks.rados.rados.0.smithi160.stdout:423: writing smithi16030472-423 from 692554 to 1247047 tid 1 2024-09-21T18:07:35.047 INFO:tasks.rados.rados.0.smithi160.stdout:423: writing smithi16030472-423 from 1778107 to 2457912 tid 2 2024-09-21T18:07:35.052 INFO:tasks.rados.rados.0.smithi160.stdout:423: writing smithi16030472-423 from 2889444 to 3092554 tid 3 2024-09-21T18:07:35.052 INFO:tasks.rados.rados.0.smithi160.stdout:424: write initial oid 424 2024-09-21T18:07:35.052 INFO:tasks.rados.rados.0.smithi160.stdout:424: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-21T18:07:35.060 INFO:tasks.rados.rados.0.smithi160.stdout:424: writing smithi16030472-424 from 709361 to 1339103 tid 1 2024-09-21T18:07:35.067 INFO:tasks.rados.rados.0.smithi160.stdout:424: writing smithi16030472-424 from 1836589 to 2376405 tid 2 2024-09-21T18:07:35.072 INFO:tasks.rados.rados.0.smithi160.stdout:424: writing smithi16030472-424 from 3033220 to 3109361 tid 3 2024-09-21T18:07:35.072 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.120 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:35.114+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:35.194 INFO:tasks.rados.rados.0.smithi160.stdout:420: finishing write tid 1 to smithi16030472-420 2024-09-21T18:07:35.195 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.195 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.210 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:35.221 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:35.216+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:35.301 INFO:tasks.rados.rados.0.smithi160.stdout:420: finishing write tid 2 to smithi16030472-420 2024-09-21T18:07:35.301 INFO:tasks.rados.rados.0.smithi160.stdout:420: finishing write tid 3 to smithi16030472-420 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:420: finishing write tid 4 to smithi16030472-420 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:420: oid 420 updating version 0 to 304 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:420: oid 420 version 304 is already newer than 303 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:420: oid 420 updating version 304 to 305 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 420 v 305 (ObjNum 419 snap 0 seq_num 419) dirty exists 2024-09-21T18:07:35.302 INFO:tasks.rados.rados.0.smithi160.stdout:420: left oid 420 (ObjNum 419 snap 0 seq_num 419) 2024-09-21T18:07:35.303 INFO:tasks.rados.rados.0.smithi160.stdout:423: finishing write tid 1 to smithi16030472-423 2024-09-21T18:07:35.303 INFO:tasks.rados.rados.0.smithi160.stdout:420: done (15 left) 2024-09-21T18:07:35.303 INFO:tasks.rados.rados.0.smithi160.stdout:425: write initial oid 425 2024-09-21T18:07:35.303 INFO:tasks.rados.rados.0.smithi160.stdout:425: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-21T18:07:35.310 INFO:tasks.rados.rados.0.smithi160.stdout:425: writing smithi16030472-425 from 726168 to 1431159 tid 1 2024-09-21T18:07:35.316 INFO:tasks.rados.rados.0.smithi160.stdout:425: writing smithi16030472-425 from 1895071 to 2578545 tid 2 2024-09-21T18:07:35.316 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.316 INFO:tasks.rados.rados.0.smithi160.stdout:423: finishing write tid 2 to smithi16030472-423 2024-09-21T18:07:35.322 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:35.316+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:35.372 INFO:tasks.rados.rados.0.smithi160.stdout:416: finishing write tid 1 to smithi16030472-416 2024-09-21T18:07:35.372 INFO:tasks.rados.rados.0.smithi160.stdout:416: finishing write tid 2 to smithi16030472-416 2024-09-21T18:07:35.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:07:35.379 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.379 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:35.403 INFO:tasks.rados.rados.0.smithi160.stdout:423: finishing write tid 3 to smithi16030472-423 2024-09-21T18:07:35.403 INFO:tasks.rados.rados.0.smithi160.stdout:423: finishing write tid 4 to smithi16030472-423 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: finishing write tid 5 to smithi16030472-423 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: oid 423 updating version 0 to 308 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: oid 423 updating version 308 to 309 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: oid 423 version 309 is already newer than 307 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: oid 423 version 309 is already newer than 306 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 423 v 309 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: left oid 423 (ObjNum 422 snap 0 seq_num 422) 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:423: done (15 left) 2024-09-21T18:07:35.404 INFO:tasks.rados.rados.0.smithi160.stdout:426: write initial oid 426 2024-09-21T18:07:35.405 INFO:tasks.rados.rados.0.smithi160.stdout:426: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-21T18:07:35.414 INFO:tasks.rados.rados.0.smithi160.stdout:426: writing smithi16030472-426 from 742975 to 1523215 tid 1 2024-09-21T18:07:35.420 INFO:tasks.rados.rados.0.smithi160.stdout:426: writing smithi16030472-426 from 1953553 to 2380685 tid 2 2024-09-21T18:07:35.425 INFO:tasks.rados.rados.0.smithi160.stdout:426: writing smithi16030472-426 from 2971713 to 3142975 tid 3 2024-09-21T18:07:35.425 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.441 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:35.438+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:35.521 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.521 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.524 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:35.518+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:35.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: finishing write tid 2 to smithi16030472-409 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: finishing write tid 3 to smithi16030472-409 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: finishing write tid 4 to smithi16030472-409 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: finishing write tid 5 to smithi16030472-409 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: oid 409 updating version 0 to 281 2024-09-21T18:07:35.578 INFO:tasks.rados.rados.0.smithi160.stdout:409: oid 409 updating version 281 to 283 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:409: oid 409 updating version 283 to 284 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:409: oid 409 version 284 is already newer than 282 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 284 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:409: left oid 409 (ObjNum 408 snap 0 seq_num 408) 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:416: finishing write tid 3 to smithi16030472-416 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:416: finishing write tid 4 to smithi16030472-416 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:409: done (15 left) 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:427: write initial oid 427 2024-09-21T18:07:35.579 INFO:tasks.rados.rados.0.smithi160.stdout:427: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-21T18:07:35.590 INFO:tasks.rados.rados.0.smithi160.stdout:427: writing smithi16030472-427 from 759782 to 1331624 tid 1 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:427: writing smithi16030472-427 from 2012035 to 2699178 tid 2 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: finishing write tid 5 to smithi16030472-416 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: oid 416 updating version 0 to 283 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: oid 416 version 283 is already newer than 282 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: oid 416 updating version 283 to 285 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: oid 416 version 285 is already newer than 284 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 285 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: left oid 416 (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:419: finishing write tid 1 to smithi16030472-419 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:419: finishing write tid 2 to smithi16030472-419 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:416: done (15 left) 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:428: write initial oid 428 2024-09-21T18:07:35.601 INFO:tasks.rados.rados.0.smithi160.stdout:428: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-21T18:07:35.612 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T18:07:35.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:07:35.613 INFO:tasks.rados.rados.0.smithi160.stdout:428: writing smithi16030472-428 from 776589 to 1423680 tid 1 2024-09-21T18:07:35.621 INFO:tasks.rados.rados.0.smithi160.stdout:428: writing smithi16030472-428 from 2070517 to 2501318 tid 2 2024-09-21T18:07:35.625 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:35.621+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:428: writing smithi16030472-428 from 3026559 to 3176589 tid 3 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:419: finishing write tid 3 to smithi16030472-419 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:419: finishing write tid 4 to smithi16030472-419 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:419: oid 419 updating version 0 to 287 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:419: oid 419 updating version 287 to 288 2024-09-21T18:07:35.628 INFO:tasks.rados.rados.0.smithi160.stdout:419: oid 419 version 288 is already newer than 286 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 419 v 288 (ObjNum 418 snap 0 seq_num 418) dirty exists 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:419: left oid 419 (ObjNum 418 snap 0 seq_num 418) 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:421: finishing write tid 1 to smithi16030472-421 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:421: finishing write tid 2 to smithi16030472-421 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: finishing write tid 1 to smithi16030472-404 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: finishing write tid 2 to smithi16030472-404 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: finishing write tid 3 to smithi16030472-404 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: finishing write tid 4 to smithi16030472-404 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: oid 404 updating version 0 to 267 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: oid 404 updating version 267 to 268 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: oid 404 version 268 is already newer than 266 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 404 v 268 (ObjNum 403 snap 0 seq_num 403) dirty exists 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:404: left oid 404 (ObjNum 403 snap 0 seq_num 403) 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:405: finishing write tid 1 to smithi16030472-405 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:405: finishing write tid 2 to smithi16030472-405 2024-09-21T18:07:35.629 INFO:tasks.rados.rados.0.smithi160.stdout:405: finishing write tid 3 to smithi16030472-405 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: finishing write tid 4 to smithi16030472-405 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: oid 405 updating version 0 to 271 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: oid 405 version 271 is already newer than 269 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: oid 405 version 271 is already newer than 270 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 405 v 271 (ObjNum 404 snap 0 seq_num 404) dirty exists 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: left oid 405 (ObjNum 404 snap 0 seq_num 404) 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:406: finishing write tid 1 to smithi16030472-406 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:406: finishing write tid 2 to smithi16030472-406 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:406: finishing write tid 3 to smithi16030472-406 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:406: finishing write tid 4 to smithi16030472-406 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:404: done (15 left) 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:405: done (14 left) 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:419: done (13 left) 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:429: write initial oid 429 2024-09-21T18:07:35.630 INFO:tasks.rados.rados.0.smithi160.stdout:429: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-21T18:07:35.641 INFO:tasks.rados.rados.0.smithi160.stdout:429: writing smithi16030472-429 from 793396 to 1515736 tid 1 2024-09-21T18:07:35.651 INFO:tasks.rados.rados.0.smithi160.stdout:429: writing smithi16030472-429 from 2128999 to 2819811 tid 2 2024-09-21T18:07:35.652 INFO:tasks.rados.rados.0.smithi160.stdout:430: write initial oid 430 2024-09-21T18:07:35.652 INFO:tasks.rados.rados.0.smithi160.stdout:430: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-21T18:07:35.660 INFO:tasks.rados.rados.0.smithi160.stdout:430: writing smithi16030472-430 from 410203 to 1207792 tid 1 2024-09-21T18:07:35.667 INFO:tasks.rados.rados.0.smithi160.stdout:430: writing smithi16030472-430 from 1787481 to 2221951 tid 2 2024-09-21T18:07:35.673 INFO:tasks.rados.rados.0.smithi160.stdout:430: writing smithi16030472-430 from 2681405 to 3210203 tid 3 2024-09-21T18:07:35.673 INFO:tasks.rados.rados.0.smithi160.stdout:431: write initial oid 431 2024-09-21T18:07:35.673 INFO:tasks.rados.rados.0.smithi160.stdout:431: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-21T18:07:35.678 INFO:tasks.rados.rados.0.smithi160.stdout:431: writing smithi16030472-431 from 427010 to 899848 tid 1 2024-09-21T18:07:35.685 INFO:tasks.rados.rados.0.smithi160.stdout:431: writing smithi16030472-431 from 1329610 to 2024091 tid 2 2024-09-21T18:07:35.692 INFO:tasks.rados.rados.0.smithi160.stdout:431: writing smithi16030472-431 from 2708828 to 3227010 tid 3 2024-09-21T18:07:35.692 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:35.697 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.697 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.710 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:35.725 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:35.721+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:35.826 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:35.822+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:35.829 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:35.829 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:35.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:35.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:35.926 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:07:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:07:35.934 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:35.930+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:35.945 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-21T18:07:35.063666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1340250799,"num_objects":502,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":941,"num_read_kb":420,"num_write":1733,"num_write_kb":468822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":143,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1491,"ondisk_log_size":1491,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":1656572,"kb_used_data":1195020,"kb_used_omap":24,"kb_used_meta":461223,"kb_avail":1508292868,"statfs":{"total":1546188226560,"available":1544491896832,"internally_reserved":0,"allocated":1223700480,"data_stored":1233930699,"data_compressed":18900499,"data_compressed_allocated":20164608,"data_compressed_original":41885696,"omap_allocated":25520,"internal_metadata":472292432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4,15,39,47,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":492032267,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":308,"num_read_kb":132,"num_write":621,"num_write_kb":184311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":35,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.747909"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":198,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.656912+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:07:30.656912+0000","last_peered":"2024-09-21T18:07:30.656912+0000","last_clean":"2024-09-21T18:07:30.656912+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:07:30.656912+0000","last_undegraded":"2024-09-21T18:07:30.656912+0000","last_fullsized":"2024-09-21T18:07:30.656912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.556907+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:07:30.556907+0000","last_peered":"2024-09-21T18:07:30.556907+0000","last_clean":"2024-09-21T18:07:30.556907+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:07:30.556907+0000","last_undegraded":"2024-09-21T18:07:30.556907+0000","last_fullsized":"2024-09-21T18:07:30.556907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"51'239","reported_seq":425,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:31.933951+0000","last_change":"2024-09-21T18:07:01.287376+0000","last_active":"2024-09-21T18:07:31.933951+0000","last_peered":"2024-09-21T18:07:31.933951+0000","last_clean":"2024-09-21T18:07:31.933951+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:07:31.933951+0000","last_undegraded":"2024-09-21T18:07:31.933951+0000","last_fullsized":"2024-09-21T18:07:31.933951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:21.792563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221671757,"num_objects":90,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":183,"num_read_kb":47,"num_write":291,"num_write_kb":101666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.765808+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:07:30.765808+0000","last_peered":"2024-09-21T18:07:30.765808+0000","last_clean":"2024-09-21T18:07:30.765808+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:07:30.765808+0000","last_undegraded":"2024-09-21T18:07:30.765808+0000","last_fullsized":"2024-09-21T18:07:30.765808+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"50'277","reported_seq":428,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.301354+0000","last_change":"2024-09-21T18:07:01.490023+0000","last_active":"2024-09-21T18:07:30.301354+0000","last_peered":"2024-09-21T18:07:30.301354+0000","last_clean":"2024-09-21T18:07:30.301354+0000","last_became_active":"2024-09-21T18:07:01.389573+0000","last_became_peered":"2024-09-21T18:07:01.389573+0000","last_unstale":"2024-09-21T18:07:30.301354+0000","last_undegraded":"2024-09-21T18:07:30.301354+0000","last_fullsized":"2024-09-21T18:07:30.301354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:03.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240093121,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":170,"num_read_kb":64,"num_write":348,"num_write_kb":107748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.754149+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:07:30.754149+0000","last_peered":"2024-09-21T18:07:30.754149+0000","last_clean":"2024-09-21T18:07:30.754149+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:07:30.754149+0000","last_undegraded":"2024-09-21T18:07:30.754149+0000","last_fullsized":"2024-09-21T18:07:30.754149+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"51'256","reported_seq":468,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.680056+0000","last_change":"2024-09-21T18:07:01.405525+0000","last_active":"2024-09-21T18:07:32.680056+0000","last_peered":"2024-09-21T18:07:32.680056+0000","last_clean":"2024-09-21T18:07:32.680056+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:07:32.680056+0000","last_undegraded":"2024-09-21T18:07:32.680056+0000","last_fullsized":"2024-09-21T18:07:32.680056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.560432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238557258,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":203,"num_read_kb":49,"num_write":312,"num_write_kb":109170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":67,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.765833+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:07:30.765833+0000","last_peered":"2024-09-21T18:07:30.765833+0000","last_clean":"2024-09-21T18:07:30.765833+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:07:30.765833+0000","last_undegraded":"2024-09-21T18:07:30.765833+0000","last_fullsized":"2024-09-21T18:07:30.765833+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"51'258","reported_seq":424,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.789471+0000","last_change":"2024-09-21T18:07:01.288102+0000","last_active":"2024-09-21T18:07:32.789471+0000","last_peered":"2024-09-21T18:07:32.789471+0000","last_clean":"2024-09-21T18:07:32.789471+0000","last_became_active":"2024-09-21T18:07:01.287689+0000","last_became_peered":"2024-09-21T18:07:01.287689+0000","last_unstale":"2024-09-21T18:07:32.789471+0000","last_undegraded":"2024-09-21T18:07:32.789471+0000","last_fullsized":"2024-09-21T18:07:32.789471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.216721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230476334,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":175,"num_read_kb":63,"num_write":324,"num_write_kb":104515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"51'125","reported_seq":218,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.578246+0000","last_change":"2024-09-21T18:07:03.553363+0000","last_active":"2024-09-21T18:07:32.578246+0000","last_peered":"2024-09-21T18:07:32.578246+0000","last_clean":"2024-09-21T18:07:32.578246+0000","last_became_active":"2024-09-21T18:07:03.552723+0000","last_became_peered":"2024-09-21T18:07:03.552723+0000","last_unstale":"2024-09-21T18:07:32.578246+0000","last_undegraded":"2024-09-21T18:07:32.578246+0000","last_fullsized":"2024-09-21T18:07:32.578246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:07.591572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144926101,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":116,"num_write_kb":18050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:27.330537+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:07:27.330537+0000","last_peered":"2024-09-21T18:07:27.330537+0000","last_clean":"2024-09-21T18:07:27.330537+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:07:27.330537+0000","last_undegraded":"2024-09-21T18:07:27.330537+0000","last_fullsized":"2024-09-21T18:07:27.330537+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"51'65","reported_seq":136,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:31.395928+0000","last_change":"2024-09-21T18:07:04.454454+0000","last_active":"2024-09-21T18:07:31.395928+0000","last_peered":"2024-09-21T18:07:31.395928+0000","last_clean":"2024-09-21T18:07:31.395928+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:07:31.395928+0000","last_undegraded":"2024-09-21T18:07:31.395928+0000","last_fullsized":"2024-09-21T18:07:31.395928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.533440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69345187,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":59,"num_write_kb":8354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.712137+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:07:30.712137+0000","last_peered":"2024-09-21T18:07:30.712137+0000","last_clean":"2024-09-21T18:07:30.712137+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:07:30.712137+0000","last_undegraded":"2024-09-21T18:07:30.712137+0000","last_fullsized":"2024-09-21T18:07:30.712137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"50'107","reported_seq":196,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:29.989895+0000","last_change":"2024-09-21T18:07:03.453942+0000","last_active":"2024-09-21T18:07:29.989895+0000","last_peered":"2024-09-21T18:07:29.989895+0000","last_clean":"2024-09-21T18:07:29.989895+0000","last_became_active":"2024-09-21T18:07:03.453360+0000","last_became_peered":"2024-09-21T18:07:03.453360+0000","last_unstale":"2024-09-21T18:07:29.989895+0000","last_undegraded":"2024-09-21T18:07:29.989895+0000","last_fullsized":"2024-09-21T18:07:29.989895+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:14.477508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127648703,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":96,"num_write_kb":11819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.846739+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:07:30.846739+0000","last_peered":"2024-09-21T18:07:30.846739+0000","last_clean":"2024-09-21T18:07:30.846739+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:07:30.846739+0000","last_undegraded":"2024-09-21T18:07:30.846739+0000","last_fullsized":"2024-09-21T18:07:30.846739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"51'49","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.872340+0000","last_change":"2024-09-21T18:07:04.220636+0000","last_active":"2024-09-21T18:07:30.872340+0000","last_peered":"2024-09-21T18:07:30.872340+0000","last_clean":"2024-09-21T18:07:30.872340+0000","last_became_active":"2024-09-21T18:07:04.220350+0000","last_became_peered":"2024-09-21T18:07:04.220350+0000","last_unstale":"2024-09-21T18:07:30.872340+0000","last_undegraded":"2024-09-21T18:07:30.872340+0000","last_fullsized":"2024-09-21T18:07:30.872340+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.012491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66941951,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":40,"num_write_kb":4826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":23,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.735407+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:07:30.735407+0000","last_peered":"2024-09-21T18:07:30.735407+0000","last_clean":"2024-09-21T18:07:30.735407+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:07:30.735407+0000","last_undegraded":"2024-09-21T18:07:30.735407+0000","last_fullsized":"2024-09-21T18:07:30.735407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":408861942,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":311,"num_write_kb":43049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":143,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383578112,"data_stored":403440059,"data_compressed":19141322,"data_compressed_allocated":20418560,"data_compressed_original":41713664,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":930798470,"num_objects":357,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":731,"num_read_kb":223,"num_write":1275,"num_write_kb":423099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":876269568,"data_stored":869745230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1030,"ondisk_log_size":1030,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":201188,"kb_used_data":170004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94170652,"statfs":{"total":96636764160,"available":96430747648,"internally_reserved":0,"allocated":174084096,"data_stored":176664026,"data_compressed":2787437,"data_compressed_allocated":2973696,"data_compressed_original":6856704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,7,4,6,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":528384,"data_stored":273298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145344,"kb_used_data":114160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94226496,"statfs":{"total":96636764160,"available":96487931904,"internally_reserved":0,"allocated":116899840,"data_stored":115785717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,4,19,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313229,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188456,"kb_used_data":157272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94183384,"statfs":{"total":96636764160,"available":96443785216,"internally_reserved":0,"allocated":161046528,"data_stored":162770537,"data_compressed":3018161,"data_compressed_allocated":3223552,"data_compressed_original":6160384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,7,23,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313230,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":273634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":278804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159400,"kb_used_data":128216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94212440,"statfs":{"total":96636764160,"available":96473538560,"internally_reserved":0,"allocated":131293184,"data_stored":132290484,"data_compressed":2028526,"data_compressed_allocated":2158592,"data_compressed_original":4235264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":268813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152948,"kb_used_data":125920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94218892,"statfs":{"total":96636764160,"available":96480145408,"internally_reserved":0,"allocated":128942080,"data_stored":129822175,"data_compressed":1597141,"data_compressed_allocated":1703936,"data_compressed_original":3657728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":524288,"data_stored":276231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":278827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126408,"kb_used_data":99380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94245432,"statfs":{"total":96636764160,"available":96507322368,"internally_reserved":0,"allocated":101765120,"data_stored":100809589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151676,"kb_used_data":120488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94220164,"statfs":{"total":96636764160,"available":96481447936,"internally_reserved":0,"allocated":123379712,"data_stored":129505553,"data_compressed":5843567,"data_compressed_allocated":6242304,"data_compressed_original":13082624,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":193068,"kb_used_data":161880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94178772,"statfs":{"total":96636764160,"available":96439062528,"internally_reserved":0,"allocated":165765120,"data_stored":168572691,"data_compressed":3625667,"data_compressed_allocated":3862528,"data_compressed_original":7892992,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,8,15],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":524288,"data_stored":275647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141232,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94230608,"statfs":{"total":96636764160,"available":96492142592,"internally_reserved":0,"allocated":116944896,"data_stored":115784673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116363264,"data_stored":115509287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104099840,"data_stored":103347866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":111099904,"data_stored":110233364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":103829504,"data_stored":103063956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":107679744,"data_stored":106879522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":104775680,"data_stored":104006735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":119779328,"data_stored":118851931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":108642304,"data_stored":107852569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31715328,"data_stored":33685248,"data_compressed":2028526,"data_compressed_allocated":2158592,"data_compressed_original":4235264,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59297792,"data_stored":62188824,"data_compressed":3171769,"data_compressed_allocated":3391488,"data_compressed_original":6488064,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32071680,"data_stored":33433696,"data_compressed":1564354,"data_compressed_allocated":1662976,"data_compressed_original":3145728,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":130011136,"data_stored":136718440,"data_compressed":5996356,"data_compressed_allocated":6410240,"data_compressed_original":13606912,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":63787008,"data_stored":67118944,"data_compressed":3592880,"data_compressed_allocated":3821568,"data_compressed_original":7380992,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":66695168,"data_stored":70294907,"data_compressed":2787437,"data_compressed_allocated":2973696,"data_compressed_original":6856704,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:07:35.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:07:35.981 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:35.999 INFO:tasks.rados.rados.0.smithi160.stdout:406: finishing write tid 5 to smithi16030472-406 2024-09-21T18:07:35.999 INFO:tasks.rados.rados.0.smithi160.stdout:406: oid 406 updating version 0 to 273 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:406: oid 406 updating version 273 to 274 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:406: oid 406 version 274 is already newer than 272 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:406: oid 406 updating version 274 to 275 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 406 v 275 (ObjNum 405 snap 0 seq_num 405) dirty exists 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:406: left oid 406 (ObjNum 405 snap 0 seq_num 405) 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:407: finishing write tid 1 to smithi16030472-407 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:407: finishing write tid 2 to smithi16030472-407 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:407: finishing write tid 3 to smithi16030472-407 2024-09-21T18:07:36.000 INFO:tasks.rados.rados.0.smithi160.stdout:407: finishing write tid 4 to smithi16030472-407 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:407: oid 407 updating version 0 to 277 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:407: oid 407 updating version 277 to 278 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:407: oid 407 version 278 is already newer than 276 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 278 (ObjNum 406 snap 0 seq_num 406) dirty exists 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:407: left oid 407 (ObjNum 406 snap 0 seq_num 406) 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:411: finishing write tid 1 to smithi16030472-411 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:411: finishing write tid 2 to smithi16030472-411 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:406: done (15 left) 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:407: done (14 left) 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:432: write initial oid 432 2024-09-21T18:07:36.001 INFO:tasks.rados.rados.0.smithi160.stdout:432: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2024-09-21T18:07:36.002 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:36.003 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:36.006 INFO:tasks.rados.rados.0.smithi160.stdout:432: writing smithi16030472-432 from 443817 to 991904 tid 1 2024-09-21T18:07:36.014 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:36.014 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:36.015 INFO:tasks.rados.rados.0.smithi160.stdout:432: writing smithi16030472-432 from 1788092 to 2226231 tid 2 2024-09-21T18:07:36.022 INFO:tasks.rados.rados.0.smithi160.stdout:432: writing smithi16030472-432 from 3019898 to 3243817 tid 3 2024-09-21T18:07:36.022 INFO:tasks.rados.rados.0.smithi160.stdout:433: write initial oid 433 2024-09-21T18:07:36.022 INFO:tasks.rados.rados.0.smithi160.stdout:433: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-21T18:07:36.023 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:36.024 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:36.029 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:36.030 INFO:tasks.rados.rados.0.smithi160.stdout:433: writing smithi16030472-433 from 460624 to 1083960 tid 1 2024-09-21T18:07:36.030 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:36.024+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:36.039 INFO:tasks.rados.rados.0.smithi160.stdout:433: writing smithi16030472-433 from 1846574 to 2544724 tid 2 2024-09-21T18:07:36.043 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:36.044 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:433: writing smithi16030472-433 from 3163674 to 3260624 tid 3 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:411: finishing write tid 3 to smithi16030472-411 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:411: finishing write tid 4 to smithi16030472-411 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:411: finishing write tid 5 to smithi16030472-411 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:411: oid 411 updating version 0 to 279 2024-09-21T18:07:36.045 INFO:tasks.rados.rados.0.smithi160.stdout:411: oid 411 updating version 279 to 282 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:411: oid 411 version 282 is already newer than 280 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:411: oid 411 version 282 is already newer than 281 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 411 v 282 (ObjNum 410 snap 0 seq_num 410) dirty exists 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:411: left oid 411 (ObjNum 410 snap 0 seq_num 410) 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:412: finishing write tid 1 to smithi16030472-412 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:411: done (15 left) 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:434: write initial oid 434 2024-09-21T18:07:36.046 INFO:tasks.rados.rados.0.smithi160.stdout:434: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-21T18:07:36.052 INFO:tasks.rados.rados.0.smithi160.stdout:434: writing smithi16030472-434 from 477431 to 1176016 tid 1 2024-09-21T18:07:36.060 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:36.060 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:36.061 INFO:tasks.rados.rados.0.smithi160.stdout:434: writing smithi16030472-434 from 1905056 to 2346864 tid 2 2024-09-21T18:07:36.067 INFO:tasks.rados.rados.0.smithi160.stdout:434: writing smithi16030472-434 from 3074744 to 3277431 tid 3 2024-09-21T18:07:36.067 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:412: finishing write tid 2 to smithi16030472-412 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:412: finishing write tid 3 to smithi16030472-412 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: finishing write tid 1 to smithi16030472-415 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: finishing write tid 2 to smithi16030472-415 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: finishing write tid 3 to smithi16030472-415 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: finishing write tid 4 to smithi16030472-415 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: oid 415 updating version 0 to 288 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: oid 415 version 288 is already newer than 287 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: oid 415 version 288 is already newer than 286 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 415 v 288 (ObjNum 414 snap 0 seq_num 414) dirty exists 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: left oid 415 (ObjNum 414 snap 0 seq_num 414) 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:417: finishing write tid 1 to smithi16030472-417 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:415: done (15 left) 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:435: write initial oid 435 2024-09-21T18:07:36.068 INFO:tasks.rados.rados.0.smithi160.stdout:435: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-21T18:07:36.073 INFO:tasks.rados.rados.0.smithi160.stdout:435: writing smithi16030472-435 from 494238 to 984425 tid 1 2024-09-21T18:07:36.081 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:36.082 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:36.082 INFO:tasks.rados.rados.0.smithi160.stdout:435: writing smithi16030472-435 from 1563538 to 2265357 tid 2 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:435: writing smithi16030472-435 from 2818520 to 3294238 tid 3 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:424: finishing write tid 1 to smithi16030472-424 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:424: finishing write tid 2 to smithi16030472-424 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:424: finishing write tid 3 to smithi16030472-424 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:424: finishing write tid 4 to smithi16030472-424 2024-09-21T18:07:36.089 INFO:tasks.rados.rados.0.smithi160.stdout:424: finishing write tid 5 to smithi16030472-424 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: oid 424 updating version 0 to 290 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: oid 424 updating version 290 to 292 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: oid 424 updating version 292 to 293 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: oid 424 version 293 is already newer than 291 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 293 (ObjNum 423 snap 0 seq_num 423) dirty exists 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: left oid 424 (ObjNum 423 snap 0 seq_num 423) 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:425: finishing write tid 1 to smithi16030472-425 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:424: done (15 left) 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:436: write initial oid 436 2024-09-21T18:07:36.090 INFO:tasks.rados.rados.0.smithi160.stdout:436: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-21T18:07:36.095 INFO:tasks.rados.rados.0.smithi160.stdout:436: writing smithi16030472-436 from 511045 to 1076481 tid 1 2024-09-21T18:07:36.101 INFO:tasks.rados.rados.0.smithi160.stdout:436: writing smithi16030472-436 from 1622020 to 2067497 tid 2 2024-09-21T18:07:36.102 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:36.102 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:36.109 INFO:tasks.rados.rados.0.smithi160.stdout:436: writing smithi16030472-436 from 2729590 to 3286445 tid 3 2024-09-21T18:07:36.109 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.109 INFO:tasks.rados.rados.0.smithi160.stdout:421: finishing write tid 3 to smithi16030472-421 2024-09-21T18:07:36.109 INFO:tasks.rados.rados.0.smithi160.stdout:421: finishing write tid 4 to smithi16030472-421 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: finishing write tid 5 to smithi16030472-421 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: oid 421 updating version 0 to 289 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: oid 421 updating version 289 to 290 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: oid 421 updating version 290 to 292 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: oid 421 version 292 is already newer than 291 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 421 v 292 (ObjNum 420 snap 0 seq_num 420) dirty exists 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:421: left oid 421 (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: finishing write tid 2 to smithi16030472-425 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: finishing write tid 3 to smithi16030472-425 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: finishing write tid 4 to smithi16030472-425 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: oid 425 updating version 0 to 341 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: oid 425 updating version 341 to 342 2024-09-21T18:07:36.110 INFO:tasks.rados.rados.0.smithi160.stdout:425: oid 425 version 342 is already newer than 340 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 425 v 342 (ObjNum 424 snap 0 seq_num 424) dirty exists 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:425: left oid 425 (ObjNum 424 snap 0 seq_num 424) 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:426: finishing write tid 1 to smithi16030472-426 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: finishing write tid 1 to smithi16030472-422 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: finishing write tid 2 to smithi16030472-422 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: finishing write tid 3 to smithi16030472-422 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: finishing write tid 4 to smithi16030472-422 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: oid 422 updating version 0 to 293 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: oid 422 updating version 293 to 294 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: oid 422 updating version 294 to 295 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 422 v 295 (ObjNum 421 snap 0 seq_num 421) dirty exists 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:422: left oid 422 (ObjNum 421 snap 0 seq_num 421) 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:430: finishing write tid 1 to smithi16030472-430 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:426: finishing write tid 2 to smithi16030472-426 2024-09-21T18:07:36.111 INFO:tasks.rados.rados.0.smithi160.stdout:426: finishing write tid 3 to smithi16030472-426 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:430: finishing write tid 2 to smithi16030472-430 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:430: finishing write tid 3 to smithi16030472-430 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:421: done (15 left) 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:422: done (14 left) 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:425: done (13 left) 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:437: write initial oid 437 2024-09-21T18:07:36.112 INFO:tasks.rados.rados.0.smithi160.stdout:437: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-21T18:07:36.115 INFO:tasks.rados.rados.0.smithi160.stdout:437: writing smithi16030472-437 from 527852 to 1168537 tid 1 2024-09-21T18:07:36.121 INFO:tasks.rados.rados.0.smithi160.stdout:437: writing smithi16030472-437 from 1680502 to 2269637 tid 2 2024-09-21T18:07:36.122 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:36.127 INFO:tasks.rados.rados.0.smithi160.stdout:437: writing smithi16030472-437 from 2757013 to 3327852 tid 3 2024-09-21T18:07:36.127 INFO:tasks.rados.rados.0.smithi160.stdout:438: write initial oid 438 2024-09-21T18:07:36.127 INFO:tasks.rados.rados.0.smithi160.stdout:438: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-21T18:07:36.129 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:36.126+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:36.134 INFO:tasks.rados.rados.0.smithi160.stdout:438: writing smithi16030472-438 from 544659 to 1260593 tid 1 2024-09-21T18:07:36.138 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:36.138 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:36.139 INFO:tasks.rados.rados.0.smithi160.stdout:438: writing smithi16030472-438 from 1738984 to 2188130 tid 2 2024-09-21T18:07:36.145 INFO:tasks.rados.rados.0.smithi160.stdout:438: writing smithi16030472-438 from 2784436 to 3344659 tid 3 2024-09-21T18:07:36.146 INFO:tasks.rados.rados.0.smithi160.stdout:439: write initial oid 439 2024-09-21T18:07:36.146 INFO:tasks.rados.rados.0.smithi160.stdout:439: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-21T18:07:36.153 INFO:tasks.rados.rados.0.smithi160.stdout:439: writing smithi16030472-439 from 561466 to 1352649 tid 1 2024-09-21T18:07:36.159 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:36.160 INFO:tasks.rados.rados.0.smithi160.stdout:439: writing smithi16030472-439 from 2081113 to 2673917 tid 2 2024-09-21T18:07:36.164 INFO:tasks.rados.rados.0.smithi160.stdout:439: writing smithi16030472-439 from 3095506 to 3361466 tid 3 2024-09-21T18:07:36.164 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.179 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: finishing write tid 4 to smithi16030472-412 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: finishing write tid 5 to smithi16030472-412 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: oid 412 updating version 0 to 283 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: oid 412 updating version 283 to 284 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: oid 412 updating version 284 to 285 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: oid 412 updating version 285 to 286 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 286 (ObjNum 411 snap 0 seq_num 411) dirty exists 2024-09-21T18:07:36.203 INFO:tasks.rados.rados.0.smithi160.stdout:412: left oid 412 (ObjNum 411 snap 0 seq_num 411) 2024-09-21T18:07:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:417: finishing write tid 2 to smithi16030472-417 2024-09-21T18:07:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:417: finishing write tid 3 to smithi16030472-417 2024-09-21T18:07:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:412: done (15 left) 2024-09-21T18:07:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:440: write initial oid 440 2024-09-21T18:07:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:440: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-21T18:07:36.206 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:29 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'19, mlcod=43'17)", 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.357035171, 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20988762999999999, 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845721+0000", 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0570000000000001e-06 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847734+0000", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020132850000000001 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847771+0000", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6415000000000003e-05 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847794+0000", 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3730000000000001e-05 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948080+0000", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100285443 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948115+0000", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5098999999999997e-05 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055500+0000", 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107384654 2024-09-21T18:07:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055605+0000", 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010594699999999999 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:106 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'20, mlcod=43'17)", 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.354900740000001, 2024-09-21T18:07:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20783109999999999, 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847855+0000", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4569999999999999e-06 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849293+0000", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014381109999999999 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849325+0000", 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2361999999999999e-05 2024-09-21T18:07:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949454+0000", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128488 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950524+0000", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010704860000000001 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950583+0000", 2024-09-21T18:07:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8643000000000001e-05 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055613+0000", 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10503064500000001 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055683+0000", 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9907999999999995e-05 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:154 4.2 e44/33 4:6abed166:::smithi16030472-39:head v 44'23, mlcod=43'17)", 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.289952154000002, 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20421822000000001, 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912802+0000", 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3030000000000001e-06 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913279+0000", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000476928 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913294+0000", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5209e-05 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013383+0000", 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10008903600000001 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013498+0000", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011462 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013532+0000", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4069000000000002e-05 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.116937+0000", 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103404914 2024-09-21T18:07:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117019+0000", 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2140999999999998e-05 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:44 4.0 e44/33 4:01b9dd46:::smithi16030472-42:head v 44'28, mlcod=43'23)", 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 22.085082890999999, 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.117588128, 2024-09-21T18:07:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117670+0000", 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117673+0000", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4899999999999999e-06 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225500+0000", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107826882 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225515+0000", 2024-09-21T18:07:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5379999999999998e-05 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225524+0000", 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4029999999999996e-06 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225651+0000", 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012647299999999999 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.225722+0000", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1401000000000003e-05 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.235172+0000", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0094498140000000008 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.235258+0000", 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6285000000000005e-05 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:54 4.0 e44/33 4:1f69a690:::smithi16030472-48:head v 44'36, mlcod=43'23)", 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.863889645, 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.118230935, 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338863+0000", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.338865+0000", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.15e-06 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.347997+0000", 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0091320800000000008 2024-09-21T18:07:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348022+0000", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5681999999999999e-05 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348036+0000", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3570000000000001e-05 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348151+0000", 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011503 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.348186+0000", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5046000000000003e-05 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457034+0000", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10884822199999999 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.457094+0000", 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0155000000000003e-05 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:241 4.2 e48/33 4:6513c01e:::smithi16030472-210:head v 48'96, mlcod=48'94)", 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.081910047999999, 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.216069437, 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120847+0000", 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1960000000000001e-06 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125270+0000", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044228310000000003 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125303+0000", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2975999999999997e-05 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125320+0000", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.717e-05 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225582+0000", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100261425 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225619+0000", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6962000000000003e-05 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336827+0000", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111208272 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336912+0000", 2024-09-21T18:07:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5605000000000005e-05 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:242 4.2 e48/33 4:50d6db3f:::smithi16030472-204:head v 48'97, mlcod=48'94)", 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.06909134, 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.203351217, 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133665+0000", 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2389999999999998e-06 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136624+0000", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002958666 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136661+0000", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7225999999999999e-05 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136685+0000", 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4275999999999999e-05 2024-09-21T18:07:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.330181+0000", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19349608500000001 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.330257+0000", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6172000000000002e-05 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336920+0000", 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0066630810000000004 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337013+0000", 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2472000000000005e-05 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:902 4.2 e48/33 4:50d6db3f:::smithi16030472-204:head v 48'98, mlcod=48'94)", 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.138707+0000", 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.064046183, 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.198439157, 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138707+0000", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138707+0000", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138716+0000", 2024-09-21T18:07:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4970000000000007e-06 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138883+0000", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000166656 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138913+0000", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0009e-05 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.138934+0000", 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1146e-05 2024-09-21T18:07:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.335072+0000", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.196138224 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.335127+0000", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4661999999999999e-05 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337070+0000", 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019431850000000001 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337146+0000", 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5778000000000005e-05 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:903 4.2 e48/33 4:50d6db3f:::smithi16030472-204:head v 48'99, mlcod=48'94)", 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.140966+0000", 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.061787152000001, 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.196685898, 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.140966+0000", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.140966+0000", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.140974+0000", 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3529999999999995e-06 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.140986+0000", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1634000000000001e-05 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.141012+0000", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6202000000000001e-05 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.141035+0000", 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2657e-05 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336284+0000", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.195248792 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336335+0000", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1822e-05 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337576+0000", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012402299999999999 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337652+0000", 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6207999999999999e-05 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1068 4.0 e49/33 4:02c3e29a:::smithi16030472-237:head v 49'81, mlcod=48'78)", 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 10.226456598, 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20186737099999999, 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976313+0000", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6697e-05 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.977143+0000", 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000829476 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.977165+0000", 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2673000000000001e-05 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077321+0000", 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10015566100000001 2024-09-21T18:07:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077479+0000", 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015782700000000001 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077516+0000", 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7029000000000001e-05 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.178099+0000", 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10058284000000001 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.178164+0000", 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5167999999999997e-05 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:294 4.2 e49/33 4:52980938:::smithi16030472-250:head v 49'108, mlcod=49'105)", 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:27.164923+0000", 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.0378295709999996, 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19649957300000001, 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.164923+0000", 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.164923+0000", 2024-09-21T18:07:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.164926+0000", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.166481+0000", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015545820000000001 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.166508+0000", 2024-09-21T18:07:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7013e-05 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.166523+0000", 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4926000000000001e-05 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.360080+0000", 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19355691999999999 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.360095+0000", 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5500999999999999e-05 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.361384+0000", 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001288974 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:27.361423+0000", 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8850000000000002e-05 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1524 4.0 e51/33 4:3d0b4c2b:::smithi16030472-340:head v 51'114, mlcod=50'111)", 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.9292562970000002, 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21167023200000001, 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273500+0000", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8830000000000002e-06 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274509+0000", 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001009427 2024-09-21T18:07:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274536+0000", 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6962e-05 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274550+0000", 2024-09-21T18:07:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4066999999999999e-05 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274686+0000", 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000136243 2024-09-21T18:07:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274722+0000", 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5639000000000002e-05 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485101+0000", 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21037930399999999 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485167+0000", 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5707e-05 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1525 4.0 e51/33 4:3d0b4c2b:::smithi16030472-340:head v 51'115, mlcod=50'111)", 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.927074374, 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20957785800000001, 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275681+0000", 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5799999999999999e-06 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276882+0000", 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012012189999999999 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276913+0000", 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0877999999999998e-05 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276937+0000", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3532000000000001e-05 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.377834+0000", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100897295 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.377870+0000", 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6057999999999999e-05 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485176+0000", 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10730627 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485257+0000", 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0025999999999996e-05 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:407 4.0 e51/33 4:0382d963:::smithi16030472-337:head v 51'116, mlcod=50'112)", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.9244527299999996, 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207031938, 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278303+0000", 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8229999999999999e-06 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283172+0000", 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048691050000000003 2024-09-21T18:07:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283198+0000", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6225e-05 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283213+0000", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4742999999999999e-05 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.480047+0000", 2024-09-21T18:07:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19683339 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.480080+0000", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3881999999999998e-05 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485263+0000", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0051822600000000002 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485332+0000", 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9510000000000004e-05 2024-09-21T18:07:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1511 4.0 e51/33 4:0382d963:::smithi16030472-337:head v 51'117, mlcod=50'112)", 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.288967+0000", 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.9137864570000005, 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197079105, 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.288967+0000", 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.288967+0000", 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.288969+0000", 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7860000000000001e-06 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.288981+0000", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1676e-05 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289007+0000", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6282e-05 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289026+0000", 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9159000000000001e-05 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.484337+0000", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19531011400000001 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.484369+0000", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2372000000000001e-05 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485978+0000", 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001609299 2024-09-21T18:07:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.486046+0000", 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7416999999999995e-05 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1526 4.0 e51/33 4:3d0b4c2b:::smithi16030472-340:head v 51'118, mlcod=50'112)", 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.289548+0000", 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.9132050940000003, 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19654112400000001, 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289548+0000", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289548+0000", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289551+0000", 2024-09-21T18:07:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8629999999999999e-06 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289562+0000", 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1413000000000001e-05 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289589+0000", 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6642e-05 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.289602+0000", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3473e-05 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.484760+0000", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19515771000000001 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.484793+0000", 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2954999999999999e-05 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.486049+0000", 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001256456 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.486089+0000", 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9612e-05 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.606951875, 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1535 4.2 e51/33 4:7b415eca:::smithi16030472-342:head v 51'119, mlcod=50'116)", 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.4622335, 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20316891400000001, 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740523+0000", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9859999999999999e-06 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741028+0000", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050500000000000002 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741056+0000", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8466000000000002e-05 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741075+0000", 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8739000000000001e-05 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.841326+0000", 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1002511 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.841362+0000", 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5744999999999997e-05 2024-09-21T18:07:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943604+0000", 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10224288500000001 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943688+0000", 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3992999999999994e-05 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1536 4.2 e51/33 4:7b415eca:::smithi16030472-342:head v 51'120, mlcod=50'116)", 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.4600954110000002, 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20178523700000001, 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742660+0000", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8159999999999998e-06 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742672+0000", 2024-09-21T18:07:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1535999999999999e-05 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742700+0000", 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7606e-05 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742718+0000", 2024-09-21T18:07:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8629000000000001e-05 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943001+0000", 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20028306900000001 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943064+0000", 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2343999999999998e-05 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.944327+0000", 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001263602 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.944443+0000", 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115635 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:448 4.0 e51/33 4:385a7674:::smithi16030472-358:head v 51'136, mlcod=51'134)", 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 3.320205369, 2024-09-21T18:07:36.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20784413099999999, 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:36.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882551+0000", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3170000000000002e-06 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.885156+0000", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026050969999999998 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.885184+0000", 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8348e-05 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.985325+0000", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014103000000001 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.085601+0000", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100275138 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.085657+0000", 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6705e-05 2024-09-21T18:07:36.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.090301+0000", 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046439000000000003 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.090392+0000", 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0595999999999998e-05 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:36.295 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:36.298 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:36.293+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:36.298 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:07:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:07:36.300 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:36.301 INFO:tasks.rados.rados.0.smithi160.stdout:440: writing smithi16030472-440 from 578273 to 1044705 tid 1 2024-09-21T18:07:36.301 INFO:tasks.rados.rados.0.smithi160.stdout:440: writing smithi16030472-440 from 1739595 to 2192410 tid 2 2024-09-21T18:07:36.301 INFO:tasks.rados.rados.0.smithi160.stdout:440: writing smithi16030472-440 from 2722929 to 3378273 tid 3 2024-09-21T18:07:36.301 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: finishing write tid 4 to smithi16030472-417 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: oid 417 updating version 0 to 291 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: oid 417 version 291 is already newer than 290 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: oid 417 version 291 is already newer than 289 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 291 (ObjNum 416 snap 0 seq_num 416) dirty exists 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: left oid 417 (ObjNum 416 snap 0 seq_num 416) 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:417: done (15 left) 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:441: write initial oid 441 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:441: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:441: writing smithi16030472-441 from 595080 to 1136761 tid 1 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:441: writing smithi16030472-441 from 1798077 to 2394550 tid 2 2024-09-21T18:07:36.302 INFO:tasks.rados.rados.0.smithi160.stdout:441: writing smithi16030472-441 from 3150352 to 3395080 tid 3 2024-09-21T18:07:36.303 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.303 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-21T18:07:35.063666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1340250799,"num_objects":502,"num_object_clones":0,"num_object_copies":1004,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":467,"num_whiteouts":0,"num_read":941,"num_read_kb":420,"num_write":1733,"num_write_kb":468822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":143,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1491,"ondisk_log_size":1491,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":1656572,"kb_used_data":1195020,"kb_used_omap":24,"kb_used_meta":461223,"kb_avail":1508292868,"statfs":{"total":1546188226560,"available":1544491896832,"internally_reserved":0,"allocated":1223700480,"data_stored":1233930699,"data_compressed":18900499,"data_compressed_allocated":20164608,"data_compressed_original":41885696,"omap_allocated":25520,"internal_metadata":472292432},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,4,15,39,47,6],"upper_bound":2048},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":492032267,"num_objects":169,"num_object_clones":0,"num_object_copies":338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":308,"num_read_kb":132,"num_write":621,"num_write_kb":184311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":35,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.747909"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":198,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.656912+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:07:30.656912+0000","last_peered":"2024-09-21T18:07:30.656912+0000","last_clean":"2024-09-21T18:07:30.656912+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:07:30.656912+0000","last_undegraded":"2024-09-21T18:07:30.656912+0000","last_fullsized":"2024-09-21T18:07:30.656912+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.556907+0000","last_change":"2024-09-21T18:06:47.097292+0000","last_active":"2024-09-21T18:07:30.556907+0000","last_peered":"2024-09-21T18:07:30.556907+0000","last_clean":"2024-09-21T18:07:30.556907+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:07:30.556907+0000","last_undegraded":"2024-09-21T18:07:30.556907+0000","last_fullsized":"2024-09-21T18:07:30.556907+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:45.676239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"51'239","reported_seq":425,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:31.933951+0000","last_change":"2024-09-21T18:07:01.287376+0000","last_active":"2024-09-21T18:07:31.933951+0000","last_peered":"2024-09-21T18:07:31.933951+0000","last_clean":"2024-09-21T18:07:31.933951+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:07:31.933951+0000","last_undegraded":"2024-09-21T18:07:31.933951+0000","last_fullsized":"2024-09-21T18:07:31.933951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:21.792563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221671757,"num_objects":90,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":0,"num_read":183,"num_read_kb":47,"num_write":291,"num_write_kb":101666,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.765808+0000","last_change":"2024-09-21T18:06:46.889674+0000","last_active":"2024-09-21T18:07:30.765808+0000","last_peered":"2024-09-21T18:07:30.765808+0000","last_clean":"2024-09-21T18:07:30.765808+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:07:30.765808+0000","last_undegraded":"2024-09-21T18:07:30.765808+0000","last_fullsized":"2024-09-21T18:07:30.765808+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:53.969734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"50'277","reported_seq":428,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.301354+0000","last_change":"2024-09-21T18:07:01.490023+0000","last_active":"2024-09-21T18:07:30.301354+0000","last_peered":"2024-09-21T18:07:30.301354+0000","last_clean":"2024-09-21T18:07:30.301354+0000","last_became_active":"2024-09-21T18:07:01.389573+0000","last_became_peered":"2024-09-21T18:07:01.389573+0000","last_unstale":"2024-09-21T18:07:30.301354+0000","last_undegraded":"2024-09-21T18:07:30.301354+0000","last_fullsized":"2024-09-21T18:07:30.301354+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:03.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240093121,"num_objects":84,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":170,"num_read_kb":64,"num_write":348,"num_write_kb":107748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.754149+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:07:30.754149+0000","last_peered":"2024-09-21T18:07:30.754149+0000","last_clean":"2024-09-21T18:07:30.754149+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:07:30.754149+0000","last_undegraded":"2024-09-21T18:07:30.754149+0000","last_fullsized":"2024-09-21T18:07:30.754149+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"51'256","reported_seq":468,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.680056+0000","last_change":"2024-09-21T18:07:01.405525+0000","last_active":"2024-09-21T18:07:32.680056+0000","last_peered":"2024-09-21T18:07:32.680056+0000","last_clean":"2024-09-21T18:07:32.680056+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:07:32.680056+0000","last_undegraded":"2024-09-21T18:07:32.680056+0000","last_fullsized":"2024-09-21T18:07:32.680056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.560432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238557258,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":203,"num_read_kb":49,"num_write":312,"num_write_kb":109170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":67,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.765833+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:07:30.765833+0000","last_peered":"2024-09-21T18:07:30.765833+0000","last_clean":"2024-09-21T18:07:30.765833+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:07:30.765833+0000","last_undegraded":"2024-09-21T18:07:30.765833+0000","last_fullsized":"2024-09-21T18:07:30.765833+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"51'258","reported_seq":424,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.789471+0000","last_change":"2024-09-21T18:07:01.288102+0000","last_active":"2024-09-21T18:07:32.789471+0000","last_peered":"2024-09-21T18:07:32.789471+0000","last_clean":"2024-09-21T18:07:32.789471+0000","last_became_active":"2024-09-21T18:07:01.287689+0000","last_became_peered":"2024-09-21T18:07:01.287689+0000","last_unstale":"2024-09-21T18:07:32.789471+0000","last_undegraded":"2024-09-21T18:07:32.789471+0000","last_fullsized":"2024-09-21T18:07:32.789471+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.216721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230476334,"num_objects":86,"num_object_clones":0,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":0,"num_read":175,"num_read_kb":63,"num_write":324,"num_write_kb":104515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"51'125","reported_seq":218,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:32.578246+0000","last_change":"2024-09-21T18:07:03.553363+0000","last_active":"2024-09-21T18:07:32.578246+0000","last_peered":"2024-09-21T18:07:32.578246+0000","last_clean":"2024-09-21T18:07:32.578246+0000","last_became_active":"2024-09-21T18:07:03.552723+0000","last_became_peered":"2024-09-21T18:07:03.552723+0000","last_unstale":"2024-09-21T18:07:32.578246+0000","last_undegraded":"2024-09-21T18:07:32.578246+0000","last_fullsized":"2024-09-21T18:07:32.578246+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:07.591572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144926101,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":116,"num_write_kb":18050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":64,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:27.330537+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:07:27.330537+0000","last_peered":"2024-09-21T18:07:27.330537+0000","last_clean":"2024-09-21T18:07:27.330537+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:07:27.330537+0000","last_undegraded":"2024-09-21T18:07:27.330537+0000","last_fullsized":"2024-09-21T18:07:27.330537+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"51'65","reported_seq":136,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:31.395928+0000","last_change":"2024-09-21T18:07:04.454454+0000","last_active":"2024-09-21T18:07:31.395928+0000","last_peered":"2024-09-21T18:07:31.395928+0000","last_clean":"2024-09-21T18:07:31.395928+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:07:31.395928+0000","last_undegraded":"2024-09-21T18:07:31.395928+0000","last_fullsized":"2024-09-21T18:07:31.395928+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.533440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69345187,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":59,"num_write_kb":8354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":26,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":64,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.712137+0000","last_change":"2024-09-21T18:06:46.980854+0000","last_active":"2024-09-21T18:07:30.712137+0000","last_peered":"2024-09-21T18:07:30.712137+0000","last_clean":"2024-09-21T18:07:30.712137+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:07:30.712137+0000","last_undegraded":"2024-09-21T18:07:30.712137+0000","last_fullsized":"2024-09-21T18:07:30.712137+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:47.695849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"50'107","reported_seq":196,"reported_epoch":50,"state":"active+clean","last_fresh":"2024-09-21T18:07:29.989895+0000","last_change":"2024-09-21T18:07:03.453942+0000","last_active":"2024-09-21T18:07:29.989895+0000","last_peered":"2024-09-21T18:07:29.989895+0000","last_clean":"2024-09-21T18:07:29.989895+0000","last_became_active":"2024-09-21T18:07:03.453360+0000","last_became_peered":"2024-09-21T18:07:03.453360+0000","last_unstale":"2024-09-21T18:07:29.989895+0000","last_undegraded":"2024-09-21T18:07:29.989895+0000","last_fullsized":"2024-09-21T18:07:29.989895+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:14.477508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127648703,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":0,"num_read":39,"num_read_kb":39,"num_write":96,"num_write_kb":11819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":65,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.846739+0000","last_change":"2024-09-21T18:06:47.325299+0000","last_active":"2024-09-21T18:07:30.846739+0000","last_peered":"2024-09-21T18:07:30.846739+0000","last_clean":"2024-09-21T18:07:30.846739+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:07:30.846739+0000","last_undegraded":"2024-09-21T18:07:30.846739+0000","last_fullsized":"2024-09-21T18:07:30.846739+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:01.215609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"51'49","reported_seq":116,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.872340+0000","last_change":"2024-09-21T18:07:04.220636+0000","last_active":"2024-09-21T18:07:30.872340+0000","last_peered":"2024-09-21T18:07:30.872340+0000","last_clean":"2024-09-21T18:07:30.872340+0000","last_became_active":"2024-09-21T18:07:04.220350+0000","last_became_peered":"2024-09-21T18:07:04.220350+0000","last_unstale":"2024-09-21T18:07:30.872340+0000","last_undegraded":"2024-09-21T18:07:30.872340+0000","last_fullsized":"2024-09-21T18:07:30.872340+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.012491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66941951,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":19,"num_read_kb":19,"num_write":40,"num_write_kb":4826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":23,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":63,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-09-21T18:07:30.735407+0000","last_change":"2024-09-21T18:06:47.325330+0000","last_active":"2024-09-21T18:07:30.735407+0000","last_peered":"2024-09-21T18:07:30.735407+0000","last_clean":"2024-09-21T18:07:30.735407+0000","last_became_active":"2024-09-21T18:06:45.022009+0000","last_became_peered":"2024-09-21T18:06:45.022009+0000","last_unstale":"2024-09-21T18:07:30.735407+0000","last_undegraded":"2024-09-21T18:07:30.735407+0000","last_fullsized":"2024-09-21T18:07:30.735407+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:07:44.577427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":408861942,"num_objects":141,"num_object_clones":0,"num_object_copies":282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":106,"num_whiteouts":0,"num_read":124,"num_read_kb":124,"num_write":311,"num_write_kb":43049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":143,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":383578112,"data_stored":403440059,"data_compressed":19141322,"data_compressed_allocated":20418560,"data_compressed_original":41713664,"omap_allocated":0,"internal_metadata":0},"log_size":346,"ondisk_log_size":346,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":930798470,"num_objects":357,"num_object_clones":0,"num_object_copies":714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":0,"num_read":731,"num_read_kb":223,"num_write":1275,"num_write_kb":423099,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":876269568,"data_stored":869745230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1030,"ondisk_log_size":1030,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":201188,"kb_used_data":170004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94170652,"statfs":{"total":96636764160,"available":96430747648,"internally_reserved":0,"allocated":174084096,"data_stored":176664026,"data_compressed":2787437,"data_compressed_allocated":2973696,"data_compressed_original":6856704,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,7,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,7,4,6,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27544,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344296,"statfs":{"total":96636764160,"available":96608559104,"internally_reserved":0,"allocated":528384,"data_stored":273298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247821,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":145344,"kb_used_data":114160,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94226496,"statfs":{"total":96636764160,"available":96487931904,"internally_reserved":0,"allocated":116899840,"data_stored":115785717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,0,4,19,3],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313229,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":188456,"kb_used_data":157272,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94183384,"statfs":{"total":96636764160,"available":96443785216,"internally_reserved":0,"allocated":161046528,"data_stored":162770537,"data_compressed":3018161,"data_compressed_allocated":3223552,"data_compressed_original":6160384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,3,5,6,7,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,1,0,7,23,3],"upper_bound":2048},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313230,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31660,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340180,"statfs":{"total":96636764160,"available":96604344320,"internally_reserved":0,"allocated":483328,"data_stored":273634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,7,8,9,10,11,12,13],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":278804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,7,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280524,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159400,"kb_used_data":128216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94212440,"statfs":{"total":96636764160,"available":96473538560,"internally_reserved":0,"allocated":131293184,"data_stored":132290484,"data_compressed":2028526,"data_compressed_allocated":2158592,"data_compressed_original":4235264,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,3,5,6,7,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280524,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27472,"kb_used_data":444,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":454656,"data_stored":268813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280525,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152948,"kb_used_data":125920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94218892,"statfs":{"total":96636764160,"available":96480145408,"internally_reserved":0,"allocated":128942080,"data_stored":129822175,"data_compressed":1597141,"data_compressed_allocated":1703936,"data_compressed_original":3657728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,7,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":524288,"data_stored":276231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,2,3,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27548,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":532480,"data_stored":278827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126408,"kb_used_data":99380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94245432,"statfs":{"total":96636764160,"available":96507322368,"internally_reserved":0,"allocated":101765120,"data_stored":100809589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,7,8,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,3,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280524,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":151676,"kb_used_data":120488,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94220164,"statfs":{"total":96636764160,"available":96481447936,"internally_reserved":0,"allocated":123379712,"data_stored":129505553,"data_compressed":5843567,"data_compressed_allocated":6242304,"data_compressed_original":13082624,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[1,3,5,7,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280525,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":193068,"kb_used_data":161880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94178772,"statfs":{"total":96636764160,"available":96439062528,"internally_reserved":0,"allocated":165765120,"data_stored":168572691,"data_compressed":3625667,"data_compressed_allocated":3862528,"data_compressed_original":7892992,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4,8,15],"upper_bound":1024},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27540,"kb_used_data":512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344300,"statfs":{"total":96636764160,"available":96608563200,"internally_reserved":0,"allocated":524288,"data_stored":275647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,7,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280525,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":141232,"kb_used_data":114204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94230608,"statfs":{"total":96636764160,"available":96492142592,"internally_reserved":0,"allocated":116944896,"data_stored":115784673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,7,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116363264,"data_stored":115509287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":104099840,"data_stored":103347866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":111099904,"data_stored":110233364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":103829504,"data_stored":103063956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":107679744,"data_stored":106879522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":104775680,"data_stored":104006735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":119779328,"data_stored":118851931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":108642304,"data_stored":107852569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":31715328,"data_stored":33685248,"data_compressed":2028526,"data_compressed_allocated":2158592,"data_compressed_original":4235264,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":59297792,"data_stored":62188824,"data_compressed":3171769,"data_compressed_allocated":3391488,"data_compressed_original":6488064,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":32071680,"data_stored":33433696,"data_compressed":1564354,"data_compressed_allocated":1662976,"data_compressed_original":3145728,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":130011136,"data_stored":136718440,"data_compressed":5996356,"data_compressed_allocated":6410240,"data_compressed_original":13606912,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":63787008,"data_stored":67118944,"data_compressed":3592880,"data_compressed_allocated":3821568,"data_compressed_original":7380992,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":66695168,"data_stored":70294907,"data_compressed":2787437,"data_compressed_allocated":2973696,"data_compressed_original":6856704,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:07:36.305 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T18:07:36.323 INFO:tasks.rados.rados.0.smithi160.stdout:437: finishing write tid 1 to smithi16030472-437 2024-09-21T18:07:36.323 INFO:tasks.rados.rados.0.smithi160.stdout:437: finishing write tid 2 to smithi16030472-437 2024-09-21T18:07:36.331 INFO:tasks.rados.rados.0.smithi160.stdout:437: finishing write tid 3 to smithi16030472-437 2024-09-21T18:07:36.331 INFO:tasks.rados.rados.0.smithi160.stdout:437: finishing write tid 4 to smithi16030472-437 2024-09-21T18:07:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:07:36.365 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:36.358+0000 7f728550b640 -1 osd.14 52 handle_osd_map injecting map crc failure 2024-09-21T18:07:36.398 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:36.392+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:36.429 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:36.429 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:36.432 INFO:tasks.rados.rados.0.smithi160.stdout:437: finishing write tid 5 to smithi16030472-437 2024-09-21T18:07:36.433 INFO:tasks.rados.rados.0.smithi160.stdout:437: oid 437 updating version 0 to 311 2024-09-21T18:07:36.433 INFO:tasks.rados.rados.0.smithi160.stdout:437: oid 437 updating version 311 to 312 2024-09-21T18:07:36.433 INFO:tasks.rados.rados.0.smithi160.stdout:437: oid 437 updating version 312 to 313 2024-09-21T18:07:36.433 INFO:tasks.rados.rados.0.smithi160.stdout:437: oid 437 version 313 is already newer than 310 2024-09-21T18:07:36.433 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 437 v 313 (ObjNum 436 snap 0 seq_num 436) dirty exists 2024-09-21T18:07:36.434 INFO:tasks.rados.rados.0.smithi160.stdout:437: left oid 437 (ObjNum 436 snap 0 seq_num 436) 2024-09-21T18:07:36.434 INFO:tasks.rados.rados.0.smithi160.stdout:437: done (15 left) 2024-09-21T18:07:36.434 INFO:tasks.rados.rados.0.smithi160.stdout:442: write initial oid 442 2024-09-21T18:07:36.434 INFO:tasks.rados.rados.0.smithi160.stdout:442: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-21T18:07:36.440 INFO:tasks.rados.rados.0.smithi160.stdout:442: writing smithi16030472-442 from 611887 to 1345170 tid 1 2024-09-21T18:07:36.444 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:36.446 INFO:tasks.rados.rados.0.smithi160.stdout:442: writing smithi16030472-442 from 1972912 to 2429396 tid 2 2024-09-21T18:07:36.451 INFO:tasks.rados.rados.0.smithi160.stdout:442: writing smithi16030472-442 from 2894128 to 3411887 tid 3 2024-09-21T18:07:36.451 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.499 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:36.494+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:36.569 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:36.569 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:36.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:36.599 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:36.597+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:36.632 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:36.626+0000 7fca4e93e640 -1 osd.2 52 handle_osd_map injecting map crc failure 2024-09-21T18:07:36.700 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:36.694+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:36.707 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:36.707 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:36.722 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:36.801 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:36.796+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:430: finishing write tid 4 to smithi16030472-430 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: finishing write tid 4 to smithi16030472-426 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: finishing write tid 5 to smithi16030472-426 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: oid 426 updating version 0 to 344 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: oid 426 version 344 is already newer than 343 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: oid 426 updating version 344 to 345 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: oid 426 updating version 345 to 346 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 426 v 346 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:426: left oid 426 (ObjNum 425 snap 0 seq_num 425) 2024-09-21T18:07:36.838 INFO:tasks.rados.rados.0.smithi160.stdout:427: finishing write tid 1 to smithi16030472-427 2024-09-21T18:07:36.839 INFO:tasks.rados.rados.0.smithi160.stdout:426: done (15 left) 2024-09-21T18:07:36.839 INFO:tasks.rados.rados.0.smithi160.stdout:443: write initial oid 443 2024-09-21T18:07:36.839 INFO:tasks.rados.rados.0.smithi160.stdout:443: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-21T18:07:36.844 INFO:tasks.rados.rados.0.smithi160.stdout:443: writing smithi16030472-443 from 628694 to 1037226 tid 1 2024-09-21T18:07:36.851 INFO:tasks.rados.rados.0.smithi160.stdout:443: writing smithi16030472-443 from 1515041 to 2115183 tid 2 2024-09-21T18:07:36.858 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:36.858 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:36.858 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:36.860 INFO:tasks.rados.rados.0.smithi160.stdout:443: writing smithi16030472-443 from 2805198 to 3357310 tid 3 2024-09-21T18:07:36.860 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.860 INFO:tasks.rados.rados.0.smithi160.stdout:427: finishing write tid 2 to smithi16030472-427 2024-09-21T18:07:36.860 INFO:tasks.rados.rados.0.smithi160.stdout:427: finishing write tid 3 to smithi16030472-427 2024-09-21T18:07:36.860 INFO:tasks.rados.rados.0.smithi160.stdout:427: finishing write tid 4 to smithi16030472-427 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:427: oid 427 updating version 0 to 348 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:427: oid 427 version 348 is already newer than 347 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:427: oid 427 updating version 348 to 349 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 349 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:427: left oid 427 (ObjNum 426 snap 0 seq_num 426) 2024-09-21T18:07:36.861 INFO:tasks.rados.rados.0.smithi160.stdout:428: finishing write tid 1 to smithi16030472-428 2024-09-21T18:07:36.862 INFO:tasks.rados.rados.0.smithi160.stdout:428: finishing write tid 2 to smithi16030472-428 2024-09-21T18:07:36.862 INFO:tasks.rados.rados.0.smithi160.stdout:427: done (15 left) 2024-09-21T18:07:36.862 INFO:tasks.rados.rados.0.smithi160.stdout:444: write initial oid 444 2024-09-21T18:07:36.862 INFO:tasks.rados.rados.0.smithi160.stdout:444: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-21T18:07:36.863 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:36.863 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:36.865 INFO:tasks.rados.rados.0.smithi160.stdout:444: writing smithi16030472-444 from 645501 to 1129282 tid 1 2024-09-21T18:07:36.871 INFO:tasks.rados.rados.0.smithi160.stdout:444: writing smithi16030472-444 from 1573523 to 2033676 tid 2 2024-09-21T18:07:36.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:36.879 INFO:tasks.rados.rados.0.smithi160.stdout:444: writing smithi16030472-444 from 2548974 to 3312358 tid 3 2024-09-21T18:07:36.879 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:428: finishing write tid 3 to smithi16030472-428 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:430: finishing write tid 5 to smithi16030472-430 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:430: oid 430 updating version 0 to 299 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:430: oid 430 version 299 is already newer than 297 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:430: oid 430 version 299 is already newer than 296 2024-09-21T18:07:36.880 INFO:tasks.rados.rados.0.smithi160.stdout:430: oid 430 version 299 is already newer than 298 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 430 v 299 (ObjNum 429 snap 0 seq_num 429) dirty exists 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:430: left oid 430 (ObjNum 429 snap 0 seq_num 429) 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:428: finishing write tid 4 to smithi16030472-428 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:428: finishing write tid 5 to smithi16030472-428 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:428: oid 428 updating version 0 to 353 2024-09-21T18:07:36.881 INFO:tasks.rados.rados.0.smithi160.stdout:428: oid 428 version 353 is already newer than 352 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:428: oid 428 version 353 is already newer than 350 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:428: oid 428 version 353 is already newer than 351 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 428 v 353 (ObjNum 427 snap 0 seq_num 427) dirty exists 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:428: left oid 428 (ObjNum 427 snap 0 seq_num 427) 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:429: finishing write tid 1 to smithi16030472-429 2024-09-21T18:07:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:429: finishing write tid 2 to smithi16030472-429 2024-09-21T18:07:36.883 INFO:tasks.rados.rados.0.smithi160.stdout:429: finishing write tid 3 to smithi16030472-429 2024-09-21T18:07:36.883 INFO:tasks.rados.rados.0.smithi160.stdout:428: done (15 left) 2024-09-21T18:07:36.883 INFO:tasks.rados.rados.0.smithi160.stdout:430: done (14 left) 2024-09-21T18:07:36.883 INFO:tasks.rados.rados.0.smithi160.stdout:445: write initial oid 445 2024-09-21T18:07:36.883 INFO:tasks.rados.rados.0.smithi160.stdout:445: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-21T18:07:36.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:36.884 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:36.886 INFO:tasks.rados.rados.0.smithi160.stdout:445: writing smithi16030472-445 from 662308 to 1221338 tid 1 2024-09-21T18:07:36.892 INFO:tasks.rados.rados.0.smithi160.stdout:445: writing smithi16030472-445 from 1632005 to 2235816 tid 2 2024-09-21T18:07:36.899 INFO:tasks.rados.rados.0.smithi160.stdout:445: writing smithi16030472-445 from 2860044 to 3462308 tid 3 2024-09-21T18:07:36.899 INFO:tasks.rados.rados.0.smithi160.stdout:446: write initial oid 446 2024-09-21T18:07:36.900 INFO:tasks.rados.rados.0.smithi160.stdout:446: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-21T18:07:36.902 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:36.897+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:36.905 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:36.907 INFO:tasks.rados.rados.0.smithi160.stdout:446: writing smithi16030472-446 from 679115 to 1313394 tid 1 2024-09-21T18:07:36.913 INFO:tasks.rados.rados.0.smithi160.stdout:446: writing smithi16030472-446 from 2090487 to 2554309 tid 2 2024-09-21T18:07:36.919 INFO:tasks.rados.rados.0.smithi160.stdout:446: writing smithi16030472-446 from 3003820 to 3479115 tid 3 2024-09-21T18:07:36.919 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.919 INFO:tasks.rados.rados.0.smithi160.stdout:431: finishing write tid 1 to smithi16030472-431 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: finishing write tid 2 to smithi16030472-431 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: finishing write tid 3 to smithi16030472-431 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: finishing write tid 4 to smithi16030472-431 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:434: finishing write tid 1 to smithi16030472-434 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:434: finishing write tid 2 to smithi16030472-434 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: finishing write tid 5 to smithi16030472-431 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: oid 431 updating version 0 to 357 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: oid 431 updating version 357 to 358 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: oid 431 updating version 358 to 360 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: oid 431 version 360 is already newer than 359 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 431 v 360 (ObjNum 430 snap 0 seq_num 430) dirty exists 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:431: left oid 431 (ObjNum 430 snap 0 seq_num 430) 2024-09-21T18:07:36.920 INFO:tasks.rados.rados.0.smithi160.stdout:432: finishing write tid 1 to smithi16030472-432 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:432: finishing write tid 2 to smithi16030472-432 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:432: finishing write tid 3 to smithi16030472-432 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: finishing write tid 4 to smithi16030472-429 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: oid 429 updating version 0 to 356 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: oid 429 version 356 is already newer than 355 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: oid 429 version 356 is already newer than 354 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 356 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: left oid 429 (ObjNum 428 snap 0 seq_num 428) 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:434: finishing write tid 3 to smithi16030472-434 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:432: finishing write tid 4 to smithi16030472-432 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:429: done (15 left) 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:431: done (14 left) 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:447: write initial oid 447 2024-09-21T18:07:36.921 INFO:tasks.rados.rados.0.smithi160.stdout:447: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-21T18:07:36.927 INFO:tasks.rados.rados.0.smithi160.stdout:447: writing smithi16030472-447 from 695922 to 1405450 tid 1 2024-09-21T18:07:36.932 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:36.932 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:36.933 INFO:tasks.rados.rados.0.smithi160.stdout:447: writing smithi16030472-447 from 2032616 to 2640096 tid 2 2024-09-21T18:07:36.939 INFO:tasks.rados.rados.0.smithi160.stdout:447: writing smithi16030472-447 from 3198537 to 3495922 tid 3 2024-09-21T18:07:36.939 INFO:tasks.rados.rados.0.smithi160.stdout:448: write initial oid 448 2024-09-21T18:07:36.939 INFO:tasks.rados.rados.0.smithi160.stdout:448: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-21T18:07:36.947 INFO:tasks.rados.rados.0.smithi160.stdout:448: writing smithi16030472-448 from 712729 to 1497506 tid 1 2024-09-21T18:07:36.953 INFO:tasks.rados.rados.0.smithi160.stdout:448: writing smithi16030472-448 from 2091098 to 2558589 tid 2 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:448: writing smithi16030472-448 from 3342313 to 3512729 tid 3 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:433: finishing write tid 1 to smithi16030472-433 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: finishing write tid 5 to smithi16030472-432 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: oid 432 updating version 0 to 361 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: oid 432 updating version 361 to 364 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: oid 432 version 364 is already newer than 362 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: oid 432 version 364 is already newer than 363 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 432 v 364 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: left oid 432 (ObjNum 431 snap 0 seq_num 431) 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:433: finishing write tid 2 to smithi16030472-433 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:432: done (15 left) 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:449: write initial oid 449 2024-09-21T18:07:36.959 INFO:tasks.rados.rados.0.smithi160.stdout:449: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-21T18:07:36.965 INFO:tasks.rados.rados.0.smithi160.stdout:449: writing smithi16030472-449 from 729536 to 1189562 tid 1 2024-09-21T18:07:36.972 INFO:tasks.rados.rados.0.smithi160.stdout:449: writing smithi16030472-449 from 1749580 to 2360729 tid 2 2024-09-21T18:07:36.977 INFO:tasks.rados.rados.0.smithi160.stdout:449: writing smithi16030472-449 from 2853383 to 3305833 tid 3 2024-09-21T18:07:36.977 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:36.991 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:36.992 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:36.992 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:36.992 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:36.992 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.004 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:37.000+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:37.006 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:37.064 INFO:tasks.rados.rados.0.smithi160.stdout:446: finishing write tid 1 to smithi16030472-446 2024-09-21T18:07:37.081 INFO:tasks.rados.rados.0.smithi160.stdout:434: finishing write tid 4 to smithi16030472-434 2024-09-21T18:07:37.081 INFO:tasks.rados.rados.0.smithi160.stdout:434: finishing write tid 5 to smithi16030472-434 2024-09-21T18:07:37.081 INFO:tasks.rados.rados.0.smithi160.stdout:434: oid 434 updating version 0 to 303 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:434: oid 434 version 303 is already newer than 301 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:434: oid 434 version 303 is already newer than 300 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:434: oid 434 version 303 is already newer than 302 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 434 v 303 (ObjNum 433 snap 0 seq_num 433) dirty exists 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:434: left oid 434 (ObjNum 433 snap 0 seq_num 433) 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:438: finishing write tid 1 to smithi16030472-438 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:434: done (15 left) 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:450: write initial oid 450 2024-09-21T18:07:37.082 INFO:tasks.rados.rados.0.smithi160.stdout:450: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-21T18:07:37.089 INFO:tasks.rados.rados.0.smithi160.stdout:450: writing smithi16030472-450 from 746343 to 1397971 tid 1 2024-09-21T18:07:37.096 INFO:tasks.rados.rados.0.smithi160.stdout:450: writing smithi16030472-450 from 1924415 to 2679222 tid 2 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:450: writing smithi16030472-450 from 3397159 to 3546343 tid 3 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: finishing write tid 1 to smithi16030472-435 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: finishing write tid 2 to smithi16030472-435 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: finishing write tid 3 to smithi16030472-435 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: finishing write tid 4 to smithi16030472-435 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: finishing write tid 5 to smithi16030472-435 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: oid 435 updating version 0 to 368 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: oid 435 version 368 is already newer than 367 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: oid 435 updating version 368 to 369 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:435: oid 435 updating version 369 to 370 2024-09-21T18:07:37.102 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 435 v 370 (ObjNum 434 snap 0 seq_num 434) dirty exists 2024-09-21T18:07:37.103 INFO:tasks.rados.rados.0.smithi160.stdout:435: left oid 435 (ObjNum 434 snap 0 seq_num 434) 2024-09-21T18:07:37.103 INFO:tasks.rados.rados.0.smithi160.stdout:439: finishing write tid 1 to smithi16030472-439 2024-09-21T18:07:37.103 INFO:tasks.rados.rados.0.smithi160.stdout:435: done (15 left) 2024-09-21T18:07:37.103 INFO:tasks.rados.rados.0.smithi160.stdout:451: write initial oid 451 2024-09-21T18:07:37.103 INFO:tasks.rados.rados.0.smithi160.stdout:451: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-21T18:07:37.104 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:37.100+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:37.109 INFO:tasks.rados.rados.0.smithi160.stdout:451: writing smithi16030472-451 from 763150 to 1490027 tid 1 2024-09-21T18:07:37.116 INFO:tasks.rados.rados.0.smithi160.stdout:451: writing smithi16030472-451 from 1982897 to 2597715 tid 2 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:451: writing smithi16030472-451 from 3024582 to 3499576 tid 3 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: finishing write tid 2 to smithi16030472-439 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: finishing write tid 3 to smithi16030472-439 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: finishing write tid 4 to smithi16030472-439 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: finishing write tid 5 to smithi16030472-439 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: oid 439 updating version 0 to 373 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: oid 439 version 373 is already newer than 372 2024-09-21T18:07:37.121 INFO:tasks.rados.rados.0.smithi160.stdout:439: oid 439 updating version 373 to 374 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:439: oid 439 version 374 is already newer than 371 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 439 v 374 (ObjNum 438 snap 0 seq_num 438) dirty exists 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:439: left oid 439 (ObjNum 438 snap 0 seq_num 438) 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:443: finishing write tid 1 to smithi16030472-443 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:433: finishing write tid 3 to smithi16030472-433 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:433: finishing write tid 4 to smithi16030472-433 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:439: done (15 left) 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:452: write initial oid 452 2024-09-21T18:07:37.122 INFO:tasks.rados.rados.0.smithi160.stdout:452: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-21T18:07:37.127 INFO:tasks.rados.rados.0.smithi160.stdout:452: writing smithi16030472-452 from 779957 to 1182083 tid 1 2024-09-21T18:07:37.136 INFO:tasks.rados.rados.0.smithi160.stdout:452: writing smithi16030472-452 from 1925026 to 2683502 tid 2 2024-09-21T18:07:37.141 INFO:tasks.rados.rados.0.smithi160.stdout:452: writing smithi16030472-452 from 3335652 to 3579957 tid 3 2024-09-21T18:07:37.141 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.175 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:37.175 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:37.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:37.176 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:37.176 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.181 INFO:tasks.rados.rados.0.smithi160.stdout:433: finishing write tid 5 to smithi16030472-433 2024-09-21T18:07:37.181 INFO:tasks.rados.rados.0.smithi160.stdout:433: oid 433 updating version 0 to 366 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:433: oid 433 updating version 366 to 368 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:433: oid 433 version 368 is already newer than 365 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:433: oid 433 version 368 is already newer than 367 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 368 (ObjNum 432 snap 0 seq_num 432) dirty exists 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:433: left oid 433 (ObjNum 432 snap 0 seq_num 432) 2024-09-21T18:07:37.182 INFO:tasks.rados.rados.0.smithi160.stdout:443: finishing write tid 2 to smithi16030472-443 2024-09-21T18:07:37.183 INFO:tasks.rados.rados.0.smithi160.stdout:433: done (15 left) 2024-09-21T18:07:37.183 INFO:tasks.rados.rados.0.smithi160.stdout:453: write initial oid 453 2024-09-21T18:07:37.183 INFO:tasks.rados.rados.0.smithi160.stdout:443: finishing write tid 3 to smithi16030472-443 2024-09-21T18:07:37.183 INFO:tasks.rados.rados.0.smithi160.stdout:453: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-21T18:07:37.188 INFO:tasks.rados.rados.0.smithi160.stdout:453: writing smithi16030472-453 from 796764 to 1274139 tid 1 2024-09-21T18:07:37.190 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:37.196 INFO:tasks.rados.rados.0.smithi160.stdout:453: writing smithi16030472-453 from 1983508 to 2601995 tid 2 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:453: writing smithi16030472-453 from 3363075 to 3596764 tid 3 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: finishing write tid 4 to smithi16030472-443 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: finishing write tid 5 to smithi16030472-443 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: oid 443 updating version 0 to 376 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: oid 443 updating version 376 to 377 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: oid 443 version 377 is already newer than 375 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:443: oid 443 updating version 377 to 378 2024-09-21T18:07:37.202 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 443 v 378 (ObjNum 442 snap 0 seq_num 442) dirty exists 2024-09-21T18:07:37.203 INFO:tasks.rados.rados.0.smithi160.stdout:443: left oid 443 (ObjNum 442 snap 0 seq_num 442) 2024-09-21T18:07:37.203 INFO:tasks.rados.rados.0.smithi160.stdout:443: done (15 left) 2024-09-21T18:07:37.203 INFO:tasks.rados.rados.0.smithi160.stdout:454: write initial oid 454 2024-09-21T18:07:37.203 INFO:tasks.rados.rados.0.smithi160.stdout:454: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-21T18:07:37.205 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:37.198+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:37.207 INFO:tasks.rados.rados.0.smithi160.stdout:454: writing smithi16030472-454 from 413571 to 966195 tid 1 2024-09-21T18:07:37.216 INFO:tasks.rados.rados.0.smithi160.stdout:454: writing smithi16030472-454 from 1641990 to 2404135 tid 2 2024-09-21T18:07:37.222 INFO:tasks.rados.rados.0.smithi160.stdout:454: writing smithi16030472-454 from 2990498 to 3415661 tid 3 2024-09-21T18:07:37.222 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.243 INFO:tasks.rados.rados.0.smithi160.stdout:438: finishing write tid 2 to smithi16030472-438 2024-09-21T18:07:37.243 INFO:tasks.rados.rados.0.smithi160.stdout:438: finishing write tid 3 to smithi16030472-438 2024-09-21T18:07:37.243 INFO:tasks.rados.rados.0.smithi160.stdout:440: finishing write tid 1 to smithi16030472-440 2024-09-21T18:07:37.243 INFO:tasks.rados.rados.0.smithi160.stdout:440: finishing write tid 2 to smithi16030472-440 2024-09-21T18:07:37.299 INFO:tasks.rados.rados.0.smithi160.stdout:436: finishing write tid 1 to smithi16030472-436 2024-09-21T18:07:37.300 INFO:tasks.rados.rados.0.smithi160.stdout:441: finishing write tid 1 to smithi16030472-441 2024-09-21T18:07:37.300 INFO:tasks.rados.rados.0.smithi160.stdout:441: finishing write tid 2 to smithi16030472-441 2024-09-21T18:07:37.306 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:37.304+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:37.309 INFO:tasks.rados.rados.0.smithi160.stdout:445: finishing write tid 1 to smithi16030472-445 2024-09-21T18:07:37.318 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:37.319 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:37.319 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:37.319 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.326 INFO:tasks.rados.rados.0.smithi160.stdout:450: finishing write tid 1 to smithi16030472-450 2024-09-21T18:07:37.330 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:37.330 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:37.333 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:37.341 INFO:tasks.rados.rados.0.smithi160.stdout:440: finishing write tid 3 to smithi16030472-440 2024-09-21T18:07:37.407 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:37.404+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:37.456 INFO:tasks.rados.rados.0.smithi160.stdout:440: finishing write tid 4 to smithi16030472-440 2024-09-21T18:07:37.457 INFO:tasks.rados.rados.0.smithi160.stdout:440: finishing write tid 5 to smithi16030472-440 2024-09-21T18:07:37.457 INFO:tasks.rados.rados.0.smithi160.stdout:440: oid 440 updating version 0 to 310 2024-09-21T18:07:37.457 INFO:tasks.rados.rados.0.smithi160.stdout:440: oid 440 version 310 is already newer than 309 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:440: oid 440 version 310 is already newer than 307 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:440: oid 440 version 310 is already newer than 308 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 440 v 310 (ObjNum 439 snap 0 seq_num 439) dirty exists 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:440: left oid 440 (ObjNum 439 snap 0 seq_num 439) 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:440: done (15 left) 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:455: write initial oid 455 2024-09-21T18:07:37.458 INFO:tasks.rados.rados.0.smithi160.stdout:455: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-21T18:07:37.463 INFO:tasks.rados.rados.0.smithi160.stdout:455: writing smithi16030472-455 from 430378 to 1058251 tid 1 2024-09-21T18:07:37.465 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:37.466 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:37.466 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:37.466 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:37.466 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.470 INFO:tasks.rados.rados.0.smithi160.stdout:455: writing smithi16030472-455 from 1700472 to 2322628 tid 2 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:455: writing smithi16030472-455 from 3017921 to 3630378 tid 3 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:442: finishing write tid 1 to smithi16030472-442 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:442: finishing write tid 2 to smithi16030472-442 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:442: finishing write tid 3 to smithi16030472-442 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:444: finishing write tid 1 to smithi16030472-444 2024-09-21T18:07:37.478 INFO:tasks.rados.rados.0.smithi160.stdout:444: finishing write tid 2 to smithi16030472-444 2024-09-21T18:07:37.481 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:37.530 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:37.526+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:37.549 INFO:tasks.rados.rados.0.smithi160.stdout:444: finishing write tid 3 to smithi16030472-444 2024-09-21T18:07:37.565 INFO:tasks.rados.rados.0.smithi160.stdout:438: finishing write tid 4 to smithi16030472-438 2024-09-21T18:07:37.565 INFO:tasks.rados.rados.0.smithi160.stdout:438: finishing write tid 5 to smithi16030472-438 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: oid 438 updating version 0 to 304 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: oid 438 updating version 304 to 305 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: oid 438 updating version 305 to 306 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: oid 438 updating version 306 to 307 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 438 v 307 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: left oid 438 (ObjNum 437 snap 0 seq_num 437) 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:438: done (15 left) 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:456: write initial oid 456 2024-09-21T18:07:37.566 INFO:tasks.rados.rados.0.smithi160.stdout:456: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-21T18:07:37.572 INFO:tasks.rados.rados.0.smithi160.stdout:456: writing smithi16030472-456 from 447185 to 1150307 tid 1 2024-09-21T18:07:37.579 INFO:tasks.rados.rados.0.smithi160.stdout:456: writing smithi16030472-456 from 1642601 to 2408415 tid 2 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:456: writing smithi16030472-456 from 2928991 to 3376698 tid 3 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: finishing write tid 4 to smithi16030472-442 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: finishing write tid 5 to smithi16030472-442 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: oid 442 updating version 0 to 314 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: oid 442 version 314 is already newer than 311 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: oid 442 version 314 is already newer than 313 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: oid 442 version 314 is already newer than 312 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 314 (ObjNum 441 snap 0 seq_num 441) dirty exists 2024-09-21T18:07:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:442: left oid 442 (ObjNum 441 snap 0 seq_num 441) 2024-09-21T18:07:37.586 INFO:tasks.rados.rados.0.smithi160.stdout:442: done (15 left) 2024-09-21T18:07:37.586 INFO:tasks.rados.rados.0.smithi160.stdout:457: write initial oid 457 2024-09-21T18:07:37.586 INFO:tasks.rados.rados.0.smithi160.stdout:457: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-21T18:07:37.591 INFO:tasks.rados.rados.0.smithi160.stdout:457: writing smithi16030472-457 from 463992 to 1242363 tid 1 2024-09-21T18:07:37.597 INFO:tasks.rados.rados.0.smithi160.stdout:457: writing smithi16030472-457 from 1701083 to 2326908 tid 2 2024-09-21T18:07:37.600 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:37.600 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:37.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:37.600 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:37.600 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:457: writing smithi16030472-457 from 2956414 to 3615393 tid 3 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:448: finishing write tid 1 to smithi16030472-448 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:448: finishing write tid 2 to smithi16030472-448 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:448: finishing write tid 3 to smithi16030472-448 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:446: finishing write tid 2 to smithi16030472-446 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:446: finishing write tid 3 to smithi16030472-446 2024-09-21T18:07:37.606 INFO:tasks.rados.rados.0.smithi160.stdout:446: finishing write tid 4 to smithi16030472-446 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: finishing write tid 5 to smithi16030472-446 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: oid 446 updating version 0 to 315 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: oid 446 version 315 is already newer than 314 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: oid 446 updating version 315 to 317 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: oid 446 version 317 is already newer than 316 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 317 (ObjNum 445 snap 0 seq_num 445) dirty exists 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: left oid 446 (ObjNum 445 snap 0 seq_num 445) 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: finishing write tid 4 to smithi16030472-448 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: finishing write tid 5 to smithi16030472-448 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: oid 448 updating version 0 to 382 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: oid 448 version 382 is already newer than 380 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: oid 448 version 382 is already newer than 379 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: oid 448 version 382 is already newer than 381 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 382 (ObjNum 447 snap 0 seq_num 447) dirty exists 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: left oid 448 (ObjNum 447 snap 0 seq_num 447) 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:446: done (15 left) 2024-09-21T18:07:37.607 INFO:tasks.rados.rados.0.smithi160.stdout:448: done (14 left) 2024-09-21T18:07:37.608 INFO:tasks.rados.rados.0.smithi160.stdout:458: write initial oid 458 2024-09-21T18:07:37.608 INFO:tasks.rados.rados.0.smithi160.stdout:458: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-21T18:07:37.609 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:37.602+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:37.612 INFO:tasks.rados.rados.0.smithi160.stdout:458: writing smithi16030472-458 from 480799 to 1050772 tid 1 2024-09-21T18:07:37.615 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:37.619 INFO:tasks.rados.rados.0.smithi160.stdout:458: writing smithi16030472-458 from 1475918 to 2245401 tid 2 2024-09-21T18:07:37.626 INFO:tasks.rados.rados.0.smithi160.stdout:458: writing smithi16030472-458 from 2700190 to 3286794 tid 3 2024-09-21T18:07:37.626 INFO:tasks.rados.rados.0.smithi160.stdout:459: write initial oid 459 2024-09-21T18:07:37.626 INFO:tasks.rados.rados.0.smithi160.stdout:459: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-21T18:07:37.632 INFO:tasks.rados.rados.0.smithi160.stdout:459: writing smithi16030472-459 from 497606 to 1142828 tid 1 2024-09-21T18:07:37.640 INFO:tasks.rados.rados.0.smithi160.stdout:459: writing smithi16030472-459 from 1934400 to 2563894 tid 2 2024-09-21T18:07:37.647 INFO:tasks.rados.rados.0.smithi160.stdout:459: writing smithi16030472-459 from 3243966 to 3697606 tid 3 2024-09-21T18:07:37.647 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.647 INFO:tasks.rados.rados.0.smithi160.stdout:453: finishing write tid 1 to smithi16030472-453 2024-09-21T18:07:37.647 INFO:tasks.rados.rados.0.smithi160.stdout:453: finishing write tid 2 to smithi16030472-453 2024-09-21T18:07:37.647 INFO:tasks.rados.rados.0.smithi160.stdout:459: finishing write tid 1 to smithi16030472-459 2024-09-21T18:07:37.710 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:37.704+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:37.727 INFO:tasks.rados.rados.0.smithi160.stdout:453: finishing write tid 3 to smithi16030472-453 2024-09-21T18:07:37.751 INFO:tasks.rados.rados.0.smithi160.stdout:444: finishing write tid 4 to smithi16030472-444 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: finishing write tid 5 to smithi16030472-444 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: oid 444 updating version 0 to 316 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: oid 444 version 316 is already newer than 315 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: oid 444 version 316 is already newer than 314 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: oid 444 updating version 316 to 317 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 444 v 317 (ObjNum 443 snap 0 seq_num 443) dirty exists 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: left oid 444 (ObjNum 443 snap 0 seq_num 443) 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:444: done (15 left) 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:460: write initial oid 460 2024-09-21T18:07:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:460: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2046 4.3 4:ff17507d:::smithi16030472-456:head [write 447185~703122 in=703122b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.571386+0000", 2024-09-21T18:07:37.754 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.176728733, 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.17677435499999999, 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2046 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571386+0000", 2024-09-21T18:07:37.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571386+0000", 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571387+0000", 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6750000000000001e-06 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571603+0000", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021572400000000001 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571614+0000", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.128e-05 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571621+0000", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3829999999999997e-06 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.571669+0000", 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8442999999999997e-05 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2011 4.3 4:d5e652b8:::smithi16030472-449:head [write 729536~460026 in=460026b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079404+0000", 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.668710111, 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66886011700000003, 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.758 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2011 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079404+0000", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079404+0000", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079405+0000", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8599999999999999e-07 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079563+0000", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000158634 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079571+0000", 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5769999999999996e-06 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079575+0000", 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3460000000000004e-06 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139587+0000", 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060011637 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.761 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2047 4.3 4:ff17507d:::smithi16030472-456:head [write 1642601~765814 in=765814b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.578425+0000", 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16968973200000001, 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16992789899999999, 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2047 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.762 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578425+0000", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578425+0000", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578427+0000", 2024-09-21T18:07:37.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.148e-06 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578824+0000", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039704200000000001 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578841+0000", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7033000000000001e-05 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578850+0000", 2024-09-21T18:07:37.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0259999999999996e-06 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.578920+0000", 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9856000000000003e-05 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66853320599999999, 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66885446100000001, 2024-09-21T18:07:37.765 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2048 4.3 4:ff17507d:::smithi16030472-456:head [write 2928991~447707 in=447707b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.583765+0000", 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16434901099999999, 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16475727800000001, 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2048 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.769 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583765+0000", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583765+0000", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583767+0000", 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0490000000000002e-06 2024-09-21T18:07:37.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583918+0000", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000150366 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583933+0000", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4789e-05 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583941+0000", 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4970000000000001e-06 2024-09-21T18:07:37.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584005+0000", 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4237000000000002e-05 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66833865999999997, 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66882932799999995, 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.772 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2049 4.3 4:ff17507d:::smithi16030472-456:head [setxattr _header (57) in=64b,truncate 3647185] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.583961+0000", 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16415292400000001, 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16472920699999999, 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2049 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583961+0000", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583961+0000", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583962+0000", 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5199999999999995e-07 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583966+0000", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5750000000000001e-06 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583972+0000", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3169999999999998e-06 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583977+0000", 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3900000000000003e-06 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584119+0000", 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000142244 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66805227199999995, 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66873591200000004, 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.779 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:37.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:07:37.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2050 4.3 4:ff17507d:::smithi16030472-456:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.583998+0000", 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.1641167, 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16488709100000001, 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2050 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583998+0000", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583998+0000", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.583998+0000", 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8800000000000002e-07 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584001+0000", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.4700000000000001e-06 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584006+0000", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4280000000000004e-06 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584010+0000", 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.34e-06 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584229+0000", 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021808600000000001 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.584249+0000", 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0852999999999998e-05 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.786 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66803619199999997, 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66889726400000005, 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:37.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:07:37.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:07:37.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2056 4.3 4:d0c1cf97:::smithi16030472-458:head [write 480799~569973 in=569973b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.611610+0000", 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.13650456899999999, 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.137448449, 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.790 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2056 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611610+0000", 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611610+0000", 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611615+0000", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.006e-06 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611861+0000", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024656100000000001 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611900+0000", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8828999999999997e-05 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.611920+0000", 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9836000000000001e-05 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.612023+0000", 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102707 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2057 4.3 4:d0c1cf97:::smithi16030472-458:head [write 1475918~769483 in=769483b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.619111+0000", 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12900328899999999, 2024-09-21T18:07:37.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.130029004, 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2057 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619111+0000", 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619111+0000", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619113+0000", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9e-06 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619458+0000", 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034478699999999999 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619473+0000", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5718000000000001e-05 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619482+0000", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0960000000000001e-06 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.619548+0000", 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6356e-05 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2058 4.3 4:d0c1cf97:::smithi16030472-458:head [write 2700190~586604 in=586604b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.625391+0000", 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.122722874, 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.123831223, 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2058 2024-09-21T18:07:37.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625391+0000", 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625391+0000", 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625396+0000", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3950000000000003e-06 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625847+0000", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045130099999999998 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625875+0000", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8112000000000001e-05 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625891+0000", 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5798e-05 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625990+0000", 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9050999999999997e-05 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2059 4.3 4:d0c1cf97:::smithi16030472-458:head [setxattr _header (57) in=64b,truncate 3680799] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.625925+0000", 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.122189828, 2024-09-21T18:07:37.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.123380925, 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2059 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625925+0000", 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625925+0000", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625926+0000", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2759999999999999e-06 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625932+0000", 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6000000000000003e-06 2024-09-21T18:07:37.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.625942+0000", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5040000000000008e-06 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726046+0000", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100104017 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726179+0000", 2024-09-21T18:07:37.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013267699999999999 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.11.0:543 4.1 e53/33)", 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.723649+0000", 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.024464952000000002, 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.025731074, 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723649+0000", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723649+0000", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723651+0000", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.052e-06 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723660+0000", 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1179999999999992e-06 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723676+0000", 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6473999999999998e-05 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.723686+0000", 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5410000000000006e-06 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.806 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.807 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2060 4.3 4.e9f3830b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.807 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.726068+0000", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.022046057000000001, 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023380607000000001, 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2060 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726068+0000", 2024-09-21T18:07:37.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726068+0000", 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726070+0000", 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6330000000000001e-06 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726076+0000", 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.285e-06 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726084+0000", 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0800000000000006e-06 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.726092+0000", 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2949999999999996e-06 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.63910893999999996, 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.640520058, 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.632889865, 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63438285100000003, 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.814 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:37.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:07:37.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.62778827000000004, 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.62940396300000001, 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.62745432199999995, 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.62923261500000005, 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.821 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.824 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.62743384300000005, 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.62937559799999998, 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:37.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1989 4.3 4:fd16f4c4:::smithi16030472-444:head [setxattr _header (57) in=64b,truncate 3445501] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978041+0000", 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.77007304200000004, 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77218209000000004, 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.828 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1989 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978041+0000", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978041+0000", 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978044+0000", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2680000000000001e-06 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978052+0000", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8869999999999998e-06 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978061+0000", 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5359999999999992e-06 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978067+0000", 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2230000000000004e-06 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.137325+0000", 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.159257445 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1990 4.3 4:fd16f4c4:::smithi16030472-444:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.831 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978074+0000", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.770040693, 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77231997399999996, 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1990 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978074+0000", 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978074+0000", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978074+0000", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1600000000000002e-07 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978076+0000", 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8020000000000001e-06 2024-09-21T18:07:37.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978080+0000", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.777e-06 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978082+0000", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8169999999999999e-06 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.137511+0000", 2024-09-21T18:07:37.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.159428454 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2001 4.3 4:c4c554e1:::smithi16030472-447:head [write 695922~709528 in=709528b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978087+0000", 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.77002712299999998, 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77247733600000001, 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.835 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2001 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978087+0000", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978087+0000", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978088+0000", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5499999999999999e-07 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978267+0000", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017910300000000001 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978273+0000", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3559999999999998e-06 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978277+0000", 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1620000000000001e-06 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.137740+0000", 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15946237799999999 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2034 4.1 4:94e02f4e:::smithi16030472-453:head [setxattr _header (57) in=64b,truncate 3596764] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.201396+0000", 2024-09-21T18:07:37.838 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.54671878500000004, 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54933670000000001, 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2034 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201396+0000", 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201396+0000", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201397+0000", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3009999999999999e-06 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201403+0000", 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3260000000000001e-06 2024-09-21T18:07:37.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201412+0000", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1519999999999998e-06 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201421+0000", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5269999999999998e-06 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.296961+0000", 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095539995000000003 2024-09-21T18:07:37.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.297033+0000", 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2547999999999997e-05 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2002 4.3 4:c4c554e1:::smithi16030472-447:head [write 2032616~607480 in=607480b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978283+0000", 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.76983149699999998, 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.772635195, 2024-09-21T18:07:37.842 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2002 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978283+0000", 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978283+0000", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978283+0000", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5699999999999998e-07 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978428+0000", 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014499700000000001 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978434+0000", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9020000000000004e-06 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978437+0000", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0709999999999999e-06 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.138259+0000", 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15982159600000001 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2035 4.1 4:94e02f4e:::smithi16030472-453:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.201434+0000", 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.54668022999999999, 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54965266300000004, 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2035 2024-09-21T18:07:37.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201434+0000", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201434+0000", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201435+0000", 2024-09-21T18:07:37.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8199999999999998e-07 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201439+0000", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2629999999999997e-06 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201455+0000", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5390999999999999e-05 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.201464+0000", 2024-09-21T18:07:37.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9660000000000002e-06 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.297118+0000", 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095654823 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.297231+0000", 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011247399999999999 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2003 4.3 4:c4c554e1:::smithi16030472-447:head [write 3198537~297385 in=297385b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.76967183900000002, 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.77283078400000005, 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2003 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.46e-07 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978588+0000", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014504599999999999 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978594+0000", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9660000000000001e-06 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078703+0000", 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10010912399999999 2024-09-21T18:07:37.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.138478+0000", 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.059774543999999999 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.541105316, 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.54443228700000001, 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.853 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:07:37.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:07:37.856 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.53264513000000002, 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.53606465199999997, 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.857 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52671742499999996, 2024-09-21T18:07:37.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.53022999800000004, 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:07:37.863 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2039 4.3 4:e9fae96f:::smithi16030472-454:head [setxattr _header (57) in=64b,truncate 3613571] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221702+0000", 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52641249599999995, 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.53000839700000002, 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.864 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2039 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221702+0000", 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221702+0000", 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221702+0000", 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2600000000000002e-07 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221706+0000", 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221711+0000", 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7759999999999997e-06 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221715+0000", 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9600000000000002e-06 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342698+0000", 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12098321200000001 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2004 4.3 4:c4c554e1:::smithi16030472-447:head [setxattr _header (57) in=64b,truncate 3495922] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.867 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.078723+0000", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66939162399999996, 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.67306953999999997, 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2004 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078723+0000", 2024-09-21T18:07:37.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078723+0000", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078724+0000", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078732+0000", 2024-09-21T18:07:37.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9170000000000006e-06 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078741+0000", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7579999999999998e-06 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078750+0000", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4100000000000008e-06 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.138926+0000", 2024-09-21T18:07:37.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060176854000000002 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2040 4.3 4:e9fae96f:::smithi16030472-454:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221721+0000", 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.52639361799999995, 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.53015220200000002, 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.871 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2040 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221721+0000", 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221721+0000", 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221721+0000", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7300000000000001e-07 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221723+0000", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0729999999999999e-06 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221728+0000", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8629999999999999e-06 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221732+0000", 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9580000000000001e-06 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342918+0000", 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12118557100000001 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2005 4.3 4:c4c554e1:::smithi16030472-447:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.078756+0000", 2024-09-21T18:07:37.874 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66935799100000004, 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.67321233300000005, 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2005 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078756+0000", 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078756+0000", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078757+0000", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5400000000000001e-07 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078759+0000", 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5730000000000002e-06 2024-09-21T18:07:37.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078764+0000", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9470000000000002e-06 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078769+0000", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3669999999999999e-06 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139318+0000", 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060549049000000001 2024-09-21T18:07:37.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 35 2024-09-21T18:07:37.878 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:37.878 INFO:tasks.rados.rados.0.smithi160.stdout:460: writing smithi16030472-460 from 514413 to 1234884 tid 1 2024-09-21T18:07:37.878 INFO:tasks.rados.rados.0.smithi160.stdout:460: writing smithi16030472-460 from 1876529 to 2649681 tid 2 2024-09-21T18:07:37.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:460: writing smithi16030472-460 from 3438683 to 3714413 tid 3 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: finishing write tid 2 to smithi16030472-436 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: finishing write tid 3 to smithi16030472-436 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: finishing write tid 4 to smithi16030472-436 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: finishing write tid 5 to smithi16030472-436 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: oid 436 updating version 0 to 297 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: oid 436 version 297 is already newer than 295 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: oid 436 version 297 is already newer than 296 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: oid 436 version 297 is already newer than 294 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 436 v 297 (ObjNum 435 snap 0 seq_num 435) dirty exists 2024-09-21T18:07:37.882 INFO:tasks.rados.rados.0.smithi160.stdout:436: left oid 436 (ObjNum 435 snap 0 seq_num 435) 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:436: done (15 left) 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:461: write initial oid 461 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:461: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:461: writing smithi16030472-461 from 531220 to 1326940 tid 1 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:461: writing smithi16030472-461 from 1935011 to 2451821 tid 2 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:461: writing smithi16030472-461 from 3066106 to 3486526 tid 3 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: finishing write tid 3 to smithi16030472-441 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: finishing write tid 4 to smithi16030472-441 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: finishing write tid 5 to smithi16030472-441 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: oid 441 updating version 0 to 298 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: oid 441 version 298 is already newer than 296 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: oid 441 updating version 298 to 299 2024-09-21T18:07:37.883 INFO:tasks.rados.rados.0.smithi160.stdout:441: oid 441 version 299 is already newer than 295 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 299 (ObjNum 440 snap 0 seq_num 440) dirty exists 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:441: left oid 441 (ObjNum 440 snap 0 seq_num 440) 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:441: done (15 left) 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:462: write initial oid 462 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:462: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:462: writing smithi16030472-462 from 548027 to 1018996 tid 1 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:462: writing smithi16030472-462 from 1593493 to 2370314 tid 2 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout:462: writing smithi16030472-462 from 3093529 to 3725221 tid 3 2024-09-21T18:07:37.884 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.884 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:37.877+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:37.926 INFO:tasks.rados.rados.0.smithi160.stdout:450: finishing write tid 2 to smithi16030472-450 2024-09-21T18:07:37.926 INFO:tasks.rados.rados.0.smithi160.stdout:450: finishing write tid 3 to smithi16030472-450 2024-09-21T18:07:37.928 INFO:tasks.rados.rados.0.smithi160.stdout:445: finishing write tid 2 to smithi16030472-445 2024-09-21T18:07:37.928 INFO:tasks.rados.rados.0.smithi160.stdout:445: finishing write tid 3 to smithi16030472-445 2024-09-21T18:07:37.929 INFO:tasks.rados.rados.0.smithi160.stdout:445: finishing write tid 4 to smithi16030472-445 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: finishing write tid 5 to smithi16030472-445 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: oid 445 updating version 0 to 300 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: oid 445 updating version 300 to 302 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: oid 445 version 302 is already newer than 297 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: oid 445 version 302 is already newer than 301 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 302 (ObjNum 444 snap 0 seq_num 444) dirty exists 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: left oid 445 (ObjNum 444 snap 0 seq_num 444) 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:445: done (15 left) 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:463: write initial oid 463 2024-09-21T18:07:37.930 INFO:tasks.rados.rados.0.smithi160.stdout:463: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-21T18:07:37.939 INFO:tasks.rados.rados.0.smithi160.stdout:463: writing smithi16030472-463 from 564834 to 1111052 tid 1 2024-09-21T18:07:37.946 INFO:tasks.rados.rados.0.smithi160.stdout:463: writing smithi16030472-463 from 1651975 to 2172454 tid 2 2024-09-21T18:07:37.953 INFO:tasks.rados.rados.0.smithi160.stdout:463: writing smithi16030472-463 from 2720952 to 3280269 tid 3 2024-09-21T18:07:37.953 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:37.983 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:37.981+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:38.018 INFO:tasks.rados.rados.0.smithi160.stdout:453: finishing write tid 4 to smithi16030472-453 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:447: finishing write tid 1 to smithi16030472-447 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:447: finishing write tid 2 to smithi16030472-447 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:453: finishing write tid 5 to smithi16030472-453 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:453: oid 453 updating version 0 to 385 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:453: oid 453 updating version 385 to 386 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:453: oid 453 version 386 is already newer than 384 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:453: oid 453 version 386 is already newer than 383 2024-09-21T18:07:38.019 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 453 v 386 (ObjNum 452 snap 0 seq_num 452) dirty exists 2024-09-21T18:07:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:453: left oid 453 (ObjNum 452 snap 0 seq_num 452) 2024-09-21T18:07:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:453: done (15 left) 2024-09-21T18:07:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:464: write initial oid 464 2024-09-21T18:07:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:464: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-21T18:07:38.027 INFO:tasks.rados.rados.0.smithi160.stdout:464: writing smithi16030472-464 from 581641 to 1203108 tid 1 2024-09-21T18:07:38.038 INFO:tasks.rados.rados.0.smithi160.stdout:464: writing smithi16030472-464 from 1994104 to 2774594 tid 2 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:464: writing smithi16030472-464 from 3432022 to 3781641 tid 3 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:452: finishing write tid 1 to smithi16030472-452 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:452: finishing write tid 2 to smithi16030472-452 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:455: finishing write tid 1 to smithi16030472-455 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:450: finishing write tid 4 to smithi16030472-450 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:450: finishing write tid 5 to smithi16030472-450 2024-09-21T18:07:38.046 INFO:tasks.rados.rados.0.smithi160.stdout:450: oid 450 updating version 0 to 318 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:450: oid 450 updating version 318 to 319 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:450: oid 450 updating version 319 to 320 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:450: oid 450 version 320 is already newer than 315 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 450 v 320 (ObjNum 449 snap 0 seq_num 449) dirty exists 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:450: left oid 450 (ObjNum 449 snap 0 seq_num 449) 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:450: done (15 left) 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:465: write initial oid 465 2024-09-21T18:07:38.047 INFO:tasks.rados.rados.0.smithi160.stdout:465: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-21T18:07:38.053 INFO:tasks.rados.rados.0.smithi160.stdout:465: writing smithi16030472-465 from 598448 to 1011517 tid 1 2024-09-21T18:07:38.062 INFO:tasks.rados.rados.0.smithi160.stdout:465: writing smithi16030472-465 from 1768939 to 2293087 tid 2 2024-09-21T18:07:38.064 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:38.064 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:38.064 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:38.064 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:38.064 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:38.070 INFO:tasks.rados.rados.0.smithi160.stdout:465: writing smithi16030472-465 from 2775798 to 3357659 tid 3 2024-09-21T18:07:38.074 INFO:tasks.rados.rados.0.smithi160.stdout:465: writing smithi16030472-465 from 3781488 to 3798448 tid 4 2024-09-21T18:07:38.074 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:38.122 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:38.118+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:447: finishing write tid 3 to smithi16030472-447 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:447: finishing write tid 4 to smithi16030472-447 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:463: finishing write tid 1 to smithi16030472-463 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:447: finishing write tid 5 to smithi16030472-447 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:447: oid 447 updating version 0 to 320 2024-09-21T18:07:38.125 INFO:tasks.rados.rados.0.smithi160.stdout:447: oid 447 version 320 is already newer than 318 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:447: oid 447 updating version 320 to 321 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:447: oid 447 version 321 is already newer than 319 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 447 v 321 (ObjNum 446 snap 0 seq_num 446) dirty exists 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:447: left oid 447 (ObjNum 446 snap 0 seq_num 446) 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:449: finishing write tid 1 to smithi16030472-449 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:447: done (15 left) 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:466: write initial oid 466 2024-09-21T18:07:38.126 INFO:tasks.rados.rados.0.smithi160.stdout:466: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-21T18:07:38.132 INFO:tasks.rados.rados.0.smithi160.stdout:466: writing smithi16030472-466 from 615255 to 1103573 tid 1 2024-09-21T18:07:38.142 INFO:tasks.rados.rados.0.smithi160.stdout:466: writing smithi16030472-466 from 1827421 to 2611580 tid 2 2024-09-21T18:07:38.151 INFO:tasks.rados.rados.0.smithi160.stdout:466: writing smithi16030472-466 from 3203221 to 3815255 tid 3 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout:459: finishing write tid 2 to smithi16030472-459 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout:459: finishing write tid 3 to smithi16030472-459 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout:459: finishing write tid 4 to smithi16030472-459 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout:459: finishing write tid 5 to smithi16030472-459 2024-09-21T18:07:38.152 INFO:tasks.rados.rados.0.smithi160.stdout:459: oid 459 updating version 0 to 321 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:459: oid 459 version 321 is already newer than 316 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:459: oid 459 updating version 321 to 323 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:459: oid 459 version 323 is already newer than 322 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 323 (ObjNum 458 snap 0 seq_num 458) dirty exists 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:459: left oid 459 (ObjNum 458 snap 0 seq_num 458) 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:463: finishing write tid 2 to smithi16030472-463 2024-09-21T18:07:38.153 INFO:tasks.rados.rados.0.smithi160.stdout:459: done (15 left) 2024-09-21T18:07:38.154 INFO:tasks.rados.rados.0.smithi160.stdout:467: write initial oid 467 2024-09-21T18:07:38.154 INFO:tasks.rados.rados.0.smithi160.stdout:467: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-21T18:07:38.159 INFO:tasks.rados.rados.0.smithi160.stdout:467: writing smithi16030472-467 from 632062 to 1195629 tid 1 2024-09-21T18:07:38.168 INFO:tasks.rados.rados.0.smithi160.stdout:467: writing smithi16030472-467 from 1885903 to 2413720 tid 2 2024-09-21T18:07:38.176 INFO:tasks.rados.rados.0.smithi160.stdout:467: writing smithi16030472-467 from 2830644 to 3551402 tid 3 2024-09-21T18:07:38.176 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.176 INFO:tasks.rados.rados.0.smithi160.stdout:463: finishing write tid 3 to smithi16030472-463 2024-09-21T18:07:38.184 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:38.178+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:38.222 INFO:tasks.rados.rados.0.smithi160.stdout:455: finishing write tid 2 to smithi16030472-455 2024-09-21T18:07:38.223 INFO:tasks.rados.rados.0.smithi160.stdout:452: finishing write tid 3 to smithi16030472-452 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:95 4.3 4:fbf38165:::smithi16030472-24:head [setxattr _header (56) in=7b,truncate 2386561] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e42)", 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.925295759000001, 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.124057401, 2024-09-21T18:07:38.254 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 95 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323857+0000", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323859+0000", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.336e-06 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323864+0000", 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.49e-06 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323873+0000", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4759999999999998e-06 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323880+0000", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426219+0000", 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102339044 2024-09-21T18:07:38.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426347+0000", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012758800000000001 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.789980+0000", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36363283400000002 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790091+0000", 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110766 2024-09-21T18:07:38.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790205+0000", 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011423 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.791431+0000", 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012262099999999999 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447768+0000", 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65633668700000003 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447842+0000", 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4401999999999997e-05 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447915+0000", 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2640999999999998e-05 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:96 4.3 4:fbf38165:::smithi16030472-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e42)", 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.925262926999999, 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.124285918, 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 96 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323890+0000", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323891+0000", 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1299999999999998e-07 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323895+0000", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9400000000000004e-06 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323904+0000", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7469999999999994e-06 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.323917+0000", 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2917e-05 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426413+0000", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102496121 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.426513+0000", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000100098 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790579+0000", 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36406573399999997 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:12.790657+0000", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7955999999999997e-05 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.447958+0000", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65730143699999999 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448078+0000", 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011979799999999999 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448176+0000", 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8257000000000005e-05 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.104187272000001, 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.266 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:07:38.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:07:38.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:07:38.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:07:38.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.104140962999999, 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.272 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:143 4.3 4:d903dcdd:::smithi16030472-36:head [setxattr _header (56) in=63b,truncate 2588245] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.102893028, 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0834338750000001, 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 143 2024-09-21T18:07:38.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146260+0000", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146261+0000", 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0330000000000001e-06 2024-09-21T18:07:38.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146266+0000", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6229999999999996e-06 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146276+0000", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0326000000000001e-05 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146295+0000", 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9097e-05 2024-09-21T18:07:38.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246122+0000", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099826456999999993 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246156+0000", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4924e-05 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.124447+0000", 2024-09-21T18:07:38.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87829058999999998 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.124584+0000", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013713700000000001 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.124808+0000", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022399400000000001 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.133191+0000", 2024-09-21T18:07:38.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0083824520000000003 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.229402+0000", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.096211110000000002 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.229544+0000", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000141891 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.229694+0000", 2024-09-21T18:07:38.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015024099999999999 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:144 4.3 4:d903dcdd:::smithi16030472-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.102847652000001, 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840101149999999, 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 144 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146305+0000", 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146306+0000", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146310+0000", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7239999999999998e-06 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146318+0000", 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2689999999999999e-06 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.146325+0000", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0400000000000004e-06 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246212+0000", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099886246999999997 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246253+0000", 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1752999999999998e-05 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.125225+0000", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87897203000000002 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.125443+0000", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021719600000000001 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.229802+0000", 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.104358926 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.230041+0000", 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023972199999999999 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.230316+0000", 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000274252 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.289 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.102045050000001, 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.295 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.101381934999999, 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:07:38.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:07:38.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:07:38.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:07:38.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:07:38.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:07:38.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.101345435999999, 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:07:38.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:07:38.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.967707264000001, 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:07:38.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:07:38.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:07:38.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.966995473000001, 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.966957910000001, 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.325 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:94 3.2 3:4999db3b:::smithi16030472-106:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e46)", 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.318076+0000", 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.931077101, 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0725352349999999, 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318076+0000", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318076+0000", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318077+0000", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.003e-06 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318082+0000", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4870000000000002e-06 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318091+0000", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6500000000000002e-06 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.318099+0000", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9370000000000004e-06 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.323213+0000", 2024-09-21T18:07:38.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051137589999999998 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.323322+0000", 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000109042 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.386994+0000", 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.063672476 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.387112+0000", 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117528 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.390611+0000", 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034993530000000002 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:100 3.2 3:6269f105:::smithi16030472-107:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e46)", 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.320376+0000", 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.928777444000001, 2024-09-21T18:07:38.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.062384853, 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320376+0000", 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320376+0000", 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320376+0000", 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2399999999999997e-07 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320380+0000", 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2459999999999999e-06 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320388+0000", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0800000000000006e-06 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320395+0000", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4200000000000001e-06 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.529389+0000", 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.208993703 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.529516+0000", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012729500000000001 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.382536+0000", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85301953399999997 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.382650+0000", 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011452599999999999 2024-09-21T18:07:38.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.382760+0000", 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000110225 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:101 3.2 3:6269f105:::smithi16030472-107:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e46)", 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.320404+0000", 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.928748986999999, 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0664736290000001, 2024-09-21T18:07:38.335 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320404+0000", 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320404+0000", 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320405+0000", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9299999999999997e-07 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320409+0000", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3499999999999999e-06 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320420+0000", 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0569e-05 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.320428+0000", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2039999999999997e-06 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.529581+0000", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20915312 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.529654+0000", 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2688000000000006e-05 2024-09-21T18:07:38.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.382838+0000", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.85318437700000005 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.382952+0000", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011397500000000001 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.386878+0000", 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003925553 2024-09-21T18:07:38.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:551 3.2 3:412eff35:::smithi16030472-123:head [write 1505862~544592] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672823+0000", 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.576330329000001, 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0813029190000001, 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672823+0000", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672823+0000", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672826+0000", 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.066e-06 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673155+0000", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000329642 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673186+0000", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0846000000000002e-05 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673202+0000", 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6030000000000001e-05 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.390715+0000", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71751243399999998 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.390842+0000", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012704799999999999 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.391058+0000", 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021611300000000001 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.397041+0000", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0059826849999999997 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.753870+0000", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.356829436 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754012+0000", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014223400000000001 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754126+0000", 2024-09-21T18:07:38.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000113385 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 3.2 3:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.673253+0000", 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.575899609, 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.081368565, 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673253+0000", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673253+0000", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673255+0000", 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.64e-06 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673262+0000", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673276+0000", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4343e-05 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.673286+0000", 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5370000000000003e-06 2024-09-21T18:07:38.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.393173+0000", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.71988735500000001 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.393308+0000", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134435 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.393512+0000", 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000204663 2024-09-21T18:07:38.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.399599+0000", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0060864170000000002 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754256+0000", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35465675400000002 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754414+0000", 2024-09-21T18:07:38.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000158388 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754622+0000", 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00020803999999999999 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:125 3.2 3:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.685104+0000", 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.564049436000001, 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0702337879999999, 2024-09-21T18:07:38.350 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685104+0000", 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685104+0000", 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685107+0000", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3000000000000002e-06 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685117+0000", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0067e-05 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685147+0000", 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0389999999999999e-05 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.685163+0000", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6008999999999999e-05 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.394045+0000", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70888121900000001 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.394237+0000", 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019219800000000001 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754798+0000", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.36056154000000001 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.755043+0000", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000244966 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.755337+0000", 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029409899999999998 2024-09-21T18:07:38.354 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 3.2 3:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.693020+0000", 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.556132976000001, 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0626951529999999, 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693020+0000", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693020+0000", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693024+0000", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0729999999999998e-06 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693872+0000", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00084829700000000003 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693903+0000", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0108999999999999e-05 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.693918+0000", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5777000000000002e-05 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.394328+0000", 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.70040991900000005 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.394523+0000", 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019447 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.394734+0000", 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000211227 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.400075+0000", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0053408179999999998 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.755460+0000", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35538515300000001 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.755605+0000", 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000145098 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.755715+0000", 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011021199999999999 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2003 4.3 4:c4c554e1:::smithi16030472-447:head [write 3198537~297385 in=297385b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 1.2707105439999999, 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.142464457, 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.360 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2003 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:38.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978443+0000", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.46e-07 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978588+0000", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014504599999999999 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:36.978594+0000", 2024-09-21T18:07:38.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9660000000000001e-06 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.078703+0000", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10010912399999999 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.138478+0000", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.059774543999999999 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.120898+0000", 2024-09-21T18:07:38.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98242085499999998 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.120907+0000", 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4759999999999998e-06 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.364 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:38.366 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:38.367 INFO:tasks.rados.rados.0.smithi160.stdout:463: finishing write tid 4 to smithi16030472-463 2024-09-21T18:07:38.367 INFO:tasks.rados.rados.0.smithi160.stdout:463: finishing write tid 5 to smithi16030472-463 2024-09-21T18:07:38.367 INFO:tasks.rados.rados.0.smithi160.stdout:463: oid 463 updating version 0 to 387 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:463: oid 463 version 387 is already newer than 386 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:463: oid 463 updating version 387 to 389 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:463: oid 463 version 389 is already newer than 388 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 463 v 389 (ObjNum 462 snap 0 seq_num 462) dirty exists 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:463: left oid 463 (ObjNum 462 snap 0 seq_num 462) 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:463: done (15 left) 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:468: write initial oid 468 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:468: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:468: writing smithi16030472-468 from 648869 to 1287685 tid 1 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:468: writing smithi16030472-468 from 1828032 to 2615860 tid 2 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:468: writing smithi16030472-468 from 3141714 to 3673744 tid 3 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:452: finishing write tid 4 to smithi16030472-452 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:457: finishing write tid 1 to smithi16030472-457 2024-09-21T18:07:38.368 INFO:tasks.rados.rados.0.smithi160.stdout:462: finishing write tid 1 to smithi16030472-462 2024-09-21T18:07:38.369 INFO:tasks.rados.rados.0.smithi160.stdout:455: finishing write tid 3 to smithi16030472-455 2024-09-21T18:07:38.369 INFO:tasks.rados.rados.0.smithi160.stdout:455: finishing write tid 4 to smithi16030472-455 2024-09-21T18:07:38.369 INFO:tasks.rados.rados.0.smithi160.stdout:457: finishing write tid 2 to smithi16030472-457 2024-09-21T18:07:38.369 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:38.362+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:38.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:07:38.440 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 1 to smithi16030472-465 2024-09-21T18:07:38.467 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:38.462+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:38.530 INFO:tasks.rados.rados.0.smithi160.stdout:449: finishing write tid 2 to smithi16030472-449 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: finishing write tid 3 to smithi16030472-449 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: finishing write tid 4 to smithi16030472-449 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: finishing write tid 5 to smithi16030472-449 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: oid 449 updating version 0 to 324 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: oid 449 version 324 is already newer than 322 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: oid 449 version 324 is already newer than 323 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: oid 449 updating version 324 to 325 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 325 (ObjNum 448 snap 0 seq_num 448) dirty exists 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:449: left oid 449 (ObjNum 448 snap 0 seq_num 448) 2024-09-21T18:07:38.531 INFO:tasks.rados.rados.0.smithi160.stdout:451: finishing write tid 1 to smithi16030472-451 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: finishing write tid 2 to smithi16030472-451 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: finishing write tid 3 to smithi16030472-451 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: finishing write tid 4 to smithi16030472-451 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: finishing write tid 5 to smithi16030472-451 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: oid 451 updating version 0 to 326 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: oid 451 updating version 326 to 329 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: oid 451 version 329 is already newer than 328 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: oid 451 version 329 is already newer than 327 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 451 v 329 (ObjNum 450 snap 0 seq_num 450) dirty exists 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:451: left oid 451 (ObjNum 450 snap 0 seq_num 450) 2024-09-21T18:07:38.532 INFO:tasks.rados.rados.0.smithi160.stdout:454: finishing write tid 1 to smithi16030472-454 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: finishing write tid 2 to smithi16030472-454 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: finishing write tid 3 to smithi16030472-454 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: finishing write tid 4 to smithi16030472-454 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: finishing write tid 5 to smithi16030472-454 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: oid 454 updating version 0 to 330 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: oid 454 updating version 330 to 331 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: oid 454 updating version 331 to 333 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: oid 454 version 333 is already newer than 332 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 454 v 333 (ObjNum 453 snap 0 seq_num 453) dirty exists 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: left oid 454 (ObjNum 453 snap 0 seq_num 453) 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:456: finishing write tid 1 to smithi16030472-456 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:449: done (15 left) 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:451: done (14 left) 2024-09-21T18:07:38.533 INFO:tasks.rados.rados.0.smithi160.stdout:454: done (13 left) 2024-09-21T18:07:38.534 INFO:tasks.rados.rados.0.smithi160.stdout:469: write initial oid 469 2024-09-21T18:07:38.534 INFO:tasks.rados.rados.0.smithi160.stdout:469: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-21T18:07:38.540 INFO:tasks.rados.rados.0.smithi160.stdout:469: writing smithi16030472-469 from 665676 to 1379741 tid 1 2024-09-21T18:07:38.548 INFO:tasks.rados.rados.0.smithi160.stdout:469: writing smithi16030472-469 from 1886514 to 2418000 tid 2 2024-09-21T18:07:38.550 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:38.550 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:38.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4.8a532ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.32268833499999999, 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32272585799999998, 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:07:38.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.554 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2102 3.3 3:e43034e2:::smithi16030472-467:head [write 632062~563567 in=563567b,stat out=16b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.259924+0000", 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.285459354, 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28560097499999998, 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.259924+0000", 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.259924+0000", 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.259928+0000", 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8199999999999998e-06 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.556 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.260163+0000", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023539200000000001 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.260186+0000", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2986e-05 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360323+0000", 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100137209 2024-09-21T18:07:38.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360422+0000", 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8624999999999998e-05 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360639+0000", 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000216775 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360896+0000", 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025728699999999998 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.368782+0000", 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0078855799999999997 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2098 4.2 4.8a532ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.323185+0000", 2024-09-21T18:07:38.559 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.222198117, 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22243575500000001, 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2098 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323185+0000", 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323185+0000", 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323189+0000", 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9190000000000001e-06 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323546+0000", 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035723200000000002 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323570+0000", 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3476e-05 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.562 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323589+0000", 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9473000000000001e-05 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2099 4.2 4.8a532ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.323615+0000", 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22176833100000001, 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22207756400000001, 2024-09-21T18:07:38.563 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2099 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323615+0000", 2024-09-21T18:07:38.564 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323615+0000", 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323617+0000", 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3e-06 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.565 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323869+0000", 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025142799999999999 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323885+0000", 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6256000000000002e-05 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.566 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323901+0000", 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5569999999999998e-05 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2100 4.2 4.8a532ea (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.323922+0000", 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22146152399999999, 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.221842492, 2024-09-21T18:07:38.567 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2100 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.568 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323922+0000", 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323922+0000", 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323924+0000", 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.626e-06 2024-09-21T18:07:38.569 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323941+0000", 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323954+0000", 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3709e-05 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.570 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323969+0000", 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4744e-05 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2101 4.2 4.8a532ea (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.323988+0000", 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.221395063, 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22184796300000001, 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.571 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2101 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323988+0000", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323988+0000", 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323990+0000", 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6139999999999999e-06 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323997+0000", 2024-09-21T18:07:38.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8159999999999996e-06 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324013+0000", 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6453e-05 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324032+0000", 2024-09-21T18:07:38.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8853000000000001e-05 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2107 4.2 4.13666c2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.324053+0000", 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.221330534, 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22185327599999999, 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.575 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2107 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324053+0000", 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324053+0000", 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324055+0000", 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.657e-06 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324297+0000", 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024236700000000001 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324315+0000", 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324330+0000", 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.537e-05 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.578 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2108 4.2 4.13666c2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.324350+0000", 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22103357100000001, 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22162570500000001, 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.579 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2108 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324350+0000", 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324350+0000", 2024-09-21T18:07:38.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324352+0000", 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7770000000000001e-06 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324646+0000", 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029453599999999997 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324662+0000", 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5457000000000001e-05 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324677+0000", 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5353e-05 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.582 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2054 4.2 4:67a383ae:::smithi16030472-457:head [setxattr _header (57) in=64b,truncate 3663992] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.698321+0000", 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.84706279600000001, 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.84773648099999999, 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.583 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2054 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.698321+0000", 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.698321+0000", 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.698322+0000", 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.263e-06 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.698328+0000", 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0440000000000001e-06 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.698339+0000", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1456e-05 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798486+0000", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10014691100000001 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798630+0000", 2024-09-21T18:07:38.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014389700000000001 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798732+0000", 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000102169 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.448576+0000", 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64984412000000003 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.448731+0000", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000154745 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.448943+0000", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00021223800000000001 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450399+0000", 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001455097 2024-09-21T18:07:38.588 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2109 4.2 4.13666c2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.324696+0000", 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.220687366, 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22147682499999999, 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.589 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2109 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324696+0000", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324696+0000", 2024-09-21T18:07:38.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324698+0000", 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1239999999999999e-06 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324931+0000", 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000233031 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324947+0000", 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6223e-05 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324964+0000", 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6446000000000001e-05 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2110 4.2 4.13666c2e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.324985+0000", 2024-09-21T18:07:38.592 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22039883199999999, 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22125872599999999, 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2110 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324985+0000", 2024-09-21T18:07:38.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324985+0000", 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324986+0000", 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.714e-06 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.324996+0000", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5219999999999996e-06 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325014+0000", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8046999999999998e-05 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325028+0000", 2024-09-21T18:07:38.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4555999999999999e-05 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2055 4.2 4:67a383ae:::smithi16030472-457:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.798522+0000", 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.74686123000000004, 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74779562300000002, 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:38.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2055 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798522+0000", 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798522+0000", 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798525+0000", 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.067e-06 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798537+0000", 2024-09-21T18:07:38.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1579999999999999e-05 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798554+0000", 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7180000000000002e-05 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798566+0000", 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1511999999999999e-05 2024-09-21T18:07:38.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798764+0000", 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000198817 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798801+0000", 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6507999999999997e-05 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.449516+0000", 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.650715618 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.449678+0000", 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016123299999999999 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2087 3.3 3:cd0dd361:::smithi16030472-464:head [write 1994104~780490 in=780490b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.124212+0000", 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42117187099999998, 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.422214494, 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124212+0000", 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124212+0000", 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124214+0000", 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5500000000000001e-06 2024-09-21T18:07:38.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124644+0000", 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043038100000000003 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124676+0000", 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1723000000000001e-05 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124700+0000", 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3334999999999999e-05 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.233977+0000", 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.109277637 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.234093+0000", 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115832 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.234272+0000", 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017859899999999999 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.240586+0000", 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0063147890000000003 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4.13666c2e (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22033844699999999, 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22147223499999999, 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.607 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:38.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2088 3.3 3:cd0dd361:::smithi16030472-464:head [write 3432022~349619 in=349619b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.124737+0000", 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42064678599999999, 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42185730599999999, 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124737+0000", 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124737+0000", 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.124740+0000", 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0699999999999998e-06 2024-09-21T18:07:38.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125128+0000", 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038794099999999999 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125148+0000", 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0227000000000001e-05 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125162+0000", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4396000000000001e-05 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.236503+0000", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.111341158 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.236584+0000", 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0611000000000005e-05 2024-09-21T18:07:38.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.236709+0000", 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000124938 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.240600+0000", 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0038908969999999999 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2067 4.2 4:7201c251:::smithi16030472-460:head [write 1876529~773152 in=773152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.798970+0000", 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.74641320700000002, 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74772547099999997, 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.616 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2067 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798970+0000", 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798970+0000", 2024-09-21T18:07:38.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.798971+0000", 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.15e-06 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799276+0000", 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000304226 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799288+0000", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2359e-05 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799297+0000", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5769999999999994e-06 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799354+0000", 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7309000000000003e-05 2024-09-21T18:07:38.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799388+0000", 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4569000000000001e-05 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2089 3.3 3:cd0dd361:::smithi16030472-464:head [setxattr _header (57) in=64b,truncate 3781641] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.125180+0000", 2024-09-21T18:07:38.620 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.42020355599999998, 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.42161043399999998, 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125180+0000", 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125180+0000", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125181+0000", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.297e-06 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125188+0000", 2024-09-21T18:07:38.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3729999999999998e-06 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125200+0000", 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2283000000000001e-05 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.125208+0000", 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3669999999999997e-06 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.239774+0000", 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.114565447 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.239898+0000", 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012439999999999999 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.240079+0000", 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000181323 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.241292+0000", 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0012128289999999999 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.625 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2068 4.2 4:7201c251:::smithi16030472-460:head [write 3438683~275730 in=275730b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.799314+0000", 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.74606936499999998, 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74757121100000001, 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2068 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.626 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799314+0000", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799314+0000", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799315+0000", 2024-09-21T18:07:38.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7000000000000003e-07 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799438+0000", 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012292299999999999 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799450+0000", 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2223e-05 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799461+0000", 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0825999999999999e-05 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799514+0000", 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2842e-05 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799549+0000", 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5549000000000001e-05 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2103 3.3 3:e43034e2:::smithi16030472-467:head [write 1885903~527817 in=527817b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.360373+0000", 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.18501092799999999, 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.186605049, 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360373+0000", 2024-09-21T18:07:38.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360373+0000", 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360376+0000", 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1470000000000001e-06 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360619+0000", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024337 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360641+0000", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1923999999999999e-05 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360653+0000", 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2486e-05 2024-09-21T18:07:38.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.363076+0000", 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.002422184 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.363191+0000", 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115358 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.363368+0000", 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017662600000000001 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.368795+0000", 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054275599999999997 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2069 4.2 4:7201c251:::smithi16030472-460:head [setxattr _header (57) in=64b,truncate 3714413] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.799479+0000", 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.74590489599999998, 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74759579499999995, 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.635 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2069 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799479+0000", 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799479+0000", 2024-09-21T18:07:38.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799480+0000", 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.023e-06 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799485+0000", 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0699999999999997e-06 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799493+0000", 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5790000000000004e-06 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799501+0000", 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5939999999999996e-06 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799588+0000", 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7357999999999999e-05 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799622+0000", 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3491000000000002e-05 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2104 3.3 3:e43034e2:::smithi16030472-467:head [write 2830644~720758 in=720758b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.360672+0000", 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.18471115799999999, 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.18649338500000001, 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360672+0000", 2024-09-21T18:07:38.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360672+0000", 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360674+0000", 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.373e-06 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361019+0000", 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034575200000000001 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361033+0000", 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3677e-05 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361043+0000", 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0256e-05 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.366160+0000", 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0051167970000000002 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.366289+0000", 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012918000000000001 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.366467+0000", 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000178159 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.369622+0000", 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031547350000000001 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2070 4.2 4:7201c251:::smithi16030472-460:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.799513+0000", 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.74587039700000002, 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.74774690799999999, 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2070 2024-09-21T18:07:38.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799513+0000", 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799513+0000", 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799514+0000", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0699999999999996e-07 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799517+0000", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5260000000000002e-06 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799525+0000", 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9370000000000004e-06 2024-09-21T18:07:38.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799533+0000", 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7389999999999999e-06 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799645+0000", 2024-09-21T18:07:38.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000111494 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.799670+0000", 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5420000000000001e-05 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2105 3.3 3:e43034e2:::smithi16030472-467:head [setxattr _header (57) in=64b,truncate 3832062] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.650 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.361057+0000", 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.18432599399999999, 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.186295296, 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361057+0000", 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361057+0000", 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361059+0000", 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173e-06 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361065+0000", 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1099999999999999e-06 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.361073+0000", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7369999999999995e-06 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461153+0000", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100079642 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461323+0000", 2024-09-21T18:07:38.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017021199999999999 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461606+0000", 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028276700000000001 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462016+0000", 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000409684 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.463466+0000", 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0014500750000000001 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:2087 3.3 e53/31)", 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.430201+0000", 2024-09-21T18:07:38.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.115182173, 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11724011300000001, 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.430201+0000", 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.430201+0000", 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.430202+0000", 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4000000000000002e-07 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.430205+0000", 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.88e-06 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.430212+0000", 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.934e-06 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530346+0000", 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100134695 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:2054 4.2 e53/33)", 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.450730+0000", 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.094653669999999995, 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.096793265000000003, 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450730+0000", 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450730+0000", 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450732+0000", 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.119e-06 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450738+0000", 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5899999999999996e-06 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450756+0000", 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7197000000000002e-05 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.450769+0000", 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3346999999999999e-05 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1030 3.3 3:cd0dd361:::smithi16030472-464:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.159690+0000", 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38569344300000002, 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38790333300000002, 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.662 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.159690+0000", 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.159690+0000", 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.159693+0000", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0070000000000002e-06 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.159701+0000", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6669999999999996e-06 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.159721+0000", 2024-09-21T18:07:38.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0211e-05 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.259863+0000", 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100141969 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360116+0000", 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10025344999999999 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.360257+0000", 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140357 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1035 3.3 3:e43034e2:::smithi16030472-467:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.461178+0000", 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.084205652000000006, 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.086497927000000002, 2024-09-21T18:07:38.666 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461178+0000", 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461178+0000", 2024-09-21T18:07:38.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461180+0000", 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9640000000000002e-06 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461187+0000", 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8950000000000001e-06 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461197+0000", 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0166000000000001e-05 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.461204+0000", 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8340000000000001e-06 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462932+0000", 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001728835 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.463172+0000", 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023954 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.670 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:2088 3.3 e53/31)", 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.530386+0000", 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.014997781999999999, 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017363035999999998, 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530386+0000", 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530386+0000", 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530388+0000", 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6929999999999999e-06 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530399+0000", 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0777e-05 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530415+0000", 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6049000000000002e-05 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.530428+0000", 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3327e-05 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.673 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.72456589999999998, 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72699882199999999, 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.674 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:07:38.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:2089 3.3 e53/31)", 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.534211+0000", 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.011172829, 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013771190000000001, 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.534211+0000", 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.534211+0000", 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.534214+0000", 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0970000000000002e-06 2024-09-21T18:07:38.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.534223+0000", 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4129999999999995e-06 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.534248+0000", 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5119000000000001e-05 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.72449158000000002, 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.72714936699999999, 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:38.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:07:38.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.15.0:502 4.2 e53/33)", 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.537173+0000", 2024-09-21T18:07:38.686 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0082102210000000002, 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.010955599, 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537173+0000", 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537173+0000", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537175+0000", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5680000000000001e-06 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537180+0000", 2024-09-21T18:07:38.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2220000000000001e-06 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537193+0000", 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3137e-05 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.537202+0000", 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4789999999999996e-06 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.689 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:38.690 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 32 2024-09-21T18:07:38.690 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:38.691 INFO:tasks.rados.rados.0.smithi160.stdout:469: writing smithi16030472-469 from 3169137 to 3865676 tid 3 2024-09-21T18:07:38.691 INFO:tasks.rados.rados.0.smithi160.stdout:470: write initial oid 470 2024-09-21T18:07:38.691 INFO:tasks.rados.rados.0.smithi160.stdout:470: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-21T18:07:38.692 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:470: writing smithi16030472-470 from 682483 to 1471797 tid 1 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:470: writing smithi16030472-470 from 1944996 to 2736493 tid 2 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:470: writing smithi16030472-470 from 3196560 to 3751134 tid 3 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:471: write initial oid 471 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:471: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:471: writing smithi16030472-471 from 699290 to 1163853 tid 1 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:471: writing smithi16030472-471 from 1603478 to 2138633 tid 2 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:471: writing smithi16030472-471 from 2823983 to 3589829 tid 3 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:456: finishing write tid 2 to smithi16030472-456 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:456: finishing write tid 3 to smithi16030472-456 2024-09-21T18:07:38.695 INFO:tasks.rados.rados.0.smithi160.stdout:456: finishing write tid 4 to smithi16030472-456 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: finishing write tid 1 to smithi16030472-458 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: finishing write tid 2 to smithi16030472-458 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: finishing write tid 3 to smithi16030472-458 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: finishing write tid 4 to smithi16030472-458 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: finishing write tid 5 to smithi16030472-458 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: oid 458 updating version 0 to 338 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: oid 458 updating version 338 to 340 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: oid 458 version 340 is already newer than 339 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: oid 458 updating version 340 to 341 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 458 v 341 (ObjNum 457 snap 0 seq_num 457) dirty exists 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: left oid 458 (ObjNum 457 snap 0 seq_num 457) 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:464: finishing write tid 1 to smithi16030472-464 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:457: finishing write tid 3 to smithi16030472-457 2024-09-21T18:07:38.696 INFO:tasks.rados.rados.0.smithi160.stdout:458: done (15 left) 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:472: write initial oid 472 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:472: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:472: writing smithi16030472-472 from 716097 to 1255909 tid 1 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:472: writing smithi16030472-472 from 1945607 to 2740773 tid 2 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:472: writing smithi16030472-472 from 3535053 to 3916097 tid 3 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: finishing write tid 5 to smithi16030472-455 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: oid 455 updating version 0 to 305 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: oid 455 version 305 is already newer than 300 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: oid 455 version 305 is already newer than 301 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: oid 455 updating version 305 to 306 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 306 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-21T18:07:38.697 INFO:tasks.rados.rados.0.smithi160.stdout:455: left oid 455 (ObjNum 454 snap 0 seq_num 454) 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: finishing write tid 5 to smithi16030472-452 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: oid 452 updating version 0 to 299 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: oid 452 version 299 is already newer than 298 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: oid 452 updating version 299 to 303 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: oid 452 updating version 303 to 304 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 452 v 304 (ObjNum 451 snap 0 seq_num 451) dirty exists 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: left oid 452 (ObjNum 451 snap 0 seq_num 451) 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:460: finishing write tid 1 to smithi16030472-460 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:452: done (15 left) 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:455: done (14 left) 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:473: write initial oid 473 2024-09-21T18:07:38.698 INFO:tasks.rados.rados.0.smithi160.stdout:473: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:473: writing smithi16030472-473 from 732904 to 1464318 tid 1 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:473: writing smithi16030472-473 from 2120442 to 2659266 tid 2 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:473: writing smithi16030472-473 from 3278829 to 3783572 tid 3 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:474: write initial oid 474 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:474: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:474: writing smithi16030472-474 from 749711 to 1156374 tid 1 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:474: writing smithi16030472-474 from 1778924 to 2461406 tid 2 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:474: writing smithi16030472-474 from 3189899 to 3905914 tid 3 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:461: finishing write tid 1 to smithi16030472-461 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:456: finishing write tid 5 to smithi16030472-456 2024-09-21T18:07:38.699 INFO:tasks.rados.rados.0.smithi160.stdout:456: oid 456 updating version 0 to 334 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:456: oid 456 updating version 334 to 337 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:456: oid 456 version 337 is already newer than 335 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:456: oid 456 version 337 is already newer than 336 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 337 (ObjNum 455 snap 0 seq_num 455) dirty exists 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:456: left oid 456 (ObjNum 455 snap 0 seq_num 455) 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:461: finishing write tid 2 to smithi16030472-461 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:461: finishing write tid 3 to smithi16030472-461 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:456: done (15 left) 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:475: write initial oid 475 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:475: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:475: writing smithi16030472-475 from 766518 to 1248430 tid 1 2024-09-21T18:07:38.700 INFO:tasks.rados.rados.0.smithi160.stdout:475: writing smithi16030472-475 from 1837406 to 2379899 tid 2 2024-09-21T18:07:38.701 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:38.686+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:38.701 INFO:tasks.rados.rados.0.smithi160.stdout:475: writing smithi16030472-475 from 2933675 to 3460962 tid 3 2024-09-21T18:07:38.701 INFO:tasks.rados.rados.0.smithi160.stdout:475: writing smithi16030472-475 from 3876584 to 3966518 tid 4 2024-09-21T18:07:38.701 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: finishing write tid 4 to smithi16030472-457 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: finishing write tid 5 to smithi16030472-457 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: oid 457 updating version 0 to 308 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: oid 457 version 308 is already newer than 302 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: oid 457 version 308 is already newer than 307 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: oid 457 updating version 308 to 309 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 457 v 309 (ObjNum 456 snap 0 seq_num 456) dirty exists 2024-09-21T18:07:38.739 INFO:tasks.rados.rados.0.smithi160.stdout:457: left oid 457 (ObjNum 456 snap 0 seq_num 456) 2024-09-21T18:07:38.740 INFO:tasks.rados.rados.0.smithi160.stdout:460: finishing write tid 2 to smithi16030472-460 2024-09-21T18:07:38.740 INFO:tasks.rados.rados.0.smithi160.stdout:457: done (15 left) 2024-09-21T18:07:38.740 INFO:tasks.rados.rados.0.smithi160.stdout:476: write initial oid 476 2024-09-21T18:07:38.740 INFO:tasks.rados.rados.0.smithi160.stdout:476: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-21T18:07:38.746 INFO:tasks.rados.rados.0.smithi160.stdout:476: writing smithi16030472-476 from 783325 to 1340486 tid 1 2024-09-21T18:07:38.752 INFO:tasks.rados.rados.0.smithi160.stdout:476: writing smithi16030472-476 from 1779535 to 2465686 tid 2 2024-09-21T18:07:38.760 INFO:tasks.rados.rados.0.smithi160.stdout:476: writing smithi16030472-476 from 3244745 to 3699657 tid 3 2024-09-21T18:07:38.760 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.760 INFO:tasks.rados.rados.0.smithi160.stdout:460: finishing write tid 3 to smithi16030472-460 2024-09-21T18:07:38.792 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:38.790+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:38.826 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:38.826 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:38.827 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:38.827 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:38.827 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:38.841 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: finishing write tid 4 to smithi16030472-460 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: finishing write tid 5 to smithi16030472-460 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: oid 460 updating version 0 to 312 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: oid 460 version 312 is already newer than 311 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: oid 460 version 312 is already newer than 303 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: oid 460 version 312 is already newer than 310 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 460 v 312 (ObjNum 459 snap 0 seq_num 459) dirty exists 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: left oid 460 (ObjNum 459 snap 0 seq_num 459) 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:460: done (15 left) 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:477: write initial oid 477 2024-09-21T18:07:38.860 INFO:tasks.rados.rados.0.smithi160.stdout:477: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-21T18:07:38.865 INFO:tasks.rados.rados.0.smithi160.stdout:477: writing smithi16030472-477 from 400132 to 1032542 tid 1 2024-09-21T18:07:38.870 INFO:tasks.rados.rados.0.smithi160.stdout:477: writing smithi16030472-477 from 1438017 to 1984179 tid 2 2024-09-21T18:07:38.870 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.893 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:38.888+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: finishing write tid 2 to smithi16030472-462 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: finishing write tid 3 to smithi16030472-462 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: finishing write tid 4 to smithi16030472-462 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: finishing write tid 5 to smithi16030472-462 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: oid 462 updating version 0 to 325 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: oid 462 version 325 is already newer than 317 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: oid 462 version 325 is already newer than 324 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: oid 462 updating version 325 to 326 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 462 v 326 (ObjNum 461 snap 0 seq_num 461) dirty exists 2024-09-21T18:07:38.954 INFO:tasks.rados.rados.0.smithi160.stdout:462: left oid 462 (ObjNum 461 snap 0 seq_num 461) 2024-09-21T18:07:38.955 INFO:tasks.rados.rados.0.smithi160.stdout:469: finishing write tid 1 to smithi16030472-469 2024-09-21T18:07:38.955 INFO:tasks.rados.rados.0.smithi160.stdout:462: done (15 left) 2024-09-21T18:07:38.955 INFO:tasks.rados.rados.0.smithi160.stdout:478: write initial oid 478 2024-09-21T18:07:38.955 INFO:tasks.rados.rados.0.smithi160.stdout:478: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-21T18:07:38.960 INFO:tasks.rados.rados.0.smithi160.stdout:478: writing smithi16030472-478 from 416939 to 1124598 tid 1 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:478: writing smithi16030472-478 from 1896499 to 2016939 tid 2 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:471: finishing write tid 1 to smithi16030472-471 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 2 to smithi16030472-465 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 3 to smithi16030472-465 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 4 to smithi16030472-465 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 5 to smithi16030472-465 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: finishing write tid 6 to smithi16030472-465 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: oid 465 updating version 0 to 329 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: oid 465 updating version 329 to 330 2024-09-21T18:07:38.966 INFO:tasks.rados.rados.0.smithi160.stdout:465: oid 465 version 330 is already newer than 318 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:465: oid 465 version 330 is already newer than 327 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:465: oid 465 version 330 is already newer than 328 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 465 v 330 (ObjNum 464 snap 0 seq_num 464) dirty exists 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:465: left oid 465 (ObjNum 464 snap 0 seq_num 464) 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:476: finishing write tid 1 to smithi16030472-476 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:465: done (15 left) 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:479: write initial oid 479 2024-09-21T18:07:38.967 INFO:tasks.rados.rados.0.smithi160.stdout:479: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-21T18:07:38.974 INFO:tasks.rados.rados.0.smithi160.stdout:479: writing smithi16030472-479 from 433746 to 1216654 tid 1 2024-09-21T18:07:38.981 INFO:tasks.rados.rados.0.smithi160.stdout:479: writing smithi16030472-479 from 1954981 to 2033746 tid 2 2024-09-21T18:07:38.982 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:188 4.2 4:638b5b54:::smithi16030472-47:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:38.982 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 25.577283693999998, 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1271722230000001, 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 188 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:38.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398848+0000", 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398849+0000", 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8999999999999996e-07 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398852+0000", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.959e-06 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.398860+0000", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0409999999999998e-06 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.498966+0000", 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100106057 2024-09-21T18:07:38.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599677+0000", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10071127000000001 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.599763+0000", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6383000000000003e-05 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119023+0000", 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.51925935400000001 2024-09-21T18:07:38.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.119129+0000", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106513 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.525671+0000", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.40654210099999999 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.525825+0000", 2024-09-21T18:07:38.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015378899999999999 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.526020+0000", 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019506600000000001 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:551 4.2 4:412eff35:::smithi16030472-123:head [write 1505862~544592] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.671616+0000", 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.30451571, 2024-09-21T18:07:38.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.082968793, 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 551 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.671616+0000", 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.671616+0000", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.671619+0000", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2660000000000002e-06 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672248+0000", 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000629007 2024-09-21T18:07:38.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672278+0000", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0267e-05 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672294+0000", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5608999999999999e-05 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672426+0000", 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000131808 2024-09-21T18:07:38.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754579+0000", 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0821527799999999 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.754585+0000", 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0560000000000001e-06 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.992 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.303806471000001, 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:07:38.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:38.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:38.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.303736108999999, 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:38.997 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:38.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:07:38.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:07:39.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.291934674, 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:07:39.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.284068291000001, 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:07:39.005 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:39.009 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.283682646999999, 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:07:39.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:07:39.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.013 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.283637198000001, 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:39.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.235739281000001, 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.019 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:39.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:07:39.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:07:39.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 19.235631639000001, 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:07:39.023 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.659849024, 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.027 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:39.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:07:39.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.033 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 13.659797089, 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:39.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: finishing write tid 2 to smithi16030472-464 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: finishing write tid 3 to smithi16030472-464 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: finishing write tid 4 to smithi16030472-464 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: finishing write tid 5 to smithi16030472-464 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: oid 464 updating version 0 to 342 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: oid 464 updating version 342 to 345 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: oid 464 version 345 is already newer than 343 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:464: oid 464 version 345 is already newer than 344 2024-09-21T18:07:39.036 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 345 (ObjNum 463 snap 0 seq_num 463) dirty exists 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:464: left oid 464 (ObjNum 463 snap 0 seq_num 463) 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:467: finishing write tid 1 to smithi16030472-467 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:467: finishing write tid 2 to smithi16030472-467 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:467: finishing write tid 3 to smithi16030472-467 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:464: done (15 left) 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:480: write initial oid 480 2024-09-21T18:07:39.037 INFO:tasks.rados.rados.0.smithi160.stdout:480: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-21T18:07:39.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:07:39.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:07:39.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:07:39.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:07:39.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:07:39.040 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.893092701, 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.041 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1116 4.2 4:5b8f0a5a:::smithi16030472-248:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.183273+0000", 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.792858377, 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0874603469999999, 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:39.047 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1116 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183273+0000", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183273+0000", 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183276+0000", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9160000000000001e-06 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183286+0000", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9559999999999992e-06 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183310+0000", 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3487000000000001e-05 2024-09-21T18:07:39.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183321+0000", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1279e-05 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240163+0000", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056842507 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240218+0000", 2024-09-21T18:07:39.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4731999999999997e-05 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159019+0000", 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91880086299999997 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159270+0000", 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025092300000000001 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270446+0000", 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11117653 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270570+0000", 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000123843 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270734+0000", 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163311 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.663989443, 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:07:39.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:07:39.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.060 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.663561453, 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:39.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.066 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 12.663514884, 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:07:39.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:07:39.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:07:39.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:07:39.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:39.072 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.1633497740000003, 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:07:39.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:07:39.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:07:39.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:07:39.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1536 4.2 4:7b415eca:::smithi16030472-342:head [setxattr _header (57) in=7b,truncate 3731187] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813247+0000", 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.1628841869999995, 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1308570339999999, 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1536 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813247+0000", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813247+0000", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813249+0000", 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7940000000000001e-06 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813256+0000", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5830000000000003e-06 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813266+0000", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.027e-05 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813274+0000", 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3960000000000006e-06 2024-09-21T18:07:39.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850383+0000", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037108491 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850457+0000", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4060999999999994e-05 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.740999+0000", 2024-09-21T18:07:39.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.89054214300000001 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.741139+0000", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139949 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.741326+0000", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018674499999999999 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842265+0000", 2024-09-21T18:07:39.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100939685 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943936+0000", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101670576 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944013+0000", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7191e-05 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944104+0000", 2024-09-21T18:07:39.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1150000000000007e-05 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 8.1628458239999997, 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:39.085 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:07:39.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:39.090 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:39.091 INFO:tasks.rados.rados.0.smithi160.stdout:480: writing smithi16030472-480 from 450553 to 1025063 tid 1 2024-09-21T18:07:39.091 INFO:tasks.rados.rados.0.smithi160.stdout:480: writing smithi16030472-480 from 1613463 to 2050553 tid 2 2024-09-21T18:07:39.091 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.091 INFO:tasks.rados.rados.0.smithi160.stdout:467: finishing write tid 4 to smithi16030472-467 2024-09-21T18:07:39.091 INFO:tasks.rados.rados.0.smithi160.stdout:470: finishing write tid 1 to smithi16030472-470 2024-09-21T18:07:39.092 INFO:tasks.rados.rados.0.smithi160.stdout:470: finishing write tid 2 to smithi16030472-470 2024-09-21T18:07:39.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:39.093 INFO:tasks.rados.rados.0.smithi160.stdout:469: finishing write tid 2 to smithi16030472-469 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: finishing write tid 4 to smithi16030472-461 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: finishing write tid 5 to smithi16030472-461 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: oid 461 updating version 0 to 304 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: oid 461 updating version 304 to 306 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: oid 461 updating version 306 to 313 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: oid 461 version 313 is already newer than 305 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 461 v 313 (ObjNum 460 snap 0 seq_num 460) dirty exists 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: left oid 461 (ObjNum 460 snap 0 seq_num 460) 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:461: done (15 left) 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:481: write initial oid 481 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:481: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-21T18:07:39.094 INFO:tasks.rados.rados.0.smithi160.stdout:481: writing smithi16030472-481 from 467360 to 1117119 tid 1 2024-09-21T18:07:39.096 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:39.092+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:39.100 INFO:tasks.rados.rados.0.smithi160.stdout:481: writing smithi16030472-481 from 1671945 to 2067360 tid 2 2024-09-21T18:07:39.100 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.100 INFO:tasks.rados.rados.0.smithi160.stdout:467: finishing write tid 5 to smithi16030472-467 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: oid 467 updating version 0 to 347 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: oid 467 version 347 is already newer than 346 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: oid 467 updating version 347 to 349 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: oid 467 version 349 is already newer than 348 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 467 v 349 (ObjNum 466 snap 0 seq_num 466) dirty exists 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: left oid 467 (ObjNum 466 snap 0 seq_num 466) 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:467: done (15 left) 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:482: write initial oid 482 2024-09-21T18:07:39.101 INFO:tasks.rados.rados.0.smithi160.stdout:482: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-21T18:07:39.114 INFO:tasks.rados.rados.0.smithi160.stdout:482: writing smithi16030472-482 from 484167 to 1209175 tid 1 2024-09-21T18:07:39.119 INFO:tasks.rados.rados.0.smithi160.stdout:482: writing smithi16030472-482 from 1730427 to 2084167 tid 2 2024-09-21T18:07:39.119 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.119 INFO:tasks.rados.rados.0.smithi160.stdout:470: finishing write tid 3 to smithi16030472-470 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: finishing write tid 2 to smithi16030472-471 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: finishing write tid 3 to smithi16030472-471 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: finishing write tid 4 to smithi16030472-471 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: finishing write tid 5 to smithi16030472-471 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: oid 471 updating version 0 to 332 2024-09-21T18:07:39.161 INFO:tasks.rados.rados.0.smithi160.stdout:471: oid 471 updating version 332 to 334 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:471: oid 471 version 334 is already newer than 320 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:471: oid 471 version 334 is already newer than 333 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 471 v 334 (ObjNum 470 snap 0 seq_num 470) dirty exists 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:471: left oid 471 (ObjNum 470 snap 0 seq_num 470) 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:469: finishing write tid 3 to smithi16030472-469 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:471: done (15 left) 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:483: write initial oid 483 2024-09-21T18:07:39.162 INFO:tasks.rados.rados.0.smithi160.stdout:483: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-21T18:07:39.169 INFO:tasks.rados.rados.0.smithi160.stdout:483: writing smithi16030472-483 from 500974 to 901231 tid 1 2024-09-21T18:07:39.179 INFO:tasks.rados.rados.0.smithi160.stdout:483: writing smithi16030472-483 from 1388909 to 1946078 tid 2 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:479: finishing write tid 1 to smithi16030472-479 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: finishing write tid 4 to smithi16030472-469 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: finishing write tid 5 to smithi16030472-469 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: oid 469 updating version 0 to 331 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: oid 469 version 331 is already newer than 319 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: oid 469 updating version 331 to 335 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: oid 469 updating version 335 to 336 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 469 v 336 (ObjNum 468 snap 0 seq_num 468) dirty exists 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: left oid 469 (ObjNum 468 snap 0 seq_num 468) 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:469: done (15 left) 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:484: write initial oid 484 2024-09-21T18:07:39.180 INFO:tasks.rados.rados.0.smithi160.stdout:484: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-21T18:07:39.189 INFO:tasks.rados.rados.0.smithi160.stdout:484: writing smithi16030472-484 from 517781 to 993287 tid 1 2024-09-21T18:07:39.196 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:39.192+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:39.200 INFO:tasks.rados.rados.0.smithi160.stdout:484: writing smithi16030472-484 from 1731038 to 2117781 tid 2 2024-09-21T18:07:39.200 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.203 INFO:tasks.rados.rados.0.smithi160.stdout:470: finishing write tid 4 to smithi16030472-470 2024-09-21T18:07:39.203 INFO:tasks.rados.rados.0.smithi160.stdout:470: finishing write tid 5 to smithi16030472-470 2024-09-21T18:07:39.203 INFO:tasks.rados.rados.0.smithi160.stdout:470: oid 470 updating version 0 to 393 2024-09-21T18:07:39.203 INFO:tasks.rados.rados.0.smithi160.stdout:470: oid 470 version 393 is already newer than 391 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:470: oid 470 version 393 is already newer than 392 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:470: oid 470 version 393 is already newer than 390 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 470 v 393 (ObjNum 469 snap 0 seq_num 469) dirty exists 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:470: left oid 470 (ObjNum 469 snap 0 seq_num 469) 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:473: finishing write tid 1 to smithi16030472-473 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:473: finishing write tid 2 to smithi16030472-473 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:470: done (15 left) 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:485: write initial oid 485 2024-09-21T18:07:39.204 INFO:tasks.rados.rados.0.smithi160.stdout:485: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-21T18:07:39.214 INFO:tasks.rados.rados.0.smithi160.stdout:485: writing smithi16030472-485 from 534588 to 1085343 tid 1 2024-09-21T18:07:39.223 INFO:tasks.rados.rados.0.smithi160.stdout:485: writing smithi16030472-485 from 1789520 to 2134588 tid 2 2024-09-21T18:07:39.223 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.230 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:39.230 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:39.230 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:39.230 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:39.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:39.262 INFO:tasks.rados.rados.0.smithi160.stdout:476: finishing write tid 2 to smithi16030472-476 2024-09-21T18:07:39.267 INFO:tasks.rados.rados.0.smithi160.stdout:476: finishing write tid 3 to smithi16030472-476 2024-09-21T18:07:39.267 INFO:tasks.rados.rados.0.smithi160.stdout:476: finishing write tid 4 to smithi16030472-476 2024-09-21T18:07:39.308 INFO:tasks.rados.rados.0.smithi160.stdout:473: finishing write tid 3 to smithi16030472-473 2024-09-21T18:07:39.330 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:39.328+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:39.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:39.377 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: finishing write tid 4 to smithi16030472-473 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: finishing write tid 5 to smithi16030472-473 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: oid 473 updating version 0 to 395 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: oid 473 updating version 395 to 396 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: oid 473 updating version 396 to 397 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: oid 473 version 397 is already newer than 394 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 473 v 397 (ObjNum 472 snap 0 seq_num 472) dirty exists 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: left oid 473 (ObjNum 472 snap 0 seq_num 472) 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:473: done (15 left) 2024-09-21T18:07:39.387 INFO:tasks.rados.rados.0.smithi160.stdout:486: write initial oid 486 2024-09-21T18:07:39.388 INFO:tasks.rados.rados.0.smithi160.stdout:486: seq_num 485 ranges {551395=626004,1848002=303393} 2024-09-21T18:07:39.398 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:39.394+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:39.398 INFO:tasks.rados.rados.0.smithi160.stdout:486: writing smithi16030472-486 from 551395 to 1177399 tid 1 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:486: writing smithi16030472-486 from 1848002 to 2151395 tid 2 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:474: finishing write tid 1 to smithi16030472-474 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:476: finishing write tid 5 to smithi16030472-476 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:476: oid 476 updating version 0 to 339 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:476: oid 476 version 339 is already newer than 337 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:476: oid 476 version 339 is already newer than 321 2024-09-21T18:07:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:476: oid 476 version 339 is already newer than 338 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 476 v 339 (ObjNum 475 snap 0 seq_num 475) dirty exists 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:476: left oid 476 (ObjNum 475 snap 0 seq_num 475) 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:466: finishing write tid 1 to smithi16030472-466 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:466: finishing write tid 2 to smithi16030472-466 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:466: finishing write tid 3 to smithi16030472-466 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:476: done (15 left) 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:487: write initial oid 487 2024-09-21T18:07:39.408 INFO:tasks.rados.rados.0.smithi160.stdout:487: seq_num 486 ranges {568202=701253,1906484=261718} 2024-09-21T18:07:39.415 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:39.416 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:39.416 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:39.416 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:39.416 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:39.418 INFO:tasks.rados.rados.0.smithi160.stdout:487: writing smithi16030472-487 from 568202 to 1269455 tid 1 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout:487: writing smithi16030472-487 from 1906484 to 2168202 tid 2 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout:466: finishing write tid 4 to smithi16030472-466 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout:466: finishing write tid 5 to smithi16030472-466 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout:466: oid 466 updating version 0 to 310 2024-09-21T18:07:39.426 INFO:tasks.rados.rados.0.smithi160.stdout:466: oid 466 version 310 is already newer than 307 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:466: oid 466 version 310 is already newer than 309 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:466: oid 466 version 310 is already newer than 308 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 466 v 310 (ObjNum 465 snap 0 seq_num 465) dirty exists 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:466: left oid 466 (ObjNum 465 snap 0 seq_num 465) 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:468: finishing write tid 1 to smithi16030472-468 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:484: finishing write tid 1 to smithi16030472-484 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:466: done (15 left) 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:488: write initial oid 488 2024-09-21T18:07:39.427 INFO:tasks.rados.rados.0.smithi160.stdout:488: seq_num 487 ranges {585009=492855,1564966=620043} 2024-09-21T18:07:39.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:39.435 INFO:tasks.rados.rados.0.smithi160.stdout:488: writing smithi16030472-488 from 585009 to 1077864 tid 1 2024-09-21T18:07:39.445 INFO:tasks.rados.rados.0.smithi160.stdout:488: writing smithi16030472-488 from 1564966 to 2185009 tid 2 2024-09-21T18:07:39.445 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.445 INFO:tasks.rados.rados.0.smithi160.stdout:468: finishing write tid 2 to smithi16030472-468 2024-09-21T18:07:39.445 INFO:tasks.rados.rados.0.smithi160.stdout:468: finishing write tid 3 to smithi16030472-468 2024-09-21T18:07:39.445 INFO:tasks.rados.rados.0.smithi160.stdout:468: finishing write tid 4 to smithi16030472-468 2024-09-21T18:07:39.499 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:39.494+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:39.503 INFO:tasks.rados.rados.0.smithi160.stdout:472: finishing write tid 1 to smithi16030472-472 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: finishing write tid 2 to smithi16030472-472 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: finishing write tid 3 to smithi16030472-472 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: finishing write tid 4 to smithi16030472-472 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: finishing write tid 5 to smithi16030472-472 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: oid 472 updating version 0 to 353 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: oid 472 version 353 is already newer than 351 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: oid 472 version 353 is already newer than 350 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: oid 472 version 353 is already newer than 352 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 472 v 353 (ObjNum 471 snap 0 seq_num 471) dirty exists 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:472: left oid 472 (ObjNum 471 snap 0 seq_num 471) 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 1 to smithi16030472-475 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 2 to smithi16030472-475 2024-09-21T18:07:39.504 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 3 to smithi16030472-475 2024-09-21T18:07:39.505 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 4 to smithi16030472-475 2024-09-21T18:07:39.505 INFO:tasks.rados.rados.0.smithi160.stdout:478: finishing write tid 1 to smithi16030472-478 2024-09-21T18:07:39.505 INFO:tasks.rados.rados.0.smithi160.stdout:472: done (15 left) 2024-09-21T18:07:39.505 INFO:tasks.rados.rados.0.smithi160.stdout:489: write initial oid 489 2024-09-21T18:07:39.505 INFO:tasks.rados.rados.0.smithi160.stdout:489: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-21T18:07:39.513 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:39.513 INFO:tasks.rados.rados.0.smithi160.stdout:489: writing smithi16030472-489 from 601816 to 1169920 tid 1 2024-09-21T18:07:39.514 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:39.521 INFO:tasks.rados.rados.0.smithi160.stdout:489: writing smithi16030472-489 from 1623448 to 2075271 tid 2 2024-09-21T18:07:39.521 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.521 INFO:tasks.rados.rados.0.smithi160.stdout:478: finishing write tid 2 to smithi16030472-478 2024-09-21T18:07:39.521 INFO:tasks.rados.rados.0.smithi160.stdout:478: finishing write tid 3 to smithi16030472-478 2024-09-21T18:07:39.527 INFO:tasks.rados.rados.0.smithi160.stdout:479: finishing write tid 2 to smithi16030472-479 2024-09-21T18:07:39.527 INFO:tasks.rados.rados.0.smithi160.stdout:479: finishing write tid 3 to smithi16030472-479 2024-09-21T18:07:39.527 INFO:tasks.rados.rados.0.smithi160.stdout:479: finishing write tid 4 to smithi16030472-479 2024-09-21T18:07:39.527 INFO:tasks.rados.rados.0.smithi160.stdout:479: oid 479 updating version 0 to 322 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:479: oid 479 updating version 322 to 341 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:479: oid 479 version 341 is already newer than 340 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 479 v 341 (ObjNum 478 snap 0 seq_num 478) dirty exists 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:479: left oid 479 (ObjNum 478 snap 0 seq_num 478) 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:479: done (15 left) 2024-09-21T18:07:39.528 INFO:tasks.rados.rados.0.smithi160.stdout:490: write initial oid 490 2024-09-21T18:07:39.529 INFO:tasks.rados.rados.0.smithi160.stdout:490: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-21T18:07:39.529 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:39.524+0000 7ff409995640 -1 osd.4 53 handle_osd_map injecting map crc failure 2024-09-21T18:07:39.534 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:39.535 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:39.538 INFO:tasks.rados.rados.0.smithi160.stdout:490: writing smithi16030472-490 from 618623 to 1261976 tid 1 2024-09-21T18:07:39.547 INFO:tasks.rados.rados.0.smithi160.stdout:490: writing smithi16030472-490 from 1681930 to 2218623 tid 2 2024-09-21T18:07:39.547 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.548 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 5 to smithi16030472-475 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: finishing write tid 6 to smithi16030472-475 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: oid 475 updating version 0 to 356 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: oid 475 version 356 is already newer than 355 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: oid 475 updating version 356 to 357 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: oid 475 updating version 357 to 358 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: oid 475 version 358 is already newer than 354 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 475 v 358 (ObjNum 474 snap 0 seq_num 474) dirty exists 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: left oid 475 (ObjNum 474 snap 0 seq_num 474) 2024-09-21T18:07:39.549 INFO:tasks.rados.rados.0.smithi160.stdout:475: done (15 left) 2024-09-21T18:07:39.550 INFO:tasks.rados.rados.0.smithi160.stdout:491: write initial oid 491 2024-09-21T18:07:39.550 INFO:tasks.rados.rados.0.smithi160.stdout:491: seq_num 490 ranges {635430=718602,2140412=95018} 2024-09-21T18:07:39.555 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:39.562 INFO:tasks.rados.rados.0.smithi160.stdout:491: writing smithi16030472-491 from 635430 to 1354032 tid 1 2024-09-21T18:07:39.570 INFO:tasks.rados.rados.0.smithi160.stdout:491: writing smithi16030472-491 from 2140412 to 2235430 tid 2 2024-09-21T18:07:39.570 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.570 INFO:tasks.rados.rados.0.smithi160.stdout:477: finishing write tid 1 to smithi16030472-477 2024-09-21T18:07:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:39.576 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:39.597 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:39.600 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:39.594+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:39.605 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:39.605 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:39.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:39.620 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:39.620 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:39.641 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:39.641 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:39.661 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:39.671 INFO:tasks.rados.rados.0.smithi160.stdout:478: finishing write tid 4 to smithi16030472-478 2024-09-21T18:07:39.671 INFO:tasks.rados.rados.0.smithi160.stdout:478: oid 478 updating version 0 to 401 2024-09-21T18:07:39.671 INFO:tasks.rados.rados.0.smithi160.stdout:478: oid 478 version 401 is already newer than 400 2024-09-21T18:07:39.671 INFO:tasks.rados.rados.0.smithi160.stdout:478: oid 478 version 401 is already newer than 399 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 478 v 401 (ObjNum 477 snap 0 seq_num 477) dirty exists 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:478: left oid 478 (ObjNum 477 snap 0 seq_num 477) 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:481: finishing write tid 1 to smithi16030472-481 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: finishing write tid 2 to smithi16030472-477 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: finishing write tid 3 to smithi16030472-477 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: finishing write tid 4 to smithi16030472-477 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: oid 477 updating version 0 to 360 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: oid 477 updating version 360 to 361 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: oid 477 version 361 is already newer than 359 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 477 v 361 (ObjNum 476 snap 0 seq_num 476) dirty exists 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: left oid 477 (ObjNum 476 snap 0 seq_num 476) 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:477: done (15 left) 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:478: done (14 left) 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:492: write initial oid 492 2024-09-21T18:07:39.672 INFO:tasks.rados.rados.0.smithi160.stdout:492: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-21T18:07:39.680 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:39.680 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:39.683 INFO:tasks.rados.rados.0.smithi160.stdout:492: writing smithi16030472-492 from 652237 to 1446088 tid 1 2024-09-21T18:07:39.690 INFO:tasks.rados.rados.0.smithi160.stdout:492: writing smithi16030472-492 from 2198894 to 2252237 tid 2 2024-09-21T18:07:39.690 INFO:tasks.rados.rados.0.smithi160.stdout:493: write initial oid 493 2024-09-21T18:07:39.691 INFO:tasks.rados.rados.0.smithi160.stdout:493: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-21T18:07:39.701 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:39.701 INFO:tasks.rados.rados.0.smithi160.stdout:493: writing smithi16030472-493 from 669044 to 1138144 tid 1 2024-09-21T18:07:39.701 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:39.701 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:39.697+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:39.709 INFO:tasks.rados.rados.0.smithi160.stdout:493: writing smithi16030472-493 from 1741023 to 2200184 tid 2 2024-09-21T18:07:39.709 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.760 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:39.760 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:39.767 INFO:tasks.rados.rados.0.smithi160.stdout:482: finishing write tid 1 to smithi16030472-482 2024-09-21T18:07:39.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:39.801 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:39.798+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:39.803 INFO:tasks.rados.rados.0.smithi160.stdout:481: finishing write tid 2 to smithi16030472-481 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: finishing write tid 3 to smithi16030472-481 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: finishing write tid 4 to smithi16030472-481 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: oid 481 updating version 0 to 402 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: oid 481 updating version 402 to 403 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: oid 481 updating version 403 to 404 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 404 (ObjNum 480 snap 0 seq_num 480) dirty exists 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: left oid 481 (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:481: done (15 left) 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:494: write initial oid 494 2024-09-21T18:07:39.804 INFO:tasks.rados.rados.0.smithi160.stdout:494: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-21T18:07:39.814 INFO:tasks.rados.rados.0.smithi160.stdout:494: writing smithi16030472-494 from 685851 to 1230200 tid 1 2024-09-21T18:07:39.820 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:39.823 INFO:tasks.rados.rados.0.smithi160.stdout:494: writing smithi16030472-494 from 1799505 to 2285851 tid 2 2024-09-21T18:07:39.823 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.823 INFO:tasks.rados.rados.0.smithi160.stdout:474: finishing write tid 2 to smithi16030472-474 2024-09-21T18:07:39.823 INFO:tasks.rados.rados.0.smithi160.stdout:489: finishing write tid 1 to smithi16030472-489 2024-09-21T18:07:39.823 INFO:tasks.rados.rados.0.smithi160.stdout:489: finishing write tid 2 to smithi16030472-489 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: finishing write tid 3 to smithi16030472-489 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: finishing write tid 4 to smithi16030472-489 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: oid 489 updating version 0 to 405 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: oid 489 updating version 405 to 406 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: oid 489 updating version 406 to 407 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 489 v 407 (ObjNum 488 snap 0 seq_num 488) dirty exists 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: left oid 489 (ObjNum 488 snap 0 seq_num 488) 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:489: done (15 left) 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:495: write initial oid 495 2024-09-21T18:07:39.824 INFO:tasks.rados.rados.0.smithi160.stdout:495: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-21T18:07:39.836 INFO:tasks.rados.rados.0.smithi160.stdout:495: writing smithi16030472-495 from 702658 to 1322256 tid 1 2024-09-21T18:07:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:495: writing smithi16030472-495 from 1857987 to 2302658 tid 2 2024-09-21T18:07:39.846 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:495: finishing write tid 1 to smithi16030472-495 2024-09-21T18:07:39.847 INFO:tasks.rados.rados.0.smithi160.stdout:480: finishing write tid 1 to smithi16030472-480 2024-09-21T18:07:39.847 INFO:tasks.rados.rados.0.smithi160.stdout:480: finishing write tid 2 to smithi16030472-480 2024-09-21T18:07:39.847 INFO:tasks.rados.rados.0.smithi160.stdout:480: finishing write tid 3 to smithi16030472-480 2024-09-21T18:07:39.848 INFO:tasks.rados.rados.0.smithi160.stdout:480: finishing write tid 4 to smithi16030472-480 2024-09-21T18:07:39.848 INFO:tasks.rados.rados.0.smithi160.stdout:480: oid 480 updating version 0 to 364 2024-09-21T18:07:39.848 INFO:tasks.rados.rados.0.smithi160.stdout:480: oid 480 version 364 is already newer than 362 2024-09-21T18:07:39.848 INFO:tasks.rados.rados.0.smithi160.stdout:480: oid 480 version 364 is already newer than 363 2024-09-21T18:07:39.848 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 480 v 364 (ObjNum 479 snap 0 seq_num 479) dirty exists 2024-09-21T18:07:39.849 INFO:tasks.rados.rados.0.smithi160.stdout:480: left oid 480 (ObjNum 479 snap 0 seq_num 479) 2024-09-21T18:07:39.849 INFO:tasks.rados.rados.0.smithi160.stdout:480: done (15 left) 2024-09-21T18:07:39.849 INFO:tasks.rados.rados.0.smithi160.stdout:496: write initial oid 496 2024-09-21T18:07:39.849 INFO:tasks.rados.rados.0.smithi160.stdout:496: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-21T18:07:39.851 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:39.851 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:39.859 INFO:tasks.rados.rados.0.smithi160.stdout:496: writing smithi16030472-496 from 719465 to 1130665 tid 1 2024-09-21T18:07:39.870 INFO:tasks.rados.rados.0.smithi160.stdout:496: writing smithi16030472-496 from 1632822 to 2319465 tid 2 2024-09-21T18:07:39.870 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.870 INFO:tasks.rados.rados.0.smithi160.stdout:483: finishing write tid 1 to smithi16030472-483 2024-09-21T18:07:39.870 INFO:tasks.rados.rados.0.smithi160.stdout:483: finishing write tid 2 to smithi16030472-483 2024-09-21T18:07:39.870 INFO:tasks.rados.rados.0.smithi160.stdout:483: finishing write tid 3 to smithi16030472-483 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: finishing write tid 4 to smithi16030472-483 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: oid 483 updating version 0 to 366 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: oid 483 updating version 366 to 367 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: oid 483 version 367 is already newer than 365 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 483 v 367 (ObjNum 482 snap 0 seq_num 482) dirty exists 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: left oid 483 (ObjNum 482 snap 0 seq_num 482) 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:483: done (15 left) 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:497: write initial oid 497 2024-09-21T18:07:39.871 INFO:tasks.rados.rados.0.smithi160.stdout:497: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-21T18:07:39.871 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:39.880 INFO:tasks.rados.rados.0.smithi160.stdout:497: writing smithi16030472-497 from 736272 to 1222721 tid 1 2024-09-21T18:07:39.890 INFO:tasks.rados.rados.0.smithi160.stdout:497: writing smithi16030472-497 from 1974951 to 2336272 tid 2 2024-09-21T18:07:39.890 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.894 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:39.894 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:39.902 INFO:tasks.rados.rados.0.smithi160.stdout:474: finishing write tid 3 to smithi16030472-474 2024-09-21T18:07:39.903 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:39.900+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:39.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:39.917 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:39.947 INFO:tasks.rados.rados.0.smithi160.stdout:474: finishing write tid 4 to smithi16030472-474 2024-09-21T18:07:39.948 INFO:tasks.rados.rados.0.smithi160.stdout:474: finishing write tid 5 to smithi16030472-474 2024-09-21T18:07:39.948 INFO:tasks.rados.rados.0.smithi160.stdout:474: oid 474 updating version 0 to 401 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:474: oid 474 version 401 is already newer than 400 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:474: oid 474 version 401 is already newer than 398 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:474: oid 474 version 401 is already newer than 399 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 474 v 401 (ObjNum 473 snap 0 seq_num 473) dirty exists 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:474: left oid 474 (ObjNum 473 snap 0 seq_num 473) 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:474: done (15 left) 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:498: write initial oid 498 2024-09-21T18:07:39.949 INFO:tasks.rados.rados.0.smithi160.stdout:498: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-21T18:07:39.949 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:39.949 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:39.959 INFO:tasks.rados.rados.0.smithi160.stdout:498: writing smithi16030472-498 from 753079 to 1314777 tid 1 2024-09-21T18:07:39.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:39.967 INFO:tasks.rados.rados.0.smithi160.stdout:498: writing smithi16030472-498 from 2033433 to 2353079 tid 2 2024-09-21T18:07:39.967 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:39.973 INFO:tasks.rados.rados.0.smithi160.stdout:485: finishing write tid 1 to smithi16030472-485 2024-09-21T18:07:39.973 INFO:tasks.rados.rados.0.smithi160.stdout:487: finishing write tid 1 to smithi16030472-487 2024-09-21T18:07:39.973 INFO:tasks.rados.rados.0.smithi160.stdout:490: finishing write tid 1 to smithi16030472-490 2024-09-21T18:07:40.010 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:40.006+0000 7f66f041c640 -1 osd.7 53 handle_osd_map injecting map crc failure 2024-09-21T18:07:40.012 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:40.008+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:40.054 INFO:tasks.rados.rados.0.smithi160.stdout:468: finishing write tid 5 to smithi16030472-468 2024-09-21T18:07:40.054 INFO:tasks.rados.rados.0.smithi160.stdout:468: oid 468 updating version 0 to 316 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:468: oid 468 version 316 is already newer than 315 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:468: oid 468 version 316 is already newer than 314 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:468: oid 468 version 316 is already newer than 311 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 468 v 316 (ObjNum 467 snap 0 seq_num 467) dirty exists 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:468: left oid 468 (ObjNum 467 snap 0 seq_num 467) 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:468: done (15 left) 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:499: write initial oid 499 2024-09-21T18:07:40.055 INFO:tasks.rados.rados.0.smithi160.stdout:499: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-21T18:07:40.065 INFO:tasks.rados.rados.0.smithi160.stdout:499: writing smithi16030472-499 from 769886 to 1406833 tid 1 2024-09-21T18:07:40.074 INFO:tasks.rados.rados.0.smithi160.stdout:499: writing smithi16030472-499 from 2091915 to 2369886 tid 2 2024-09-21T18:07:40.074 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:40.082 INFO:tasks.rados.rados.0.smithi160.stdout:495: finishing write tid 2 to smithi16030472-495 2024-09-21T18:07:40.082 INFO:tasks.rados.rados.0.smithi160.stdout:495: finishing write tid 3 to smithi16030472-495 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:497: finishing write tid 1 to smithi16030472-497 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: finishing write tid 2 to smithi16030472-482 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: finishing write tid 3 to smithi16030472-482 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: finishing write tid 4 to smithi16030472-482 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: oid 482 updating version 0 to 323 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: oid 482 updating version 323 to 342 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: oid 482 updating version 342 to 343 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 482 v 343 (ObjNum 481 snap 0 seq_num 481) dirty exists 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: left oid 482 (ObjNum 481 snap 0 seq_num 481) 2024-09-21T18:07:40.085 INFO:tasks.rados.rados.0.smithi160.stdout:482: done (15 left) 2024-09-21T18:07:40.086 INFO:tasks.rados.rados.0.smithi160.stdout:500: write initial oid 500 2024-09-21T18:07:40.086 INFO:tasks.rados.rados.0.smithi160.stdout:500: seq_num 499 ranges {786693=712196,2150397=236296} 2024-09-21T18:07:40.094 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:40.099 INFO:tasks.rados.rados.0.smithi160.stdout:500: writing smithi16030472-500 from 786693 to 1498889 tid 1 2024-09-21T18:07:40.104 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:40.098+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:40.107 INFO:tasks.rados.rados.0.smithi160.stdout:500: writing smithi16030472-500 from 2150397 to 2386693 tid 2 2024-09-21T18:07:40.107 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:40.152 INFO:tasks.rados.rados.0.smithi160.stdout:484: finishing write tid 2 to smithi16030472-484 2024-09-21T18:07:40.161 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.161 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:40.176 INFO:tasks.rados.rados.0.smithi160.stdout:497: finishing write tid 2 to smithi16030472-497 2024-09-21T18:07:40.194 INFO:tasks.rados.rados.0.smithi160.stdout:495: finishing write tid 4 to smithi16030472-495 2024-09-21T18:07:40.194 INFO:tasks.rados.rados.0.smithi160.stdout:495: oid 495 updating version 0 to 408 2024-09-21T18:07:40.194 INFO:tasks.rados.rados.0.smithi160.stdout:495: oid 495 updating version 408 to 409 2024-09-21T18:07:40.194 INFO:tasks.rados.rados.0.smithi160.stdout:495: oid 495 updating version 409 to 410 2024-09-21T18:07:40.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 410 (ObjNum 494 snap 0 seq_num 494) dirty exists 2024-09-21T18:07:40.195 INFO:tasks.rados.rados.0.smithi160.stdout:495: left oid 495 (ObjNum 494 snap 0 seq_num 494) 2024-09-21T18:07:40.195 INFO:tasks.rados.rados.0.smithi160.stdout:495: done (15 left) 2024-09-21T18:07:40.195 INFO:tasks.rados.rados.0.smithi160.stdout:501: cache_evict oid 208 snap -1 2024-09-21T18:07:40.195 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:40.223 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:40.220+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:40.295 INFO:tasks.rados.rados.0.smithi160.stdout:486: finishing write tid 1 to smithi16030472-486 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: finishing write tid 2 to smithi16030472-486 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: finishing write tid 3 to smithi16030472-486 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: finishing write tid 4 to smithi16030472-486 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: oid 486 updating version 0 to 368 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: oid 486 updating version 368 to 369 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: oid 486 updating version 369 to 370 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 486 v 370 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: left oid 486 (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:491: finishing write tid 1 to smithi16030472-491 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:491: finishing write tid 2 to smithi16030472-491 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:485: finishing write tid 2 to smithi16030472-485 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:486: done (15 left) 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:502: write oid 38 current snap is 0 2024-09-21T18:07:40.296 INFO:tasks.rados.rados.0.smithi160.stdout:502: seq_num 500 ranges {403500=787445,1692526=473837} 2024-09-21T18:07:40.303 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.303 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.306 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:40.301+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:40.310 INFO:tasks.rados.rados.0.smithi160.stdout:502: writing smithi16030472-38 from 403500 to 1190945 tid 1 2024-09-21T18:07:40.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:40.322 INFO:tasks.rados.rados.0.smithi160.stdout:502: writing smithi16030472-38 from 1692526 to 2166363 tid 2 2024-09-21T18:07:40.322 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: finishing write tid 3 to smithi16030472-485 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: finishing write tid 4 to smithi16030472-485 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: oid 485 updating version 0 to 345 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: oid 485 version 345 is already newer than 344 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: oid 485 version 345 is already newer than 324 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 485 v 345 (ObjNum 484 snap 0 seq_num 484) dirty exists 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: left oid 485 (ObjNum 484 snap 0 seq_num 484) 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:501: got (2) No such file or directory 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:500: finishing write tid 1 to smithi16030472-500 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:500: finishing write tid 2 to smithi16030472-500 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:485: done (15 left) 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:501: done (14 left) 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:503: cache_try_flush oid 385 snap -1 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:500: finishing write tid 3 to smithi16030472-500 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:500: finishing write tid 4 to smithi16030472-500 2024-09-21T18:07:40.323 INFO:tasks.rados.rados.0.smithi160.stdout:500: oid 500 updating version 0 to 413 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:500: oid 500 version 413 is already newer than 411 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:500: oid 500 version 413 is already newer than 412 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 500 v 413 (ObjNum 499 snap 0 seq_num 499) dirty exists 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:500: left oid 500 (ObjNum 499 snap 0 seq_num 499) 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:500: done (14 left) 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:504: delete oid 33 current snap is 0 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:503: got (0) Success 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 0 (ObjNum 384 snap 0 seq_num 384) dirty exists 2024-09-21T18:07:40.324 INFO:tasks.rados.rados.0.smithi160.stdout:491: finishing write tid 3 to smithi16030472-491 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: finishing write tid 2 to smithi16030472-487 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: finishing write tid 3 to smithi16030472-487 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: finishing write tid 4 to smithi16030472-487 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: oid 487 updating version 0 to 346 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: oid 487 version 346 is already newer than 325 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:487: oid 487 updating version 346 to 347 2024-09-21T18:07:40.393 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 487 v 347 (ObjNum 486 snap 0 seq_num 486) dirty exists 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:487: left oid 487 (ObjNum 486 snap 0 seq_num 486) 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: finishing write tid 3 to smithi16030472-497 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: finishing write tid 4 to smithi16030472-497 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: oid 497 updating version 0 to 328 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: oid 497 version 328 is already newer than 327 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: oid 497 updating version 328 to 348 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 497 v 348 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:07:40.394 INFO:tasks.rados.rados.0.smithi160.stdout:497: left oid 497 (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:07:40.397 INFO:tasks.rados.rados.0.smithi160.stdout:490: finishing write tid 2 to smithi16030472-490 2024-09-21T18:07:40.407 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:40.404+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:40.424 INFO:tasks.rados.rados.0.smithi160.stdout:491: finishing write tid 4 to smithi16030472-491 2024-09-21T18:07:40.425 INFO:tasks.rados.rados.0.smithi160.stdout:491: oid 491 updating version 0 to 373 2024-09-21T18:07:40.425 INFO:tasks.rados.rados.0.smithi160.stdout:491: oid 491 version 373 is already newer than 372 2024-09-21T18:07:40.425 INFO:tasks.rados.rados.0.smithi160.stdout:491: oid 491 version 373 is already newer than 371 2024-09-21T18:07:40.425 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 373 (ObjNum 490 snap 0 seq_num 490) dirty exists 2024-09-21T18:07:40.425 INFO:tasks.rados.rados.0.smithi160.stdout:491: left oid 491 (ObjNum 490 snap 0 seq_num 490) 2024-09-21T18:07:40.462 INFO:tasks.rados.rados.0.smithi160.stdout:484: finishing write tid 3 to smithi16030472-484 2024-09-21T18:07:40.462 INFO:tasks.rados.rados.0.smithi160.stdout:484: finishing write tid 4 to smithi16030472-484 2024-09-21T18:07:40.462 INFO:tasks.rados.rados.0.smithi160.stdout:484: oid 484 updating version 0 to 313 2024-09-21T18:07:40.462 INFO:tasks.rados.rados.0.smithi160.stdout:484: oid 484 version 313 is already newer than 312 2024-09-21T18:07:40.462 INFO:tasks.rados.rados.0.smithi160.stdout:484: oid 484 updating version 313 to 314 2024-09-21T18:07:40.463 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 314 (ObjNum 483 snap 0 seq_num 483) dirty exists 2024-09-21T18:07:40.463 INFO:tasks.rados.rados.0.smithi160.stdout:484: left oid 484 (ObjNum 483 snap 0 seq_num 483) 2024-09-21T18:07:40.464 INFO:tasks.rados.rados.0.smithi160.stdout:488: finishing write tid 1 to smithi16030472-488 2024-09-21T18:07:40.488 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.488 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: finishing write tid 3 to smithi16030472-490 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: finishing write tid 4 to smithi16030472-490 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: oid 490 updating version 0 to 349 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: oid 490 version 349 is already newer than 326 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: oid 490 updating version 349 to 350 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 490 v 350 (ObjNum 489 snap 0 seq_num 489) dirty exists 2024-09-21T18:07:40.494 INFO:tasks.rados.rados.0.smithi160.stdout:490: left oid 490 (ObjNum 489 snap 0 seq_num 489) 2024-09-21T18:07:40.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:40.508 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:40.502+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:40.528 INFO:tasks.rados.rados.0.smithi160.stdout:494: finishing write tid 1 to smithi16030472-494 2024-09-21T18:07:40.528 INFO:tasks.rados.rados.0.smithi160.stdout:494: finishing write tid 2 to smithi16030472-494 2024-09-21T18:07:40.528 INFO:tasks.rados.rados.0.smithi160.stdout:494: finishing write tid 3 to smithi16030472-494 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:494: finishing write tid 4 to smithi16030472-494 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:494: oid 494 updating version 0 to 375 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:494: oid 494 updating version 375 to 376 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:494: oid 494 version 376 is already newer than 374 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 376 (ObjNum 493 snap 0 seq_num 493) dirty exists 2024-09-21T18:07:40.529 INFO:tasks.rados.rados.0.smithi160.stdout:494: left oid 494 (ObjNum 493 snap 0 seq_num 493) 2024-09-21T18:07:40.530 INFO:tasks.rados.rados.0.smithi160.stdout:496: finishing write tid 1 to smithi16030472-496 2024-09-21T18:07:40.530 INFO:tasks.rados.rados.0.smithi160.stdout:496: finishing write tid 2 to smithi16030472-496 2024-09-21T18:07:40.530 INFO:tasks.rados.rados.0.smithi160.stdout:496: finishing write tid 3 to smithi16030472-496 2024-09-21T18:07:40.530 INFO:tasks.rados.rados.0.smithi160.stdout:496: finishing write tid 4 to smithi16030472-496 2024-09-21T18:07:40.531 INFO:tasks.rados.rados.0.smithi160.stdout:496: oid 496 updating version 0 to 378 2024-09-21T18:07:40.531 INFO:tasks.rados.rados.0.smithi160.stdout:496: oid 496 updating version 378 to 379 2024-09-21T18:07:40.531 INFO:tasks.rados.rados.0.smithi160.stdout:496: oid 496 version 379 is already newer than 377 2024-09-21T18:07:40.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 379 (ObjNum 495 snap 0 seq_num 495) dirty exists 2024-09-21T18:07:40.531 INFO:tasks.rados.rados.0.smithi160.stdout:496: left oid 496 (ObjNum 495 snap 0 seq_num 495) 2024-09-21T18:07:40.563 INFO:tasks.rados.rados.0.smithi160.stdout:502: finishing write tid 1 to smithi16030472-38 2024-09-21T18:07:40.609 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:40.605+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:484: done (14 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:487: done (13 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:490: done (12 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:491: done (11 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:494: done (10 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:496: done (9 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:497: done (8 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:503: done (7 left) 2024-09-21T18:07:40.628 INFO:tasks.rados.rados.0.smithi160.stdout:504: done (6 left) 2024-09-21T18:07:40.629 INFO:tasks.rados.rados.0.smithi160.stdout:505: cache_evict oid 357 snap -1 2024-09-21T18:07:40.629 INFO:tasks.rados.rados.0.smithi160.stdout:506: write oid 417 current snap is 0 2024-09-21T18:07:40.629 INFO:tasks.rados.rados.0.smithi160.stdout:506: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-21T18:07:40.644 INFO:tasks.rados.rados.0.smithi160.stdout:506: writing smithi16030472-417 from 437114 to 1091410 tid 1 2024-09-21T18:07:40.658 INFO:tasks.rados.rados.0.smithi160.stdout:506: writing smithi16030472-417 from 1525843 to 2003349 tid 2 2024-09-21T18:07:40.658 INFO:tasks.rados.rados.0.smithi160.stdout:507: delete oid 96 current snap is 0 2024-09-21T18:07:40.659 INFO:tasks.rados.rados.0.smithi160.stdout:505: got (0) Success 2024-09-21T18:07:40.664 INFO:tasks.rados.rados.0.smithi160.stdout:502: finishing write tid 2 to smithi16030472-38 2024-09-21T18:07:40.664 INFO:tasks.rados.rados.0.smithi160.stdout:502: finishing write tid 3 to smithi16030472-38 2024-09-21T18:07:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:488: finishing write tid 2 to smithi16030472-488 2024-09-21T18:07:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:502: finishing write tid 4 to smithi16030472-38 2024-09-21T18:07:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:502: oid 38 updating version 0 to 318 2024-09-21T18:07:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:502: oid 38 updating version 318 to 319 2024-09-21T18:07:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:502: oid 38 version 319 is already newer than 317 2024-09-21T18:07:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 319 (ObjNum 500 snap 0 seq_num 500) dirty exists 2024-09-21T18:07:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:502: left oid 38 (ObjNum 500 snap 0 seq_num 500) 2024-09-21T18:07:40.669 INFO:tasks.rados.rados.0.smithi160.stdout:488: finishing write tid 3 to smithi16030472-488 2024-09-21T18:07:40.672 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.672 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.676 INFO:tasks.rados.rados.0.smithi160.stdout:492: finishing write tid 1 to smithi16030472-492 2024-09-21T18:07:40.677 INFO:tasks.rados.rados.0.smithi160.stdout:493: finishing write tid 1 to smithi16030472-493 2024-09-21T18:07:40.677 INFO:tasks.rados.rados.0.smithi160.stdout:498: finishing write tid 1 to smithi16030472-498 2024-09-21T18:07:40.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:40.688 INFO:tasks.rados.rados.0.smithi160.stdout:498: finishing write tid 2 to smithi16030472-498 2024-09-21T18:07:40.690 INFO:tasks.rados.rados.0.smithi160.stdout:498: finishing write tid 3 to smithi16030472-498 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:506: finishing write tid 1 to smithi16030472-417 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:498: finishing write tid 4 to smithi16030472-498 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:498: oid 498 updating version 0 to 322 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:498: oid 498 version 322 is already newer than 320 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:498: oid 498 version 322 is already newer than 321 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 498 v 322 (ObjNum 497 snap 0 seq_num 497) dirty exists 2024-09-21T18:07:40.691 INFO:tasks.rados.rados.0.smithi160.stdout:498: left oid 498 (ObjNum 497 snap 0 seq_num 497) 2024-09-21T18:07:40.694 INFO:tasks.rados.rados.0.smithi160.stdout:499: finishing write tid 1 to smithi16030472-499 2024-09-21T18:07:40.695 INFO:tasks.rados.rados.0.smithi160.stdout:499: finishing write tid 2 to smithi16030472-499 2024-09-21T18:07:40.709 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:40.705+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:40.810 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:40.806+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:40.814 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.815 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.819 INFO:tasks.rados.rados.0.smithi160.stdout:499: finishing write tid 3 to smithi16030472-499 2024-09-21T18:07:40.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:40.896 INFO:tasks.rados.rados.0.smithi160.stdout:488: finishing write tid 4 to smithi16030472-488 2024-09-21T18:07:40.896 INFO:tasks.rados.rados.0.smithi160.stdout:488: oid 488 updating version 0 to 316 2024-09-21T18:07:40.896 INFO:tasks.rados.rados.0.smithi160.stdout:488: oid 488 updating version 316 to 317 2024-09-21T18:07:40.897 INFO:tasks.rados.rados.0.smithi160.stdout:488: oid 488 version 317 is already newer than 315 2024-09-21T18:07:40.897 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 488 v 317 (ObjNum 487 snap 0 seq_num 487) dirty exists 2024-09-21T18:07:40.897 INFO:tasks.rados.rados.0.smithi160.stdout:488: left oid 488 (ObjNum 487 snap 0 seq_num 487) 2024-09-21T18:07:40.911 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:40.906+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:40.950 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:40.950 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:40.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:40.984 INFO:tasks.rados.rados.0.smithi160.stdout:506: finishing write tid 2 to smithi16030472-417 2024-09-21T18:07:40.997 INFO:tasks.rados.rados.0.smithi160.stdout:492: finishing write tid 2 to smithi16030472-492 2024-09-21T18:07:40.997 INFO:tasks.rados.rados.0.smithi160.stdout:492: finishing write tid 3 to smithi16030472-492 2024-09-21T18:07:40.999 INFO:tasks.rados.rados.0.smithi160.stdout:492: finishing write tid 4 to smithi16030472-492 2024-09-21T18:07:40.999 INFO:tasks.rados.rados.0.smithi160.stdout:492: oid 492 updating version 0 to 322 2024-09-21T18:07:40.999 INFO:tasks.rados.rados.0.smithi160.stdout:492: oid 492 version 322 is already newer than 318 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:492: oid 492 updating version 322 to 323 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 492 v 323 (ObjNum 491 snap 0 seq_num 491) dirty exists 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:492: left oid 492 (ObjNum 491 snap 0 seq_num 491) 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:493: finishing write tid 2 to smithi16030472-493 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:493: finishing write tid 3 to smithi16030472-493 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:493: finishing write tid 4 to smithi16030472-493 2024-09-21T18:07:41.000 INFO:tasks.rados.rados.0.smithi160.stdout:493: oid 493 updating version 0 to 319 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:493: oid 493 updating version 319 to 325 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:493: oid 493 version 325 is already newer than 324 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 493 v 325 (ObjNum 492 snap 0 seq_num 492) dirty exists 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:493: left oid 493 (ObjNum 492 snap 0 seq_num 492) 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:506: finishing write tid 3 to smithi16030472-417 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:506: finishing write tid 4 to smithi16030472-417 2024-09-21T18:07:41.001 INFO:tasks.rados.rados.0.smithi160.stdout:506: oid 417 updating version 0 to 326 2024-09-21T18:07:41.002 INFO:tasks.rados.rados.0.smithi160.stdout:506: oid 417 version 326 is already newer than 320 2024-09-21T18:07:41.002 INFO:tasks.rados.rados.0.smithi160.stdout:506: oid 417 version 326 is already newer than 321 2024-09-21T18:07:41.002 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 326 (ObjNum 502 snap 0 seq_num 502) dirty exists 2024-09-21T18:07:41.002 INFO:tasks.rados.rados.0.smithi160.stdout:506: left oid 417 (ObjNum 502 snap 0 seq_num 502) 2024-09-21T18:07:41.012 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:41.010+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:488: done (8 left) 2024-09-21T18:07:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:492: done (7 left) 2024-09-21T18:07:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:493: done (6 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:498: done (5 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:502: done (4 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:505: done (3 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:506: done (2 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:507: done (1 left) 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:508: write oid 424 current snap is 0 2024-09-21T18:07:41.086 INFO:tasks.rados.rados.0.smithi160.stdout:508: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-21T18:07:41.092 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.092 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.099 INFO:tasks.rados.rados.0.smithi160.stdout:508: writing smithi16030472-424 from 470728 to 875522 tid 1 2024-09-21T18:07:41.107 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:41.113 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:41.108+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:41.118 INFO:tasks.rados.rados.0.smithi160.stdout:508: writing smithi16030472-424 from 1526454 to 2007629 tid 2 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:509: cache_try_flush oid 2 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:510: cache_flush oid 52 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:511: cache_try_flush oid 30 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:512: cache_flush oid 250 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:513: cache_evict oid 185 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:514: cache_flush oid 179 snap -1 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:515: write oid 194 current snap is 0 2024-09-21T18:07:41.119 INFO:tasks.rados.rados.0.smithi160.stdout:515: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-21T18:07:41.133 INFO:tasks.rados.rados.0.smithi160.stdout:515: writing smithi16030472-194 from 487535 to 967578 tid 1 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:515: writing smithi16030472-194 from 1584936 to 2209769 tid 2 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:516: cache_evict oid 6 snap -1 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:514: got (0) Success 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 0 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:514: done (9 left) 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:517: copy_from oid 405 from oid 73 current snap is 0 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:518: read oid 485 snap -1 2024-09-21T18:07:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:518: expect (ObjNum 484 snap 0 seq_num 484) 2024-09-21T18:07:41.151 INFO:tasks.rados.rados.0.smithi160.stdout:519: cache_flush oid 222 snap -1 2024-09-21T18:07:41.151 INFO:tasks.rados.rados.0.smithi160.stdout:520: write oid 348 current snap is 0 2024-09-21T18:07:41.151 INFO:tasks.rados.rados.0.smithi160.stdout:520: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-21T18:07:41.161 INFO:tasks.rados.rados.0.smithi160.stdout:520: writing smithi16030472-348 from 504342 to 1059634 tid 1 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout:520: writing smithi16030472-348 from 1643418 to 2128262 tid 2 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout:521: copy_from oid 349 from oid 472 current snap is 0 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout:522: cache_flush oid 286 snap -1 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout:523: read oid 215 snap -1 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout:523: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-21T18:07:41.173 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:41.201 INFO:tasks.rados.rados.0.smithi160.stdout:499: finishing write tid 4 to smithi16030472-499 2024-09-21T18:07:41.201 INFO:tasks.rados.rados.0.smithi160.stdout:499: oid 499 updating version 0 to 325 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:499: oid 499 version 325 is already newer than 324 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:499: oid 499 version 325 is already newer than 323 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 325 (ObjNum 498 snap 0 seq_num 498) dirty exists 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:499: left oid 499 (ObjNum 498 snap 0 seq_num 498) 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:499: done (15 left) 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:524: read oid 137 snap -1 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout:524: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-21T18:07:41.202 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:41.214 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:41.208+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:41.242 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.242 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.251 INFO:tasks.rados.rados.0.smithi160.stdout:516: got (16) Device or resource busy 2024-09-21T18:07:41.251 INFO:tasks.rados.rados.0.smithi160.stdout:516: done (15 left) 2024-09-21T18:07:41.251 INFO:tasks.rados.rados.0.smithi160.stdout:525: write (excl) oid 367 current snap is 0 2024-09-21T18:07:41.251 INFO:tasks.rados.rados.0.smithi160.stdout:525: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-21T18:07:41.257 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:41.266 INFO:tasks.rados.rados.0.smithi160.stdout:525: writing smithi16030472-367 from 521149 to 1151690 tid 1 2024-09-21T18:07:41.281 INFO:tasks.rados.rados.0.smithi160.stdout:525: writing smithi16030472-367 from 1701900 to 2330402 tid 2 2024-09-21T18:07:41.282 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:41.315 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:41.313+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:41.359 INFO:tasks.rados.rados.0.smithi160.stdout:518: expect (ObjNum 484 snap 0 seq_num 484) 2024-09-21T18:07:41.367 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1228958 2024-09-21T18:07:41.384 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.384 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:07:41.400 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:41.416 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:41.410+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:41.477 INFO:tasks.rados.rados.0.smithi160.stdout:511: got (2) No such file or directory 2024-09-21T18:07:41.477 INFO:tasks.rados.rados.0.smithi160.stdout:513: got (2) No such file or directory 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:522: got (0) Success 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 286 v 0 (ObjNum 285 snap 0 seq_num 285) dirty exists 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:519: got (2) No such file or directory 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:520: finishing write tid 1 to smithi16030472-348 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:520: finishing write tid 2 to smithi16030472-348 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:511: done (15 left) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:513: done (14 left) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:518: done (13 left) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:519: done (12 left) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:522: done (11 left) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:526: read oid 380 snap -1 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:526: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:527: cache_evict oid 446 snap -1 2024-09-21T18:07:41.478 INFO:tasks.rados.rados.0.smithi160.stdout:528: delete oid 130 current snap is 0 2024-09-21T18:07:41.517 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:41.512+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:41.528 INFO:tasks.rados.rados.0.smithi160.stdout:524: expect (ObjNum 136 snap 0 seq_num 136) 2024-09-21T18:07:41.567 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.567 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.583 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:41.618 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:41.614+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:41.655 INFO:tasks.rados.rados.0.smithi160.stdout:515: finishing write tid 1 to smithi16030472-194 2024-09-21T18:07:41.655 INFO:tasks.rados.rados.0.smithi160.stdout:515: finishing write tid 2 to smithi16030472-194 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:524: done (13 left) 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:528: done (12 left) 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:529: copy_from oid 328 from oid 22 current snap is 0 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: finishing write tid 3 to smithi16030472-194 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: finishing write tid 4 to smithi16030472-194 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: oid 194 updating version 0 to 383 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: oid 194 version 383 is already newer than 382 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: oid 194 version 383 is already newer than 381 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 194 v 383 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:515: left oid 194 (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:07:41.656 INFO:tasks.rados.rados.0.smithi160.stdout:523: expect (ObjNum 214 snap 0 seq_num 214) 2024-09-21T18:07:41.707 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2731424 2024-09-21T18:07:41.714 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.714 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.719 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:41.714+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:41.729 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:41.738 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3443244 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:521: finishing copy_from racing read to smithi16030472-349 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: finishing write tid 3 to smithi16030472-348 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: finishing write tid 4 to smithi16030472-348 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: oid 348 updating version 0 to 351 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: oid 348 updating version 351 to 353 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: oid 348 version 353 is already newer than 352 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 348 v 353 (ObjNum 506 snap 0 seq_num 506) dirty exists 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:520: left oid 348 (ObjNum 506 snap 0 seq_num 506) 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:527: got (16) Device or resource busy 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:521: finishing copy_from to smithi16030472-349 2024-09-21T18:07:41.741 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 349 v 353 (ObjNum 471 snap 0 seq_num 471) dirty exists 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: finishing write tid 1 to smithi16030472-424 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: finishing write tid 2 to smithi16030472-424 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: finishing write tid 3 to smithi16030472-424 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: finishing write tid 4 to smithi16030472-424 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: oid 424 updating version 0 to 326 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: oid 424 updating version 326 to 328 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: oid 424 version 328 is already newer than 327 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 328 (ObjNum 504 snap 0 seq_num 504) dirty exists 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: left oid 424 (ObjNum 504 snap 0 seq_num 504) 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:508: done (12 left) 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:515: done (11 left) 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:520: done (10 left) 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:521: done (9 left) 2024-09-21T18:07:41.742 INFO:tasks.rados.rados.0.smithi160.stdout:523: done (8 left) 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:527: done (7 left) 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:530: read oid 267 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:530: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:531: cache_evict oid 496 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:532: cache_evict oid 179 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:533: cache_evict oid 66 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:534: cache_evict oid 455 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:535: read oid 281 snap -1 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:535: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:536: write oid 212 current snap is 0 2024-09-21T18:07:41.743 INFO:tasks.rados.rados.0.smithi160.stdout:536: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-21T18:07:41.763 INFO:tasks.rados.rados.0.smithi160.stdout:536: writing smithi16030472-212 from 554763 to 1335802 tid 1 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:536: writing smithi16030472-212 from 2102511 to 2554763 tid 2 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:537: read oid 263 snap -1 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:537: expect (ObjNum 262 snap 0 seq_num 262) 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:538: write (excl) oid 329 current snap is 0 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:533: got (2) No such file or directory 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:534: got (16) Device or resource busy 2024-09-21T18:07:41.784 INFO:tasks.rados.rados.0.smithi160.stdout:509: got (0) Success 2024-09-21T18:07:41.785 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 2 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T18:07:41.785 INFO:tasks.rados.rados.0.smithi160.stdout:531: got (0) Success 2024-09-21T18:07:41.785 INFO:tasks.rados.rados.0.smithi160.stdout:538: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-21T18:07:41.802 INFO:tasks.rados.rados.0.smithi160.stdout:538: writing smithi16030472-329 from 571570 to 1144211 tid 1 2024-09-21T18:07:41.820 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:41.818+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:41.822 INFO:tasks.rados.rados.0.smithi160.stdout:538: writing smithi16030472-329 from 1877346 to 2571570 tid 2 2024-09-21T18:07:41.822 INFO:tasks.rados.rados.0.smithi160.stdout:509: done (15 left) 2024-09-21T18:07:41.822 INFO:tasks.rados.rados.0.smithi160.stdout:531: done (14 left) 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:533: done (13 left) 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:534: done (12 left) 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:539: read oid 364 snap -1 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:539: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:540: delete oid 454 current snap is 0 2024-09-21T18:07:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:526: expect (ObjNum 379 snap 0 seq_num 379) 2024-09-21T18:07:41.853 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.853 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:41.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:41.921 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:41.917+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:529: finishing copy_from racing read to smithi16030472-328 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:529: finishing copy_from to smithi16030472-328 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 27 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:532: got (0) Success 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:526: done (13 left) 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:529: done (12 left) 2024-09-21T18:07:41.933 INFO:tasks.rados.rados.0.smithi160.stdout:532: done (11 left) 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:540: done (10 left) 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:541: cache_try_flush oid 155 snap -1 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:542: copy_from oid 18 from oid 236 current snap is 0 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:543: copy_from oid 92 from oid 258 current snap is 0 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:544: cache_evict oid 148 snap -1 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:545: read oid 475 snap -1 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:545: expect (ObjNum 474 snap 0 seq_num 474) 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:546: cache_flush oid 377 snap -1 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:41.934 INFO:tasks.rados.rados.0.smithi160.stdout:544: got (2) No such file or directory 2024-09-21T18:07:41.935 INFO:tasks.rados.rados.0.smithi160.stdout:544: done (15 left) 2024-09-21T18:07:41.935 INFO:tasks.rados.rados.0.smithi160.stdout:547: cache_flush oid 499 snap -1 2024-09-21T18:07:41.935 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:41.952 INFO:tasks.rados.rados.0.smithi160.stdout:535: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-21T18:07:41.963 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1915843 2024-09-21T18:07:41.971 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 399888 2024-09-21T18:07:41.992 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:41.993 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:42.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:42.009 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2487349 2024-09-21T18:07:42.017 INFO:tasks.rados.rados.0.smithi160.stdout:535: done (15 left) 2024-09-21T18:07:42.017 INFO:tasks.rados.rados.0.smithi160.stdout:548: cache_try_flush oid 36 snap -1 2024-09-21T18:07:42.017 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:42.034 INFO:tasks.rados.rados.0.smithi160.stdout:525: finishing write tid 1 to smithi16030472-367 2024-09-21T18:07:42.034 INFO:tasks.rados.rados.0.smithi160.stdout:525: finishing write tid 2 to smithi16030472-367 2024-09-21T18:07:42.055 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:42.053+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:42.071 INFO:tasks.rados.rados.0.smithi160.stdout:543: finishing copy_from racing read to smithi16030472-92 2024-09-21T18:07:42.084 INFO:tasks.rados.rados.0.smithi160.stdout:530: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:07:42.084 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:42.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:42.122 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:42.118+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:42.162 INFO:tasks.rados.rados.0.smithi160.stdout:543: finishing copy_from to smithi16030472-92 2024-09-21T18:07:42.162 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 92 v 221 (ObjNum 257 snap 0 seq_num 257) dirty exists 2024-09-21T18:07:42.163 INFO:tasks.rados.rados.0.smithi160.stdout:546: got (0) Success 2024-09-21T18:07:42.163 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 377 v 0 (ObjNum 376 snap 0 seq_num 376) dirty exists 2024-09-21T18:07:42.163 INFO:tasks.rados.rados.0.smithi160.stdout:545: expect (ObjNum 474 snap 0 seq_num 474) 2024-09-21T18:07:42.181 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3434963 2024-09-21T18:07:42.187 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:42.188 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:42.202 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:42.208 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3421156 2024-09-21T18:07:42.223 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:42.218+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:42.231 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3139432 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:517: finishing copy_from racing read to smithi16030472-405 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:530: done (15 left) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:543: done (14 left) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:545: done (13 left) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:546: done (12 left) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:549: cache_evict oid 282 snap -1 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:525: finishing write tid 3 to smithi16030472-367 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:541: got (2) No such file or directory 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:541: done (12 left) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:550: read oid 92 snap -1 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:550: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:551: write (excl) oid 435 current snap is 0 2024-09-21T18:07:42.241 INFO:tasks.rados.rados.0.smithi160.stdout:551: seq_num 512 ranges {605184=723139,1877957=727227} 2024-09-21T18:07:42.262 INFO:tasks.rados.rados.0.smithi160.stdout:551: writing smithi16030472-435 from 605184 to 1328323 tid 1 2024-09-21T18:07:42.281 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:42.283 INFO:tasks.rados.rados.0.smithi160.stdout:551: writing smithi16030472-435 from 1877957 to 2605184 tid 2 2024-09-21T18:07:42.283 INFO:tasks.rados.rados.0.smithi160.stdout:552: delete oid 184 current snap is 0 2024-09-21T18:07:42.301 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:42.301 INFO:tasks.rados.rados.0.smithi160.stdout:542: finishing copy_from racing read to smithi16030472-18 2024-09-21T18:07:42.302 INFO:tasks.rados.rados.0.smithi160.stdout:547: got (0) Success 2024-09-21T18:07:42.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 0 (ObjNum 498 snap 0 seq_num 498) dirty exists 2024-09-21T18:07:42.322 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:42.322 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:42.324 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:42.321+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:42.329 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:42.330 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:525: finishing write tid 4 to smithi16030472-367 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:525: oid 367 updating version 0 to 331 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:525: oid 367 version 331 is already newer than 329 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:525: oid 367 version 331 is already newer than 330 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 331 (ObjNum 507 snap 0 seq_num 507) dirty exists 2024-09-21T18:07:42.342 INFO:tasks.rados.rados.0.smithi160.stdout:525: left oid 367 (ObjNum 507 snap 0 seq_num 507) 2024-09-21T18:07:42.344 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:42.344 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:42.345 INFO:tasks.rados.rados.0.smithi160.stdout:512: got (0) Success 2024-09-21T18:07:42.345 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 250 v 0 (ObjNum 249 snap 0 seq_num 249) dirty exists 2024-09-21T18:07:42.345 INFO:tasks.rados.rados.0.smithi160.stdout:510: got (0) Success 2024-09-21T18:07:42.345 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 52 v 0 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-21T18:07:42.346 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:42.396 INFO:tasks.rados.rados.0.smithi160.stdout:537: expect (ObjNum 262 snap 0 seq_num 262) 2024-09-21T18:07:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:07:42.397 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:42.393+0000 7f0b6c216640 -1 osd.9 54 handle_osd_map injecting map crc failure 2024-09-21T18:07:42.420 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1949242 2024-09-21T18:07:42.425 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:42.418+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:42.436 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2068463 2024-09-21T18:07:42.445 INFO:tasks.rados.rados.0.smithi160.stdout:542: finishing copy_from to smithi16030472-18 2024-09-21T18:07:42.445 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 18 v 209 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:07:42.445 INFO:tasks.rados.rados.0.smithi160.stdout:549: got (0) Success 2024-09-21T18:07:42.451 INFO:tasks.rados.rados.0.smithi160.stdout:536: finishing write tid 1 to smithi16030472-212 2024-09-21T18:07:42.451 INFO:tasks.rados.rados.0.smithi160.stdout:536: finishing write tid 2 to smithi16030472-212 2024-09-21T18:07:42.451 INFO:tasks.rados.rados.0.smithi160.stdout:536: finishing write tid 3 to smithi16030472-212 2024-09-21T18:07:42.464 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:42.486 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:42.487 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:42.509 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:42.517 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:42.517 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:42.525 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:42.521+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:42.532 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:42.532 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:42.549 INFO:tasks.rados.rados.0.smithi160.stdout:550: expect (ObjNum 257 snap 0 seq_num 257) 2024-09-21T18:07:42.553 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:42.578 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:42.578 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:42.602 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:42.603 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:42.610 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1540338 2024-09-21T18:07:42.626 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:42.626 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:42.626 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:42.622+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:42.629 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:42.649 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:42.649 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:42.656 INFO:tasks.rados.rados.0.smithi160.stdout:536: finishing write tid 4 to smithi16030472-212 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:536: oid 212 updating version 0 to 335 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:536: oid 212 updating version 335 to 336 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:536: oid 212 version 336 is already newer than 334 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 212 v 336 (ObjNum 509 snap 0 seq_num 509) dirty exists 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:536: left oid 212 (ObjNum 509 snap 0 seq_num 509) 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:517: finishing copy_from to smithi16030472-405 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 405 v 49 (ObjNum 72 snap 0 seq_num 72) dirty exists 2024-09-21T18:07:42.657 INFO:tasks.rados.rados.0.smithi160.stdout:538: finishing write tid 1 to smithi16030472-329 2024-09-21T18:07:42.658 INFO:tasks.rados.rados.0.smithi160.stdout:548: got (0) Success 2024-09-21T18:07:42.659 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-21T18:07:42.659 INFO:tasks.rados.rados.0.smithi160.stdout:538: finishing write tid 2 to smithi16030472-329 2024-09-21T18:07:42.659 INFO:tasks.rados.rados.0.smithi160.stdout:538: finishing write tid 3 to smithi16030472-329 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:538: finishing write tid 4 to smithi16030472-329 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:538: oid 329 updating version 0 to 340 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:538: oid 329 version 340 is already newer than 339 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:538: oid 329 version 340 is already newer than 338 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 329 v 340 (ObjNum 510 snap 0 seq_num 510) dirty exists 2024-09-21T18:07:42.660 INFO:tasks.rados.rados.0.smithi160.stdout:538: left oid 329 (ObjNum 510 snap 0 seq_num 510) 2024-09-21T18:07:42.663 INFO:tasks.rados.rados.0.smithi160.stdout:539: expect (ObjNum 363 snap 0 seq_num 363) 2024-09-21T18:07:42.666 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1796622 2024-09-21T18:07:42.672 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:42.673 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:42.697 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:42.697 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:42.700 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1956221 2024-09-21T18:07:42.703 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:42.703 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:42.703 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:42.703 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:42.718 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:42.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:42.748 INFO:tasks.rados.rados.0.smithi160.stdout:510: done (14 left) 2024-09-21T18:07:42.748 INFO:tasks.rados.rados.0.smithi160.stdout:512: done (13 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:517: done (12 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:525: done (11 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:536: done (10 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:537: done (9 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:538: done (8 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:539: done (7 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:542: done (6 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:547: done (5 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:548: done (4 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:549: done (3 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:550: done (2 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:552: done (1 left) 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:553: cache_try_flush oid 115 snap -1 2024-09-21T18:07:42.749 INFO:tasks.rados.rados.0.smithi160.stdout:554: delete oid 224 current snap is 0 2024-09-21T18:07:42.763 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:42.761+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:42.829 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:42.826+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:42.845 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:42.918 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:42.919 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:42.919 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:42.919 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:42.920 INFO:tasks.rados.rados.0.smithi160.stdout:553: got (2) No such file or directory 2024-09-21T18:07:42.920 INFO:tasks.rados.rados.0.smithi160.stdout:551: finishing write tid 1 to smithi16030472-435 2024-09-21T18:07:42.920 INFO:tasks.rados.rados.0.smithi160.stdout:551: finishing write tid 2 to smithi16030472-435 2024-09-21T18:07:42.920 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:42.963 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:42.961+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:43.030 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:43.028+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:43.067 INFO:tasks.rados.rados.0.smithi160.stdout:553: done (2 left) 2024-09-21T18:07:43.067 INFO:tasks.rados.rados.0.smithi160.stdout:554: done (1 left) 2024-09-21T18:07:43.067 INFO:tasks.rados.rados.0.smithi160.stdout:555: cache_flush oid 240 snap -1 2024-09-21T18:07:43.067 INFO:tasks.rados.rados.0.smithi160.stdout:556: cache_evict oid 471 snap -1 2024-09-21T18:07:43.068 INFO:tasks.rados.rados.0.smithi160.stdout:557: cache_flush oid 174 snap -1 2024-09-21T18:07:43.068 INFO:tasks.rados.rados.0.smithi160.stdout:558: write oid 317 current snap is 0 2024-09-21T18:07:43.068 INFO:tasks.rados.rados.0.smithi160.stdout:558: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-21T18:07:43.087 INFO:tasks.rados.rados.0.smithi160.stdout:558: writing smithi16030472-317 from 655605 to 1204491 tid 1 2024-09-21T18:07:43.095 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:43.095 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:43.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:43.096 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:43.096 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:558: writing smithi16030472-317 from 1653403 to 2296581 tid 2 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:559: cache_flush oid 106 snap -1 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:560: read oid 266 snap -1 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:560: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:561: cache_evict oid 133 snap -1 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:555: got (2) No such file or directory 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:556: got (16) Device or resource busy 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:555: done (7 left) 2024-09-21T18:07:43.105 INFO:tasks.rados.rados.0.smithi160.stdout:556: done (6 left) 2024-09-21T18:07:43.106 INFO:tasks.rados.rados.0.smithi160.stdout:562: write (excl) oid 339 current snap is 0 2024-09-21T18:07:43.106 INFO:tasks.rados.rados.0.smithi160.stdout:562: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-21T18:07:43.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:43.126 INFO:tasks.rados.rados.0.smithi160.stdout:562: writing smithi16030472-339 from 672412 to 1296547 tid 1 2024-09-21T18:07:43.131 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:43.126+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:43.145 INFO:tasks.rados.rados.0.smithi160.stdout:562: writing smithi16030472-339 from 1995532 to 2672412 tid 2 2024-09-21T18:07:43.145 INFO:tasks.rados.rados.0.smithi160.stdout:563: write oid 195 current snap is 0 2024-09-21T18:07:43.146 INFO:tasks.rados.rados.0.smithi160.stdout:563: seq_num 517 ranges {689219=415737,1770367=646847} 2024-09-21T18:07:43.156 INFO:tasks.rados.rados.0.smithi160.stdout:563: writing smithi16030472-195 from 689219 to 1104956 tid 1 2024-09-21T18:07:43.168 INFO:tasks.rados.rados.0.smithi160.stdout:563: writing smithi16030472-195 from 1770367 to 2417214 tid 2 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:564: cache_try_flush oid 219 snap -1 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:565: cache_try_flush oid 427 snap -1 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:557: got (2) No such file or directory 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:557: done (9 left) 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:566: write (excl) oid 276 current snap is 0 2024-09-21T18:07:43.169 INFO:tasks.rados.rados.0.smithi160.stdout:566: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-21T18:07:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:566: writing smithi16030472-276 from 706026 to 1197012 tid 1 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:566: writing smithi16030472-276 from 1828849 to 2619354 tid 2 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:567: cache_evict oid 262 snap -1 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:568: read oid 64 snap -1 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:568: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:569: copy_from oid 183 from oid 65 current snap is 0 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:570: cache_flush oid 122 snap -1 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:571: read oid 322 snap -1 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:571: expect (ObjNum 321 snap 0 seq_num 321) 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:572: write (excl) oid 316 current snap is 0 2024-09-21T18:07:43.190 INFO:tasks.rados.rados.0.smithi160.stdout:572: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-21T18:07:43.199 INFO:tasks.rados.rados.0.smithi160.stdout:572: writing smithi16030472-316 from 722833 to 1289068 tid 1 2024-09-21T18:07:43.209 INFO:tasks.rados.rados.0.smithi160.stdout:572: writing smithi16030472-316 from 1887331 to 2537847 tid 2 2024-09-21T18:07:43.210 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:43.233 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:43.228+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:43.243 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:43.244 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:43.244 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:43.244 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:43.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:43.295 INFO:tasks.rados.rados.0.smithi160.stdout:559: got (0) Success 2024-09-21T18:07:43.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 106 v 0 (ObjNum 105 snap 0 seq_num 105) dirty exists 2024-09-21T18:07:43.295 INFO:tasks.rados.rados.0.smithi160.stdout:559: done (15 left) 2024-09-21T18:07:43.295 INFO:tasks.rados.rados.0.smithi160.stdout:573: write (excl) oid 272 current snap is 0 2024-09-21T18:07:43.295 INFO:tasks.rados.rados.0.smithi160.stdout:573: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-21T18:07:43.306 INFO:tasks.rados.rados.0.smithi160.stdout:573: writing smithi16030472-272 from 739640 to 1381124 tid 1 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:573: writing smithi16030472-272 from 1829460 to 2623634 tid 2 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: finishing write tid 3 to smithi16030472-435 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: finishing write tid 4 to smithi16030472-435 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: oid 435 updating version 0 to 415 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: oid 435 updating version 415 to 416 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: oid 435 version 416 is already newer than 414 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 435 v 416 (ObjNum 512 snap 0 seq_num 512) dirty exists 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:551: left oid 435 (ObjNum 512 snap 0 seq_num 512) 2024-09-21T18:07:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:564: got (2) No such file or directory 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:565: got (0) Success 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 0 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:551: done (15 left) 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:564: done (14 left) 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:565: done (13 left) 2024-09-21T18:07:43.317 INFO:tasks.rados.rados.0.smithi160.stdout:574: delete oid 483 current snap is 0 2024-09-21T18:07:43.333 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:43.326+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:43.386 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:43.386 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:43.386 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:43.386 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:43.386 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:43.402 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: finishing write tid 1 to smithi16030472-317 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: finishing write tid 2 to smithi16030472-317 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: finishing write tid 3 to smithi16030472-317 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: finishing write tid 4 to smithi16030472-317 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: oid 317 updating version 0 to 390 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: oid 317 updating version 390 to 391 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: oid 317 version 391 is already newer than 389 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 391 (ObjNum 515 snap 0 seq_num 515) dirty exists 2024-09-21T18:07:43.417 INFO:tasks.rados.rados.0.smithi160.stdout:558: left oid 317 (ObjNum 515 snap 0 seq_num 515) 2024-09-21T18:07:43.418 INFO:tasks.rados.rados.0.smithi160.stdout:561: got (0) Success 2024-09-21T18:07:43.418 INFO:tasks.rados.rados.0.smithi160.stdout:572: finishing write tid 1 to smithi16030472-316 2024-09-21T18:07:43.446 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:43.442+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:43.510 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:43.506+0000 7f66f041c640 -1 osd.7 54 handle_osd_map injecting map crc failure 2024-09-21T18:07:43.510 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:43.504+0000 7f8c3a0ca640 -1 osd.12 54 handle_osd_map injecting map crc failure 2024-09-21T18:07:43.518 INFO:tasks.rados.rados.0.smithi160.stdout:572: finishing write tid 2 to smithi16030472-316 2024-09-21T18:07:43.519 INFO:tasks.rados.rados.0.smithi160.stdout:572: finishing write tid 3 to smithi16030472-316 2024-09-21T18:07:43.528 INFO:tasks.rados.rados.0.smithi160.stdout:572: finishing write tid 4 to smithi16030472-316 2024-09-21T18:07:43.529 INFO:tasks.rados.rados.0.smithi160.stdout:572: oid 316 updating version 0 to 316 2024-09-21T18:07:43.529 INFO:tasks.rados.rados.0.smithi160.stdout:572: oid 316 updating version 316 to 317 2024-09-21T18:07:43.529 INFO:tasks.rados.rados.0.smithi160.stdout:572: oid 316 version 317 is already newer than 315 2024-09-21T18:07:43.529 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 316 v 317 (ObjNum 519 snap 0 seq_num 519) dirty exists 2024-09-21T18:07:43.529 INFO:tasks.rados.rados.0.smithi160.stdout:572: left oid 316 (ObjNum 519 snap 0 seq_num 519) 2024-09-21T18:07:43.535 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:43.533+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:43.569 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:43.569 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:43.569 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:43.569 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 25.235829318, 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.570 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:07:43.571 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.572 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:07:43.573 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.395977322, 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.574 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.575 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.576 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.577 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:43.578 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.292256874, 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.579 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:07:43.580 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:07:43.581 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.582 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.291482130999999, 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.290834111999999, 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:07:43.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:07:43.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.276962417, 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.592 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:07:43.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.274021400999999, 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.595 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.599 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.271004553000001, 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.603 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.268792971, 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 19.163455879000001, 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:43.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:07:43.611 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.472033504999999, 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.469003505, 2024-09-21T18:07:43.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:07:43.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.502439542999999, 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.619 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:07:43.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:07:43.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:07:43.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 11.835604108, 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:07:43.623 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:07:43.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:07:43.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.5321130270000003, 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:43.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:07:43.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.5298159939999998, 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.5277655460000004, 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:07:43.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:07:43.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.5256101849999997, 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:07:43.640 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:07:43.644 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.5248474270000001, 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:43.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:07:43.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:07:43.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 4.4229423820000004, 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.652 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.653 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.653 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:43.653 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:43.653 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: finishing write tid 1 to smithi16030472-195 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: finishing write tid 2 to smithi16030472-195 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: finishing write tid 3 to smithi16030472-195 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: finishing write tid 4 to smithi16030472-195 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: oid 195 updating version 0 to 355 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: oid 195 version 355 is already newer than 354 2024-09-21T18:07:43.654 INFO:tasks.rados.rados.0.smithi160.stdout:563: oid 195 updating version 355 to 356 2024-09-21T18:07:43.655 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 356 (ObjNum 517 snap 0 seq_num 517) dirty exists 2024-09-21T18:07:43.655 INFO:tasks.rados.rados.0.smithi160.stdout:563: left oid 195 (ObjNum 517 snap 0 seq_num 517) 2024-09-21T18:07:43.655 INFO:tasks.rados.rados.0.smithi160.stdout:568: expect (ObjNum 63 snap 0 seq_num 63) 2024-09-21T18:07:43.655 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:43.656 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:43.650+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:43.755 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:43.752+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:43.755 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:43.789 INFO:tasks.rados.rados.0.smithi160.stdout:571: expect (ObjNum 321 snap 0 seq_num 321) 2024-09-21T18:07:43.827 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:43.827 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:43.827 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:43.827 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:43.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:43.855 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:43.852+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:43.882 INFO:tasks.rados.rados.0.smithi160.stdout:560: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-21T18:07:43.882 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 70711 2024-09-21T18:07:43.908 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 140163 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:566: finishing write tid 1 to smithi16030472-276 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:566: finishing write tid 2 to smithi16030472-276 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:567: got (2) No such file or directory 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:570: got (2) No such file or directory 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:562: finishing write tid 1 to smithi16030472-339 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:562: finishing write tid 2 to smithi16030472-339 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:573: finishing write tid 1 to smithi16030472-272 2024-09-21T18:07:43.955 INFO:tasks.rados.rados.0.smithi160.stdout:573: finishing write tid 2 to smithi16030472-272 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: finishing write tid 3 to smithi16030472-276 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: finishing write tid 4 to smithi16030472-276 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: oid 276 updating version 0 to 357 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: oid 276 updating version 357 to 359 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: oid 276 version 359 is already newer than 358 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 359 (ObjNum 518 snap 0 seq_num 518) dirty exists 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:566: left oid 276 (ObjNum 518 snap 0 seq_num 518) 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: finishing write tid 3 to smithi16030472-339 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: finishing write tid 4 to smithi16030472-339 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: oid 339 updating version 0 to 417 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: oid 339 updating version 417 to 418 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: oid 339 updating version 418 to 419 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 339 v 419 (ObjNum 516 snap 0 seq_num 516) dirty exists 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:562: left oid 339 (ObjNum 516 snap 0 seq_num 516) 2024-09-21T18:07:43.956 INFO:tasks.rados.rados.0.smithi160.stdout:573: finishing write tid 3 to smithi16030472-272 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:573: finishing write tid 4 to smithi16030472-272 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:573: oid 272 updating version 0 to 420 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:573: oid 272 version 420 is already newer than 419 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:573: oid 272 updating version 420 to 421 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 272 v 421 (ObjNum 520 snap 0 seq_num 520) dirty exists 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:573: left oid 272 (ObjNum 520 snap 0 seq_num 520) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:558: done (13 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:560: done (12 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:561: done (11 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:562: done (10 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:563: done (9 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:566: done (8 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:567: done (7 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:568: done (6 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:570: done (5 left) 2024-09-21T18:07:43.957 INFO:tasks.rados.rados.0.smithi160.stdout:571: done (4 left) 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:572: done (3 left) 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:573: done (2 left) 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:574: done (1 left) 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:575: cache_try_flush oid 433 snap -1 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:576: cache_flush oid 243 snap -1 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:577: copy_from oid 381 from oid 5 current snap is 0 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:578: write (excl) oid 278 current snap is 0 2024-09-21T18:07:43.958 INFO:tasks.rados.rados.0.smithi160.stdout:578: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-21T18:07:43.959 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:43.957+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:43.964 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:43.964 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:43.964 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:43.964 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:43.964 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:43.978 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:43.979 INFO:tasks.rados.rados.0.smithi160.stdout:578: writing smithi16030472-278 from 773254 to 1565236 tid 1 2024-09-21T18:07:43.996 INFO:tasks.rados.rados.0.smithi160.stdout:578: writing smithi16030472-278 from 2346424 to 2773254 tid 2 2024-09-21T18:07:43.996 INFO:tasks.rados.rados.0.smithi160.stdout:579: write (excl) oid 273 current snap is 0 2024-09-21T18:07:43.996 INFO:tasks.rados.rados.0.smithi160.stdout:579: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-21T18:07:44.013 INFO:tasks.rados.rados.0.smithi160.stdout:579: writing smithi16030472-273 from 790061 to 1257292 tid 1 2024-09-21T18:07:44.026 INFO:tasks.rados.rados.0.smithi160.stdout:579: writing smithi16030472-273 from 2004906 to 2546407 tid 2 2024-09-21T18:07:44.026 INFO:tasks.rados.rados.0.smithi160.stdout:580: write oid 12 current snap is 0 2024-09-21T18:07:44.026 INFO:tasks.rados.rados.0.smithi160.stdout:580: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-21T18:07:44.034 INFO:tasks.rados.rados.0.smithi160.stdout:580: writing smithi16030472-12 from 406868 to 949348 tid 1 2024-09-21T18:07:44.042 INFO:tasks.rados.rados.0.smithi160.stdout:580: writing smithi16030472-12 from 1547035 to 1948547 tid 2 2024-09-21T18:07:44.048 INFO:tasks.rados.rados.0.smithi160.stdout:580: writing smithi16030472-12 from 2690354 to 2806868 tid 3 2024-09-21T18:07:44.048 INFO:tasks.rados.rados.0.smithi160.stdout:581: copy_from oid 180 from oid 210 current snap is 0 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:582: copy_from oid 408 from oid 104 current snap is 0 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:583: copy_from oid 359 from oid 345 current snap is 0 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:584: read oid 497 snap -1 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:584: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:585: cache_flush oid 29 snap -1 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:586: write oid 433 current snap is 0 2024-09-21T18:07:44.049 INFO:tasks.rados.rados.0.smithi160.stdout:586: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2024-09-21T18:07:44.056 INFO:tasks.rados.rados.0.smithi160.stdout:586: writing smithi16030472-433 from 423675 to 1157757 tid 1 2024-09-21T18:07:44.060 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:44.054+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:44.063 INFO:tasks.rados.rados.0.smithi160.stdout:586: writing smithi16030472-433 from 1721870 to 2267040 tid 2 2024-09-21T18:07:44.066 INFO:tasks.rados.rados.0.smithi160.stdout:586: writing smithi16030472-433 from 2717777 to 2823675 tid 3 2024-09-21T18:07:44.066 INFO:tasks.rados.rados.0.smithi160.stdout:587: cache_evict oid 1 snap -1 2024-09-21T18:07:44.066 INFO:tasks.rados.rados.0.smithi160.stdout:588: write oid 245 current snap is 0 2024-09-21T18:07:44.066 INFO:tasks.rados.rados.0.smithi160.stdout:588: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-21T18:07:44.071 INFO:tasks.rados.rados.0.smithi160.stdout:588: writing smithi16030472-245 from 440482 to 849813 tid 1 2024-09-21T18:07:44.076 INFO:tasks.rados.rados.0.smithi160.stdout:588: writing smithi16030472-245 from 1380352 to 1785533 tid 2 2024-09-21T18:07:44.082 INFO:tasks.rados.rados.0.smithi160.stdout:588: writing smithi16030472-245 from 2461553 to 2840482 tid 3 2024-09-21T18:07:44.082 INFO:tasks.rados.rados.0.smithi160.stdout:589: cache_flush oid 67 snap -1 2024-09-21T18:07:44.082 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.082 INFO:tasks.rados.rados.0.smithi160.stdout:576: got (2) No such file or directory 2024-09-21T18:07:44.083 INFO:tasks.rados.rados.0.smithi160.stdout:576: done (15 left) 2024-09-21T18:07:44.083 INFO:tasks.rados.rados.0.smithi160.stdout:590: cache_try_flush oid 35 snap -1 2024-09-21T18:07:44.083 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.096 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:44.096 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:44.096 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:44.096 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:07:44.096 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:44.110 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:44.152 INFO:tasks.rados.rados.0.smithi160.stdout:585: got (2) No such file or directory 2024-09-21T18:07:44.152 INFO:tasks.rados.rados.0.smithi160.stdout:585: done (15 left) 2024-09-21T18:07:44.152 INFO:tasks.rados.rados.0.smithi160.stdout:591: copy_from oid 45 from oid 272 current snap is 0 2024-09-21T18:07:44.152 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.160 INFO:tasks.rados.rados.0.smithi160.stdout:586: finishing write tid 1 to smithi16030472-433 2024-09-21T18:07:44.198 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:44.194+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:44.221 INFO:tasks.rados.rados.0.smithi160.stdout:579: finishing write tid 1 to smithi16030472-273 2024-09-21T18:07:44.237 INFO:tasks.rados.rados.0.smithi160.stdout:579: finishing write tid 2 to smithi16030472-273 2024-09-21T18:07:44.237 INFO:tasks.rados.rados.0.smithi160.stdout:581: finishing copy_from racing read to smithi16030472-180 2024-09-21T18:07:44.250 INFO:tasks.rados.rados.0.smithi160.stdout:586: finishing write tid 2 to smithi16030472-433 2024-09-21T18:07:44.261 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:44.254+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:44.280 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:44.280 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:44.281 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:44.281 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:44.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:44.316 INFO:tasks.rados.rados.0.smithi160.stdout:580: finishing write tid 1 to smithi16030472-12 2024-09-21T18:07:44.319 INFO:tasks.rados.rados.0.smithi160.stdout:579: finishing write tid 3 to smithi16030472-273 2024-09-21T18:07:44.338 INFO:tasks.rados.rados.0.smithi160.stdout:581: finishing copy_from to smithi16030472-180 2024-09-21T18:07:44.338 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 111 (ObjNum 209 snap 0 seq_num 209) dirty exists 2024-09-21T18:07:44.338 INFO:tasks.rados.rados.0.smithi160.stdout:581: done (15 left) 2024-09-21T18:07:44.338 INFO:tasks.rados.rados.0.smithi160.stdout:592: cache_try_flush oid 248 snap -1 2024-09-21T18:07:44.339 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.340 INFO:tasks.rados.rados.0.smithi160.stdout:589: got (0) Success 2024-09-21T18:07:44.340 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 67 v 0 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-09-21T18:07:44.340 INFO:tasks.rados.rados.0.smithi160.stdout:589: done (15 left) 2024-09-21T18:07:44.340 INFO:tasks.rados.rados.0.smithi160.stdout:593: write oid 126 current snap is 0 2024-09-21T18:07:44.340 INFO:tasks.rados.rados.0.smithi160.stdout:593: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-21T18:07:44.348 INFO:tasks.rados.rados.0.smithi160.stdout:593: writing smithi16030472-126 from 457289 to 941869 tid 1 2024-09-21T18:07:44.358 INFO:tasks.rados.rados.0.smithi160.stdout:593: writing smithi16030472-126 from 1438834 to 1987673 tid 2 2024-09-21T18:07:44.362 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:44.358+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:44.364 INFO:tasks.rados.rados.0.smithi160.stdout:593: writing smithi16030472-126 from 2772623 to 2857289 tid 3 2024-09-21T18:07:44.365 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.365 INFO:tasks.rados.rados.0.smithi160.stdout:569: finishing copy_from racing read to smithi16030472-183 2024-09-21T18:07:44.365 INFO:tasks.rados.rados.0.smithi160.stdout:569: finishing copy_from to smithi16030472-183 2024-09-21T18:07:44.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 42 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:07:44.365 INFO:tasks.rados.rados.0.smithi160.stdout:584: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:07:44.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:07:44.463 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:44.461+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:44.470 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:44.470 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:44.471 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:44.471 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:44.471 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: finishing write tid 1 to smithi16030472-278 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: finishing write tid 2 to smithi16030472-278 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: finishing write tid 3 to smithi16030472-278 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: finishing write tid 4 to smithi16030472-278 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: oid 278 updating version 0 to 361 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: oid 278 updating version 361 to 362 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: oid 278 version 362 is already newer than 360 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 278 v 362 (ObjNum 522 snap 0 seq_num 522) dirty exists 2024-09-21T18:07:44.481 INFO:tasks.rados.rados.0.smithi160.stdout:578: left oid 278 (ObjNum 522 snap 0 seq_num 522) 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:587: got (16) Device or resource busy 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:586: finishing write tid 3 to smithi16030472-433 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:580: finishing write tid 2 to smithi16030472-12 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:580: finishing write tid 3 to smithi16030472-12 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:579: finishing write tid 4 to smithi16030472-273 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:579: oid 273 updating version 0 to 393 2024-09-21T18:07:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:579: oid 273 updating version 393 to 395 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:579: oid 273 version 395 is already newer than 394 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 273 v 395 (ObjNum 523 snap 0 seq_num 523) dirty exists 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:579: left oid 273 (ObjNum 523 snap 0 seq_num 523) 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:586: finishing write tid 4 to smithi16030472-433 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:577: finishing copy_from to smithi16030472-381 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:07:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:569: done (15 left) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:578: done (14 left) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:579: done (13 left) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:584: done (12 left) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:587: done (11 left) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:594: read oid 493 snap -1 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:594: expect (ObjNum 492 snap 0 seq_num 492) 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:595: read oid 385 snap -1 2024-09-21T18:07:44.484 INFO:tasks.rados.rados.0.smithi160.stdout:595: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-21T18:07:44.485 INFO:tasks.rados.rados.0.smithi160.stdout:596: copy_from oid 456 from oid 362 current snap is 0 2024-09-21T18:07:44.485 INFO:tasks.rados.rados.0.smithi160.stdout:597: read oid 455 snap -1 2024-09-21T18:07:44.485 INFO:tasks.rados.rados.0.smithi160.stdout:597: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-21T18:07:44.485 INFO:tasks.rados.rados.0.smithi160.stdout:598: copy_from oid 14 from oid 306 current snap is 0 2024-09-21T18:07:44.485 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:44.486 INFO:tasks.rados.rados.0.smithi160.stdout:594: expect (ObjNum 492 snap 0 seq_num 492) 2024-09-21T18:07:44.486 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:44.488 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:44.563 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:44.558+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:44.579 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:44.606 INFO:tasks.rados.rados.0.smithi160.stdout:577: finishing copy_from racing read to smithi16030472-381 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: finishing write tid 5 to smithi16030472-433 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: oid 433 updating version 0 to 422 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: oid 433 updating version 422 to 423 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: oid 433 updating version 423 to 425 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: oid 433 version 425 is already newer than 424 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 425 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:586: left oid 433 (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:07:44.607 INFO:tasks.rados.rados.0.smithi160.stdout:597: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-21T18:07:44.608 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:44.608 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:29 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'19, mlcod=43'17)", 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.761027075000001, 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20988762999999999, 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845718+0000", 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.845721+0000", 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0570000000000001e-06 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847734+0000", 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0020132850000000001 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847771+0000", 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6415000000000003e-05 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847794+0000", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3730000000000001e-05 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948080+0000", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100285443 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.948115+0000", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5098999999999997e-05 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055500+0000", 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107384654 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055605+0000", 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010594699999999999 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:106 4.0 e43/33 4:0e09a926:::smithi16030472-27:head v 43'20, mlcod=43'17)", 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.758892643999999, 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20783109999999999, 2024-09-21T18:07:44.613 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847852+0000", 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.847855+0000", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4569999999999999e-06 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849293+0000", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0014381109999999999 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.849325+0000", 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2361999999999999e-05 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.949454+0000", 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128488 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950524+0000", 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010704860000000001 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:12.950583+0000", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8643000000000001e-05 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055613+0000", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10503064500000001 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.055683+0000", 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9907999999999995e-05 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.617 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:154 4.2 e44/33 4:6abed166:::smithi16030472-39:head v 44'23, mlcod=43'17)", 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 30.693944058, 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20421822000000001, 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:44.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912801+0000", 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.912802+0000", 2024-09-21T18:07:44.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3030000000000001e-06 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913279+0000", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000476928 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:13.913294+0000", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5209e-05 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013383+0000", 2024-09-21T18:07:44.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10008903600000001 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013498+0000", 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011462 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.013532+0000", 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4069000000000002e-05 2024-09-21T18:07:44.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.116937+0000", 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103404914 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.117019+0000", 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2140999999999998e-05 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:241 4.2 e48/33 4:6513c01e:::smithi16030472-210:head v 48'96, mlcod=48'94)", 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.485901951999999, 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.216069437, 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120847+0000", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1960000000000001e-06 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125270+0000", 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044228310000000003 2024-09-21T18:07:44.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125303+0000", 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2975999999999997e-05 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125320+0000", 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.717e-05 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225582+0000", 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100261425 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225619+0000", 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6962000000000003e-05 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336827+0000", 2024-09-21T18:07:44.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111208272 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336912+0000", 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5605000000000005e-05 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:242 4.2 e48/33 4:50d6db3f:::smithi16030472-204:head v 48'97, mlcod=48'94)", 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.473083244000001, 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.203351217, 2024-09-21T18:07:44.627 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133662+0000", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.133665+0000", 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2389999999999998e-06 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136624+0000", 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002958666 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136661+0000", 2024-09-21T18:07:44.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7225999999999999e-05 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.136685+0000", 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4275999999999999e-05 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.330181+0000", 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19349608500000001 2024-09-21T18:07:44.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.330257+0000", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6172000000000002e-05 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336920+0000", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0066630810000000004 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.337013+0000", 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2472000000000005e-05 2024-09-21T18:07:44.631 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1068 4.0 e49/33 4:02c3e29a:::smithi16030472-237:head v 49'81, mlcod=48'78)", 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 18.630448502, 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20186737099999999, 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.632 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976296+0000", 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.976313+0000", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6697e-05 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.977143+0000", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000829476 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.977165+0000", 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2673000000000001e-05 2024-09-21T18:07:44.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077321+0000", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10015566100000001 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077479+0000", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015782700000000001 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.077516+0000", 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7029000000000001e-05 2024-09-21T18:07:44.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.178099+0000", 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10058284000000001 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.178164+0000", 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5167999999999997e-05 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.636 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1524 4.0 e51/33 4:3d0b4c2b:::smithi16030472-340:head v 51'114, mlcod=50'111)", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.333248201, 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21167023200000001, 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273497+0000", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.273500+0000", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8830000000000002e-06 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274509+0000", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001009427 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274536+0000", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6962e-05 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274550+0000", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4066999999999999e-05 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274686+0000", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000136243 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.274722+0000", 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5639000000000002e-05 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485101+0000", 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21037930399999999 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485167+0000", 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5707e-05 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1525 4.0 e51/33 4:3d0b4c2b:::smithi16030472-340:head v 51'115, mlcod=50'111)", 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.331066278, 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20957785800000001, 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275679+0000", 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.275681+0000", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5799999999999999e-06 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276882+0000", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012012189999999999 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276913+0000", 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0877999999999998e-05 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.276937+0000", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3532000000000001e-05 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.377834+0000", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100897295 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.377870+0000", 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6057999999999999e-05 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485176+0000", 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10730627 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485257+0000", 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0025999999999996e-05 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.645 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:407 4.0 e51/33 4:0382d963:::smithi16030472-337:head v 51'116, mlcod=50'112)", 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.328444634, 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207031938, 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278300+0000", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.278303+0000", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8229999999999999e-06 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283172+0000", 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0048691050000000003 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283198+0000", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6225e-05 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.283213+0000", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4742999999999999e-05 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.480047+0000", 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19683339 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.480080+0000", 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3881999999999998e-05 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485263+0000", 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0051822600000000002 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.485332+0000", 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9510000000000004e-05 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.010943779, 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.650 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:07:44.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:07:44.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1535 4.2 e51/33 4:7b415eca:::smithi16030472-342:head v 51'119, mlcod=50'116)", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:44.655 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.866225404, 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20316891400000001, 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740520+0000", 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.740523+0000", 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9859999999999999e-06 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741028+0000", 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050500000000000002 2024-09-21T18:07:44.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741056+0000", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8466000000000002e-05 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.741075+0000", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8739000000000001e-05 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.841326+0000", 2024-09-21T18:07:44.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1002511 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.841362+0000", 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5744999999999997e-05 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943604+0000", 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10224288500000001 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943688+0000", 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3992999999999994e-05 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1536 4.2 e51/33 4:7b415eca:::smithi16030472-342:head v 51'120, mlcod=50'116)", 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 12.864087315000001, 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20178523700000001, 2024-09-21T18:07:44.661 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:44.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742658+0000", 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742660+0000", 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8159999999999998e-06 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742672+0000", 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1535999999999999e-05 2024-09-21T18:07:44.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742700+0000", 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7606e-05 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.742718+0000", 2024-09-21T18:07:44.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8629000000000001e-05 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943001+0000", 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20028306900000001 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.943064+0000", 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2343999999999998e-05 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.944327+0000", 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001263602 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.944443+0000", 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000115635 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.668 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:448 4.0 e51/33 4:385a7674:::smithi16030472-358:head v 51'136, mlcod=51'134)", 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 11.724197273, 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20784413099999999, 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:44.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882548+0000", 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.882551+0000", 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3170000000000002e-06 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.885156+0000", 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026050969999999998 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.885184+0000", 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8348e-05 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.985325+0000", 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014103000000001 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.085601+0000", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100275138 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.085657+0000", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6705e-05 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.090301+0000", 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0046439000000000003 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:33.090392+0000", 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0595999999999998e-05 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:514 4.0 e53/33 4:172b8221:::smithi16030472-446:head v 53'137, mlcod=51'135)", 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.029896+0000", 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.576848858, 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20986348399999999, 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.673 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.029896+0000", 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.029896+0000", 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.029897+0000", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.23e-07 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.032121+0000", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002224288 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.032147+0000", 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6251999999999999e-05 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.132289+0000", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014142500000001 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.132481+0000", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000191826 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.132524+0000", 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2898999999999997e-05 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.239656+0000", 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107132198 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.239760+0000", 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000103773 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:448 4.2 e53/33 4:6e9cef1a:::smithi16030472-436:head v 53'134, mlcod=52'131)", 2024-09-21T18:07:44.677 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.307210+0000", 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.2995345499999997, 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20908843299999999, 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.307210+0000", 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.307210+0000", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.307215+0000", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2429999999999999e-06 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.308773+0000", 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0015579470000000001 2024-09-21T18:07:44.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.308798+0000", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5392e-05 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.308813+0000", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4552000000000001e-05 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.409045+0000", 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100232616 2024-09-21T18:07:44.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.409080+0000", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4928999999999997e-05 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516206+0000", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107125711 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516299+0000", 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3042999999999997e-05 2024-09-21T18:07:44.681 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1947 4.2 e53/33 4:6e9cef1a:::smithi16030472-436:head v 53'135, mlcod=52'131)", 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.310564+0000", 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.296181077, 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205822117, 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310564+0000", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310564+0000", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310567+0000", 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0199999999999999e-06 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310890+0000", 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00032264799999999998 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310916+0000", 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6919000000000001e-05 2024-09-21T18:07:44.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.310930+0000", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3566e-05 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.511168+0000", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20023780899999999 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.511227+0000", 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.8696999999999997e-05 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516310+0000", 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.005083644 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516386+0000", 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5814000000000003e-05 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1948 4.2 e53/33 4:6e9cef1a:::smithi16030472-436:head v 53'136, mlcod=52'131)", 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.313283+0000", 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.2934624149999996, 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20331799, 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313283+0000", 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313283+0000", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313285+0000", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8229999999999999e-06 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313688+0000", 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000402925 2024-09-21T18:07:44.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313731+0000", 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2585e-05 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.313745+0000", 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3842999999999999e-05 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.513567+0000", 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19982270799999999 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.513628+0000", 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0850999999999997e-05 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516517+0000", 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0028886250000000001 2024-09-21T18:07:44.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.516601+0000", 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3629999999999997e-05 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1949 4.2 e53/33 4:6e9cef1a:::smithi16030472-436:head v 53'137, mlcod=52'132)", 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.316016+0000", 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.2907292009999995, 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20106399999999999, 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.691 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.316016+0000", 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.316016+0000", 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.316019+0000", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.033e-06 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.316031+0000", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2363e-05 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.316057+0000", 2024-09-21T18:07:44.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6129e-05 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.416191+0000", 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100133953 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.515668+0000", 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099476621000000001 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.515719+0000", 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1653999999999997e-05 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517008+0000", 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012885869999999999 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517080+0000", 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1660000000000002e-05 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1973 4.2 e53/33 4:796e5a55:::smithi16030472-441:head v 53'138, mlcod=52'132)", 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.517338+0000", 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 7.0894072640000001, 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206568531, 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517338+0000", 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517338+0000", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517340+0000", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0329999999999998e-06 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517571+0000", 2024-09-21T18:07:44.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00023122799999999999 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.517596+0000", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4989000000000001e-05 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.617750+0000", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100153782 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.718081+0000", 2024-09-21T18:07:44.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100331767 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.718141+0000", 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9675e-05 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.723830+0000", 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0056888030000000001 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.723906+0000", 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6254000000000005e-05 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.700 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.4.0:529 4.0 e53/33 4:3da33cdf:::smithi16030472-459:head v 53'145, mlcod=53'139)", 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:37.721141+0000", 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.8856037529999998, 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20963757299999999, 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:44.701 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.721141+0000", 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.721141+0000", 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:44.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.721142+0000", 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.15e-06 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.722312+0000", 2024-09-21T18:07:44.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001169949 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.722344+0000", 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1681999999999999e-05 2024-09-21T18:07:44.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.822437+0000", 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100093193 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.822571+0000", 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013385200000000001 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:44.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.822633+0000", 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1583999999999999e-05 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.930670+0000", 2024-09-21T18:07:44.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108037628 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:37.930779+0000", 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000108535 2024-09-21T18:07:44.709 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.710 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.710 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.710 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:44.710 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:44.710 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:44.714 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:44.715 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:44.706+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:44.715 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2734724 2024-09-21T18:07:44.715 INFO:tasks.rados.rados.0.smithi160.stdout:580: finishing write tid 4 to smithi16030472-12 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: finishing write tid 5 to smithi16030472-12 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: oid 12 updating version 0 to 320 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: oid 12 updating version 320 to 321 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: oid 12 version 321 is already newer than 318 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: oid 12 version 321 is already newer than 319 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 12 v 321 (ObjNum 524 snap 0 seq_num 524) dirty exists 2024-09-21T18:07:44.716 INFO:tasks.rados.rados.0.smithi160.stdout:580: left oid 12 (ObjNum 524 snap 0 seq_num 524) 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:588: finishing write tid 1 to smithi16030472-245 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:583: finishing copy_from racing read to smithi16030472-359 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:593: finishing write tid 1 to smithi16030472-126 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:577: done (15 left) 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:580: done (14 left) 2024-09-21T18:07:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:586: done (13 left) 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:594: done (12 left) 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:597: done (11 left) 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:599: cache_flush oid 458 snap -1 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:582: finishing copy_from to smithi16030472-408 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 60 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-21T18:07:44.718 INFO:tasks.rados.rados.0.smithi160.stdout:592: got (0) Success 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 248 v 0 (ObjNum 247 snap 0 seq_num 247) dirty exists 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:592: done (11 left) 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:600: cache_try_flush oid 136 snap -1 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:601: cache_evict oid 20 snap -1 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:602: delete oid 90 current snap is 0 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:593: finishing write tid 2 to smithi16030472-126 2024-09-21T18:07:44.719 INFO:tasks.rados.rados.0.smithi160.stdout:575: got (16) Device or resource busy 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:583: finishing copy_from to smithi16030472-359 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 242 (ObjNum 344 snap 0 seq_num 344) dirty exists 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:590: got (0) Success 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 0 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:588: finishing write tid 2 to smithi16030472-245 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:588: finishing write tid 3 to smithi16030472-245 2024-09-21T18:07:44.720 INFO:tasks.rados.rados.0.smithi160.stdout:588: finishing write tid 4 to smithi16030472-245 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: finishing write tid 5 to smithi16030472-245 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: oid 245 updating version 0 to 323 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: oid 245 updating version 323 to 325 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: oid 245 version 325 is already newer than 322 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: oid 245 version 325 is already newer than 324 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 245 v 325 (ObjNum 526 snap 0 seq_num 526) dirty exists 2024-09-21T18:07:44.721 INFO:tasks.rados.rados.0.smithi160.stdout:588: left oid 245 (ObjNum 526 snap 0 seq_num 526) 2024-09-21T18:07:44.722 INFO:tasks.rados.rados.0.smithi160.stdout:582: finishing copy_from racing read to smithi16030472-408 2024-09-21T18:07:44.722 INFO:tasks.rados.rados.0.smithi160.stdout:599: got (0) Success 2024-09-21T18:07:44.722 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 458 v 0 (ObjNum 457 snap 0 seq_num 457) dirty exists 2024-09-21T18:07:44.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:44.785 INFO:tasks.rados.rados.0.smithi160.stdout:591: finishing copy_from racing read to smithi16030472-45 2024-09-21T18:07:44.785 INFO:tasks.rados.rados.0.smithi160.stdout:591: finishing copy_from to smithi16030472-45 2024-09-21T18:07:44.785 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 45 v 421 (ObjNum 520 snap 0 seq_num 520) dirty exists 2024-09-21T18:07:44.786 INFO:tasks.rados.rados.0.smithi160.stdout:601: got (16) Device or resource busy 2024-09-21T18:07:44.812 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:44.806+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:44.868 INFO:tasks.rados.rados.0.smithi160.stdout:598: finishing copy_from to smithi16030472-14 2024-09-21T18:07:44.868 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 204 (ObjNum 305 snap 0 seq_num 305) dirty exists 2024-09-21T18:07:44.875 INFO:tasks.rados.rados.0.smithi160.stdout:595: expect (ObjNum 384 snap 0 seq_num 384) 2024-09-21T18:07:44.875 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:44.888 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:44.891 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:44.891 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:44.891 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:44.891 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:44.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:44.909 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:44.909 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:44.912 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1352229 2024-09-21T18:07:44.927 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:44.928 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:44.936 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 471797 2024-09-21T18:07:44.947 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:44.950 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:44.946+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:44.958 INFO:tasks.rados.rados.0.smithi160.stdout:598: finishing copy_from racing read to smithi16030472-14 2024-09-21T18:07:44.958 INFO:tasks.rados.rados.0.smithi160.stdout:600: got (0) Success 2024-09-21T18:07:44.958 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 136 v 0 (ObjNum 135 snap 0 seq_num 135) dirty exists 2024-09-21T18:07:44.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:44.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:44.968 INFO:tasks.rados.rados.0.smithi160.stdout:593: finishing write tid 3 to smithi16030472-126 2024-09-21T18:07:44.978 INFO:tasks.rados.rados.0.smithi160.stdout:593: finishing write tid 4 to smithi16030472-126 2024-09-21T18:07:44.985 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:44.989 INFO:tasks.rados.rados.0.smithi160.stdout:575: done (13 left) 2024-09-21T18:07:44.989 INFO:tasks.rados.rados.0.smithi160.stdout:582: done (12 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:583: done (11 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:588: done (10 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:590: done (9 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:591: done (8 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:595: done (7 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:598: done (6 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:599: done (5 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:600: done (4 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:601: done (3 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:602: done (2 left) 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:603: cache_try_flush oid 226 snap -1 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:604: delete oid 490 current snap is 0 2024-09-21T18:07:44.990 INFO:tasks.rados.rados.0.smithi160.stdout:603: got (2) No such file or directory 2024-09-21T18:07:45.006 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:45.006 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:45.014 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:45.010+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:45.024 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:45.025 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:45.041 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:45.041 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:45.062 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:45.063 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:45.072 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:45.072 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:45.072 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:45.072 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:45.072 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: finishing write tid 5 to smithi16030472-126 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: oid 126 updating version 0 to 421 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: oid 126 updating version 421 to 427 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: oid 126 version 427 is already newer than 426 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: oid 126 version 427 is already newer than 422 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 126 v 427 (ObjNum 527 snap 0 seq_num 527) dirty exists 2024-09-21T18:07:45.079 INFO:tasks.rados.rados.0.smithi160.stdout:593: left oid 126 (ObjNum 527 snap 0 seq_num 527) 2024-09-21T18:07:45.084 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:45.084 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:45.085 INFO:tasks.rados.rados.0.smithi160.stdout:596: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:07:45.086 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:45.106 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:45.124 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:45.130 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:45.126+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:45.146 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:45.147 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:45.165 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:45.189 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:45.189 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:45.190 INFO:tasks.rados.rados.0.smithi160.stdout:596: finishing copy_from to smithi16030472-456 2024-09-21T18:07:45.190 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 310 (ObjNum 361 snap 0 seq_num 361) dirty exists 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:593: done (3 left) 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:596: done (2 left) 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:603: done (1 left) 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:604: done (0 left) 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:605: copy_from oid 90 from oid 372 current snap is 0 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:606: copy_from oid 21 from oid 369 current snap is 0 2024-09-21T18:07:45.193 INFO:tasks.rados.rados.0.smithi160.stdout:607: delete oid 168 current snap is 0 2024-09-21T18:07:45.215 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:45.210+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:45.254 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:45.254 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:45.268 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:45.316 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:45.313+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:45.397 INFO:tasks.rados.rados.0.smithi160.stdout:607: done (2 left) 2024-09-21T18:07:45.397 INFO:tasks.rados.rados.0.smithi160.stdout:608: delete oid 491 current snap is 0 2024-09-21T18:07:45.398 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:45.398 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:45.406 INFO:tasks.rados.rados.0.smithi160.stdout:606: finishing copy_from racing read to smithi16030472-21 2024-09-21T18:07:45.414 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:07:45.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:45.417 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:45.410+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:45.419 INFO:tasks.rados.rados.0.smithi160.stdout:605: finishing copy_from racing read to smithi16030472-90 2024-09-21T18:07:45.502 INFO:tasks.rados.rados.0.smithi160.stdout:608: done (2 left) 2024-09-21T18:07:45.502 INFO:tasks.rados.rados.0.smithi160.stdout:609: read oid 291 snap -1 2024-09-21T18:07:45.502 INFO:tasks.rados.rados.0.smithi160.stdout:609: expect (ObjNum 290 snap 0 seq_num 290) 2024-09-21T18:07:45.503 INFO:tasks.rados.rados.0.smithi160.stdout:610: read oid 79 snap -1 2024-09-21T18:07:45.503 INFO:tasks.rados.rados.0.smithi160.stdout:610: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-21T18:07:45.503 INFO:tasks.rados.rados.0.smithi160.stdout:611: cache_flush oid 498 snap -1 2024-09-21T18:07:45.503 INFO:tasks.rados.rados.0.smithi160.stdout:612: write (excl) oid 17 current snap is 0 2024-09-21T18:07:45.503 INFO:tasks.rados.rados.0.smithi160.stdout:612: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-09-21T18:07:45.508 INFO:tasks.rados.rados.0.smithi160.stdout:612: writing smithi16030472-17 from 541324 to 1002149 tid 1 2024-09-21T18:07:45.514 INFO:tasks.rados.rados.0.smithi160.stdout:612: writing smithi16030472-17 from 1498538 to 1914726 tid 2 2024-09-21T18:07:45.517 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:45.514+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:612: writing smithi16030472-17 from 2393385 to 2941324 tid 3 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:613: copy_from oid 320 from oid 425 current snap is 0 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:605: finishing copy_from to smithi16030472-90 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 90 v 262 (ObjNum 371 snap 0 seq_num 371) dirty exists 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:605: done (6 left) 2024-09-21T18:07:45.521 INFO:tasks.rados.rados.0.smithi160.stdout:614: copy_from oid 374 from oid 54 current snap is 0 2024-09-21T18:07:45.522 INFO:tasks.rados.rados.0.smithi160.stdout:615: write (excl) oid 400 current snap is 0 2024-09-21T18:07:45.522 INFO:tasks.rados.rados.0.smithi160.stdout:615: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-09-21T18:07:45.529 INFO:tasks.rados.rados.0.smithi160.stdout:615: writing smithi16030472-400 from 558131 to 1210558 tid 1 2024-09-21T18:07:45.536 INFO:tasks.rados.rados.0.smithi160.stdout:615: writing smithi16030472-400 from 1673373 to 2233219 tid 2 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:615: writing smithi16030472-400 from 2820808 to 2958131 tid 3 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:616: cache_try_flush oid 410 snap -1 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:617: copy_from oid 323 from oid 9 current snap is 0 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:618: cache_evict oid 302 snap -1 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:619: write oid 119 current snap is 0 2024-09-21T18:07:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:619: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-21T18:07:45.552 INFO:tasks.rados.rados.0.smithi160.stdout:619: writing smithi16030472-119 from 574938 to 1302614 tid 1 2024-09-21T18:07:45.553 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:45.553 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:45.559 INFO:tasks.rados.rados.0.smithi160.stdout:619: writing smithi16030472-119 from 1731855 to 2435359 tid 2 2024-09-21T18:07:45.563 INFO:tasks.rados.rados.0.smithi160.stdout:619: writing smithi16030472-119 from 2848231 to 2974938 tid 3 2024-09-21T18:07:45.563 INFO:tasks.rados.rados.0.smithi160.stdout:620: delete oid 69 current snap is 0 2024-09-21T18:07:45.567 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:45.604 INFO:tasks.rados.rados.0.smithi160.stdout:611: got (0) Success 2024-09-21T18:07:45.604 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 498 v 0 (ObjNum 497 snap 0 seq_num 497) dirty exists 2024-09-21T18:07:45.618 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:45.613+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:45.707 INFO:tasks.rados.rados.0.smithi160.stdout:610: expect (ObjNum 78 snap 0 seq_num 78) 2024-09-21T18:07:45.719 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:45.714+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:45.730 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:45.731 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:45.740 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2012078 2024-09-21T18:07:45.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:45.773 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:45.820 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:45.814+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:45.851 INFO:tasks.rados.rados.0.smithi160.stdout:616: got (0) Success 2024-09-21T18:07:45.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 0 (ObjNum 409 snap 0 seq_num 409) dirty exists 2024-09-21T18:07:45.851 INFO:tasks.rados.rados.0.smithi160.stdout:615: finishing write tid 1 to smithi16030472-400 2024-09-21T18:07:45.852 INFO:tasks.rados.rados.0.smithi160.stdout:617: finishing copy_from racing read to smithi16030472-323 2024-09-21T18:07:45.852 INFO:tasks.rados.rados.0.smithi160.stdout:618: got (0) Success 2024-09-21T18:07:45.852 INFO:tasks.rados.rados.0.smithi160.stdout:606: finishing copy_from to smithi16030472-21 2024-09-21T18:07:45.852 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 251 (ObjNum 368 snap 0 seq_num 368) dirty exists 2024-09-21T18:07:45.852 INFO:tasks.rados.rados.0.smithi160.stdout:609: expect (ObjNum 290 snap 0 seq_num 290) 2024-09-21T18:07:45.865 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:45.865 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:45.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:45.904 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1726868 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:614: finishing copy_from racing read to smithi16030472-374 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:606: done (12 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:609: done (11 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:610: done (10 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:611: done (9 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:616: done (8 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:618: done (7 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:620: done (6 left) 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:621: write (excl) oid 317 current snap is 0 2024-09-21T18:07:45.917 INFO:tasks.rados.rados.0.smithi160.stdout:621: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-21T18:07:45.920 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:45.917+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:45.926 INFO:tasks.rados.rados.0.smithi160.stdout:621: writing smithi16030472-317 from 608552 to 1086726 tid 1 2024-09-21T18:07:45.936 INFO:tasks.rados.rados.0.smithi160.stdout:621: writing smithi16030472-317 from 1848819 to 2555992 tid 2 2024-09-21T18:07:45.937 INFO:tasks.rados.rados.0.smithi160.stdout:622: write (excl) oid 259 current snap is 0 2024-09-21T18:07:45.937 INFO:tasks.rados.rados.0.smithi160.stdout:614: finishing copy_from to smithi16030472-374 2024-09-21T18:07:45.937 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 374 v 34 (ObjNum 53 snap 0 seq_num 53) dirty exists 2024-09-21T18:07:45.937 INFO:tasks.rados.rados.0.smithi160.stdout:622: seq_num 537 ranges {625359=553423,1790948=567184,2930500=94859} 2024-09-21T18:07:45.944 INFO:tasks.rados.rados.0.smithi160.stdout:622: writing smithi16030472-259 from 625359 to 1178782 tid 1 2024-09-21T18:07:45.953 INFO:tasks.rados.rados.0.smithi160.stdout:622: writing smithi16030472-259 from 1790948 to 2358132 tid 2 2024-09-21T18:07:45.957 INFO:tasks.rados.rados.0.smithi160.stdout:622: writing smithi16030472-259 from 2930500 to 3025359 tid 3 2024-09-21T18:07:45.957 INFO:tasks.rados.rados.0.smithi160.stdout:614: done (7 left) 2024-09-21T18:07:45.958 INFO:tasks.rados.rados.0.smithi160.stdout:623: cache_evict oid 66 snap -1 2024-09-21T18:07:45.958 INFO:tasks.rados.rados.0.smithi160.stdout:612: finishing write tid 1 to smithi16030472-17 2024-09-21T18:07:45.958 INFO:tasks.rados.rados.0.smithi160.stdout:624: write (excl) oid 446 current snap is 0 2024-09-21T18:07:45.958 INFO:tasks.rados.rados.0.smithi160.stdout:624: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-21T18:07:45.965 INFO:tasks.rados.rados.0.smithi160.stdout:624: writing smithi16030472-446 from 642166 to 1270838 tid 1 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:624: writing smithi16030472-446 from 1849430 to 2560272 tid 2 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:625: copy_from oid 412 from oid 410 current snap is 0 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:619: finishing write tid 1 to smithi16030472-119 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:615: finishing write tid 2 to smithi16030472-400 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:615: finishing write tid 3 to smithi16030472-400 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:615: finishing write tid 4 to smithi16030472-400 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:617: finishing copy_from to smithi16030472-323 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 323 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:617: done (9 left) 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:626: write (excl) oid 11 current snap is 0 2024-09-21T18:07:45.973 INFO:tasks.rados.rados.0.smithi160.stdout:626: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-21T18:07:45.980 INFO:tasks.rados.rados.0.smithi160.stdout:626: writing smithi16030472-11 from 658973 to 1362894 tid 1 2024-09-21T18:07:45.986 INFO:tasks.rados.rados.0.smithi160.stdout:626: writing smithi16030472-11 from 1907912 to 2478765 tid 2 2024-09-21T18:07:45.990 INFO:tasks.rados.rados.0.smithi160.stdout:626: writing smithi16030472-11 from 2985346 to 3058973 tid 3 2024-09-21T18:07:45.990 INFO:tasks.rados.rados.0.smithi160.stdout:627: write (excl) oid 220 current snap is 0 2024-09-21T18:07:45.990 INFO:tasks.rados.rados.0.smithi160.stdout:627: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-21T18:07:45.996 INFO:tasks.rados.rados.0.smithi160.stdout:627: writing smithi16030472-220 from 675780 to 1171303 tid 1 2024-09-21T18:07:46.003 INFO:tasks.rados.rados.0.smithi160.stdout:627: writing smithi16030472-220 from 1682747 to 2397258 tid 2 2024-09-21T18:07:46.007 INFO:tasks.rados.rados.0.smithi160.stdout:627: writing smithi16030472-220 from 3012769 to 3075780 tid 3 2024-09-21T18:07:46.007 INFO:tasks.rados.rados.0.smithi160.stdout:628: delete oid 23 current snap is 0 2024-09-21T18:07:46.010 INFO:tasks.rados.rados.0.smithi160.stdout:628: done (11 left) 2024-09-21T18:07:46.010 INFO:tasks.rados.rados.0.smithi160.stdout:629: cache_flush oid 395 snap -1 2024-09-21T18:07:46.011 INFO:tasks.rados.rados.0.smithi160.stdout:630: delete oid 203 current snap is 0 2024-09-21T18:07:46.021 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:46.016+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:46.036 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.036 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:46.122 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:46.117+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:46.171 INFO:tasks.rados.rados.0.smithi160.stdout:624: finishing write tid 1 to smithi16030472-446 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:612: finishing write tid 2 to smithi16030472-17 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:612: finishing write tid 3 to smithi16030472-17 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:613: finishing copy_from racing read to smithi16030472-320 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:623: got (2) No such file or directory 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:612: finishing write tid 4 to smithi16030472-17 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:613: finishing copy_from to smithi16030472-320 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 320 v 342 (ObjNum 424 snap 0 seq_num 424) dirty exists 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:619: finishing write tid 2 to smithi16030472-119 2024-09-21T18:07:46.193 INFO:tasks.rados.rados.0.smithi160.stdout:612: finishing write tid 5 to smithi16030472-17 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:612: oid 17 updating version 0 to 327 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:612: oid 17 version 327 is already newer than 326 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:612: oid 17 updating version 327 to 328 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:612: oid 17 updating version 328 to 329 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 17 v 329 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:612: left oid 17 (ObjNum 532 snap 0 seq_num 532) 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:622: finishing write tid 1 to smithi16030472-259 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:621: finishing write tid 1 to smithi16030472-317 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:621: finishing write tid 2 to smithi16030472-317 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:622: finishing write tid 2 to smithi16030472-259 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:622: finishing write tid 3 to smithi16030472-259 2024-09-21T18:07:46.194 INFO:tasks.rados.rados.0.smithi160.stdout:622: finishing write tid 4 to smithi16030472-259 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: finishing write tid 3 to smithi16030472-317 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: finishing write tid 4 to smithi16030472-317 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: oid 317 updating version 0 to 401 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: oid 317 updating version 401 to 403 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: oid 317 version 403 is already newer than 402 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 403 (ObjNum 536 snap 0 seq_num 536) dirty exists 2024-09-21T18:07:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:621: left oid 317 (ObjNum 536 snap 0 seq_num 536) 2024-09-21T18:07:46.200 INFO:tasks.rados.rados.0.smithi160.stdout:625: finishing copy_from racing read to smithi16030472-412 2024-09-21T18:07:46.223 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:46.218+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:46.224 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.224 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:612: done (12 left) 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:613: done (11 left) 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:621: done (10 left) 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:623: done (9 left) 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:630: done (8 left) 2024-09-21T18:07:46.234 INFO:tasks.rados.rados.0.smithi160.stdout:631: cache_evict oid 280 snap -1 2024-09-21T18:07:46.235 INFO:tasks.rados.rados.0.smithi160.stdout:632: delete oid 54 current snap is 0 2024-09-21T18:07:46.235 INFO:tasks.rados.rados.0.smithi160.stdout:631: got (2) No such file or directory 2024-09-21T18:07:46.239 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:46.240 INFO:tasks.rados.rados.0.smithi160.stdout:625: finishing copy_from to smithi16030472-412 2024-09-21T18:07:46.241 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 335 (ObjNum 409 snap 0 seq_num 409) dirty exists 2024-09-21T18:07:46.281 INFO:tasks.rados.rados.0.smithi160.stdout:624: finishing write tid 2 to smithi16030472-446 2024-09-21T18:07:46.282 INFO:tasks.rados.rados.0.smithi160.stdout:624: finishing write tid 3 to smithi16030472-446 2024-09-21T18:07:46.282 INFO:tasks.rados.rados.0.smithi160.stdout:624: finishing write tid 4 to smithi16030472-446 2024-09-21T18:07:46.283 INFO:tasks.rados.rados.0.smithi160.stdout:624: oid 446 updating version 0 to 427 2024-09-21T18:07:46.283 INFO:tasks.rados.rados.0.smithi160.stdout:624: oid 446 version 427 is already newer than 426 2024-09-21T18:07:46.283 INFO:tasks.rados.rados.0.smithi160.stdout:624: oid 446 version 427 is already newer than 425 2024-09-21T18:07:46.283 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 427 (ObjNum 538 snap 0 seq_num 538) dirty exists 2024-09-21T18:07:46.283 INFO:tasks.rados.rados.0.smithi160.stdout:624: left oid 446 (ObjNum 538 snap 0 seq_num 538) 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: finishing write tid 5 to smithi16030472-400 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: oid 400 updating version 0 to 337 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: oid 400 version 337 is already newer than 336 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: oid 400 version 337 is already newer than 334 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: oid 400 version 337 is already newer than 335 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 400 v 337 (ObjNum 533 snap 0 seq_num 533) dirty exists 2024-09-21T18:07:46.285 INFO:tasks.rados.rados.0.smithi160.stdout:615: left oid 400 (ObjNum 533 snap 0 seq_num 533) 2024-09-21T18:07:46.293 INFO:tasks.rados.rados.0.smithi160.stdout:619: finishing write tid 3 to smithi16030472-119 2024-09-21T18:07:46.293 INFO:tasks.rados.rados.0.smithi160.stdout:627: finishing write tid 1 to smithi16030472-220 2024-09-21T18:07:46.293 INFO:tasks.rados.rados.0.smithi160.stdout:619: finishing write tid 4 to smithi16030472-119 2024-09-21T18:07:46.302 INFO:tasks.rados.rados.0.smithi160.stdout:626: finishing write tid 1 to smithi16030472-11 2024-09-21T18:07:46.302 INFO:tasks.rados.rados.0.smithi160.stdout:626: finishing write tid 2 to smithi16030472-11 2024-09-21T18:07:46.303 INFO:tasks.rados.rados.0.smithi160.stdout:626: finishing write tid 3 to smithi16030472-11 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:626: finishing write tid 4 to smithi16030472-11 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:615: done (9 left) 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:624: done (8 left) 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:625: done (7 left) 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:631: done (6 left) 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:632: done (5 left) 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:633: cache_evict oid 289 snap -1 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:634: cache_evict oid 44 snap -1 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:635: cache_evict oid 33 snap -1 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:636: copy_from oid 35 from oid 89 current snap is 0 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:637: read oid 311 snap -1 2024-09-21T18:07:46.305 INFO:tasks.rados.rados.0.smithi160.stdout:637: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:638: cache_flush oid 367 snap -1 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: finishing write tid 5 to smithi16030472-11 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: oid 11 updating version 0 to 331 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: oid 11 updating version 331 to 332 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: oid 11 updating version 332 to 333 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: oid 11 version 333 is already newer than 330 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 11 v 333 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: left oid 11 (ObjNum 539 snap 0 seq_num 539) 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:626: done (10 left) 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:639: cache_try_flush oid 152 snap -1 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:640: delete oid 293 current snap is 0 2024-09-21T18:07:46.306 INFO:tasks.rados.rados.0.smithi160.stdout:633: got (2) No such file or directory 2024-09-21T18:07:46.307 INFO:tasks.rados.rados.0.smithi160.stdout:638: got (0) Success 2024-09-21T18:07:46.307 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 0 (ObjNum 507 snap 0 seq_num 507) dirty exists 2024-09-21T18:07:46.324 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:46.318+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:46.361 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.361 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.376 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:46.404 INFO:tasks.rados.rados.0.smithi160.stdout:634: got (0) Success 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:619: finishing write tid 5 to smithi16030472-119 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:619: oid 119 updating version 0 to 429 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:619: oid 119 updating version 429 to 430 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:619: oid 119 version 430 is already newer than 425 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:619: oid 119 version 430 is already newer than 424 2024-09-21T18:07:46.405 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 119 v 430 (ObjNum 534 snap 0 seq_num 534) dirty exists 2024-09-21T18:07:46.406 INFO:tasks.rados.rados.0.smithi160.stdout:619: left oid 119 (ObjNum 534 snap 0 seq_num 534) 2024-09-21T18:07:46.406 INFO:tasks.rados.rados.0.smithi160.stdout:635: got (0) Success 2024-09-21T18:07:46.406 INFO:tasks.rados.rados.0.smithi160.stdout:639: got (2) No such file or directory 2024-09-21T18:07:46.450 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:46.446+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:46.494 INFO:tasks.rados.rados.0.smithi160.stdout:622: finishing write tid 5 to smithi16030472-259 2024-09-21T18:07:46.494 INFO:tasks.rados.rados.0.smithi160.stdout:622: oid 259 updating version 0 to 428 2024-09-21T18:07:46.494 INFO:tasks.rados.rados.0.smithi160.stdout:622: oid 259 version 428 is already newer than 426 2024-09-21T18:07:46.495 INFO:tasks.rados.rados.0.smithi160.stdout:622: oid 259 updating version 428 to 429 2024-09-21T18:07:46.495 INFO:tasks.rados.rados.0.smithi160.stdout:622: oid 259 version 429 is already newer than 427 2024-09-21T18:07:46.495 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 259 v 429 (ObjNum 537 snap 0 seq_num 537) dirty exists 2024-09-21T18:07:46.495 INFO:tasks.rados.rados.0.smithi160.stdout:622: left oid 259 (ObjNum 537 snap 0 seq_num 537) 2024-09-21T18:07:46.523 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.524 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.526 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:46.522+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:46.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:629: got (0) Success 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 395 v 0 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:627: finishing write tid 2 to smithi16030472-220 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:627: finishing write tid 3 to smithi16030472-220 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:619: done (11 left) 2024-09-21T18:07:46.595 INFO:tasks.rados.rados.0.smithi160.stdout:622: done (10 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:629: done (9 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:633: done (8 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:634: done (7 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:635: done (6 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:638: done (5 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:639: done (4 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:640: done (3 left) 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:641: write (excl) oid 382 current snap is 0 2024-09-21T18:07:46.596 INFO:tasks.rados.rados.0.smithi160.stdout:641: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-21T18:07:46.605 INFO:tasks.rados.rados.0.smithi160.stdout:641: writing smithi16030472-382 from 759815 to 1231583 tid 1 2024-09-21T18:07:46.613 INFO:tasks.rados.rados.0.smithi160.stdout:641: writing smithi16030472-382 from 1742451 to 2207958 tid 2 2024-09-21T18:07:46.621 INFO:tasks.rados.rados.0.smithi160.stdout:641: writing smithi16030472-382 from 2917178 to 3159815 tid 3 2024-09-21T18:07:46.621 INFO:tasks.rados.rados.0.smithi160.stdout:642: cache_evict oid 376 snap -1 2024-09-21T18:07:46.621 INFO:tasks.rados.rados.0.smithi160.stdout:636: finishing copy_from racing read to smithi16030472-35 2024-09-21T18:07:46.621 INFO:tasks.rados.rados.0.smithi160.stdout:636: finishing copy_from to smithi16030472-35 2024-09-21T18:07:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 25 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-21T18:07:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:636: done (4 left) 2024-09-21T18:07:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:643: delete oid 126 current snap is 0 2024-09-21T18:07:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:642: got (0) Success 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:642: done (4 left) 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:643: done (3 left) 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:644: cache_flush oid 234 snap -1 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:645: read oid 136 snap -1 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:645: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:646: copy_from oid 279 from oid 218 current snap is 0 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:647: read oid 321 snap -1 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:647: expect (ObjNum 320 snap 0 seq_num 320) 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:648: write (excl) oid 406 current snap is 0 2024-09-21T18:07:46.625 INFO:tasks.rados.rados.0.smithi160.stdout:648: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-09-21T18:07:46.627 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:46.624+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:46.637 INFO:tasks.rados.rados.0.smithi160.stdout:648: writing smithi16030472-406 from 793429 to 1415695 tid 1 2024-09-21T18:07:46.645 INFO:tasks.rados.rados.0.smithi160.stdout:648: writing smithi16030472-406 from 1859415 to 2328591 tid 2 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:648: writing smithi16030472-406 from 2972024 to 3193429 tid 3 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:649: read oid 306 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:649: expect (ObjNum 305 snap 0 seq_num 305) 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:650: read oid 373 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:650: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:651: cache_flush oid 389 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:652: read oid 448 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:652: expect (ObjNum 447 snap 0 seq_num 447) 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:653: read oid 308 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:653: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:654: read oid 483 snap -1 2024-09-21T18:07:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:654: expect deleted 2024-09-21T18:07:46.654 INFO:tasks.rados.rados.0.smithi160.stdout:655: write (excl) oid 111 current snap is 0 2024-09-21T18:07:46.654 INFO:tasks.rados.rados.0.smithi160.stdout:655: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-21T18:07:46.659 INFO:tasks.rados.rados.0.smithi160.stdout:655: writing smithi16030472-111 from 410236 to 824104 tid 1 2024-09-21T18:07:46.667 INFO:tasks.rados.rados.0.smithi160.stdout:655: writing smithi16030472-111 from 1234250 to 1963437 tid 2 2024-09-21T18:07:46.676 INFO:tasks.rados.rados.0.smithi160.stdout:655: writing smithi16030472-111 from 2715800 to 3138863 tid 3 2024-09-21T18:07:46.676 INFO:tasks.rados.rados.0.smithi160.stdout:656: read oid 284 snap -1 2024-09-21T18:07:46.676 INFO:tasks.rados.rados.0.smithi160.stdout:656: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-21T18:07:46.676 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:46.677 INFO:tasks.rados.rados.0.smithi160.stdout:645: expect (ObjNum 135 snap 0 seq_num 135) 2024-09-21T18:07:46.703 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.703 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.720 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:46.727 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:46.722+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:46.733 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:46.777 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1877470 2024-09-21T18:07:46.781 INFO:tasks.rados.rados.0.smithi160.stdout:645: done (15 left) 2024-09-21T18:07:46.781 INFO:tasks.rados.rados.0.smithi160.stdout:657: delete oid 447 current snap is 0 2024-09-21T18:07:46.828 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:46.822+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:46.850 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:46.850 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:46.865 INFO:tasks.rados.rados.0.smithi160.stdout:653: expect (ObjNum 307 snap 0 seq_num 307) 2024-09-21T18:07:46.866 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:46.868 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:46.941 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 598582 2024-09-21T18:07:46.942 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:46.938+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:47.001 INFO:tasks.rados.rados.0.smithi160.stdout:648: finishing write tid 1 to smithi16030472-406 2024-09-21T18:07:47.001 INFO:tasks.rados.rados.0.smithi160.stdout:656: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-21T18:07:47.030 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:47.026+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:47.038 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:47.038 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:47.052 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:47.059 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: finishing write tid 4 to smithi16030472-220 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: finishing write tid 5 to smithi16030472-220 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: oid 220 updating version 0 to 433 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: oid 220 version 433 is already newer than 430 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: oid 220 version 433 is already newer than 431 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:627: oid 220 version 433 is already newer than 432 2024-09-21T18:07:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 433 (ObjNum 540 snap 0 seq_num 540) dirty exists 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:627: left oid 220 (ObjNum 540 snap 0 seq_num 540) 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: finishing write tid 1 to smithi16030472-382 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: finishing write tid 2 to smithi16030472-382 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: finishing write tid 3 to smithi16030472-382 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: finishing write tid 4 to smithi16030472-382 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: finishing write tid 5 to smithi16030472-382 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: oid 382 updating version 0 to 406 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: oid 382 updating version 406 to 407 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: oid 382 updating version 407 to 408 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: oid 382 updating version 408 to 409 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 382 v 409 (ObjNum 545 snap 0 seq_num 545) dirty exists 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:641: left oid 382 (ObjNum 545 snap 0 seq_num 545) 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:647: expect (ObjNum 320 snap 0 seq_num 320) 2024-09-21T18:07:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.128 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.131 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:47.126+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:47.172 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2653569 2024-09-21T18:07:47.185 INFO:tasks.rados.rados.0.smithi160.stdout:649: expect (ObjNum 305 snap 0 seq_num 305) 2024-09-21T18:07:47.189 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1670850 2024-09-21T18:07:47.195 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:47.195 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:47.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:47.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:47.213 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 726238 2024-09-21T18:07:47.232 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:47.229+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:47.272 INFO:tasks.rados.rados.0.smithi160.stdout:655: finishing write tid 1 to smithi16030472-111 2024-09-21T18:07:47.272 INFO:tasks.rados.rados.0.smithi160.stdout:648: finishing write tid 2 to smithi16030472-406 2024-09-21T18:07:47.272 INFO:tasks.rados.rados.0.smithi160.stdout:648: finishing write tid 3 to smithi16030472-406 2024-09-21T18:07:47.272 INFO:tasks.rados.rados.0.smithi160.stdout:637: expect (ObjNum 310 snap 0 seq_num 310) 2024-09-21T18:07:47.284 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2203420 2024-09-21T18:07:47.296 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.330 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.333 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:47.329+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:47.351 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:47.352 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:47.363 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:47.366 INFO:tasks.rados.rados.0.smithi160.stdout:652: expect (ObjNum 447 snap 0 seq_num 447) 2024-09-21T18:07:47.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:47.384 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:47.404 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:47.405 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:47.415 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:47.413+0000 7f4978027640 -1 osd.5 56 handle_osd_map injecting map crc failure 2024-09-21T18:07:47.416 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:47.410+0000 7fca50942640 -1 osd.2 56 handle_osd_map injecting map crc failure 2024-09-21T18:07:47.426 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:47.434 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:47.428+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:47.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:07:47.441 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.443 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:47.443 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:651: got (0) Success 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 389 v 0 (ObjNum 388 snap 0 seq_num 388) dirty exists 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: finishing write tid 4 to smithi16030472-406 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: finishing write tid 5 to smithi16030472-406 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: oid 406 updating version 0 to 344 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: oid 406 updating version 344 to 347 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: oid 406 version 347 is already newer than 345 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: oid 406 version 347 is already newer than 346 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 406 v 347 (ObjNum 547 snap 0 seq_num 547) dirty exists 2024-09-21T18:07:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:648: left oid 406 (ObjNum 547 snap 0 seq_num 547) 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:646: finishing copy_from to smithi16030472-279 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 279 v 143 (ObjNum 217 snap 0 seq_num 217) dirty exists 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:644: got (0) Success 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 234 v 0 (ObjNum 233 snap 0 seq_num 233) dirty exists 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:646: finishing copy_from racing read to smithi16030472-279 2024-09-21T18:07:47.481 INFO:tasks.rados.rados.0.smithi160.stdout:650: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-21T18:07:47.481 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:47.481 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:47.503 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.507 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:47.507 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:47.507 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:47.507 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:47.507 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:47.516 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:47.516 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:47.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:47.531 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1507707 2024-09-21T18:07:47.535 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:47.533+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:47.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:47.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:47.544 INFO:tasks.rados.rados.0.smithi160.stdout:655: finishing write tid 2 to smithi16030472-111 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:655: finishing write tid 3 to smithi16030472-111 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:627: done (15 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:637: done (14 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:641: done (13 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:644: done (12 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:646: done (11 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:647: done (10 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:648: done (9 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:649: done (8 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:650: done (7 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:651: done (6 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:652: done (5 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:653: done (4 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:654: done (3 left) 2024-09-21T18:07:47.545 INFO:tasks.rados.rados.0.smithi160.stdout:656: done (2 left) 2024-09-21T18:07:47.546 INFO:tasks.rados.rados.0.smithi160.stdout:657: done (1 left) 2024-09-21T18:07:47.546 INFO:tasks.rados.rados.0.smithi160.stdout:658: write oid 377 current snap is 0 2024-09-21T18:07:47.546 INFO:tasks.rados.rados.0.smithi160.stdout:658: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-09-21T18:07:47.555 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:47.556 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:47.560 INFO:tasks.rados.rados.0.smithi160.stdout:658: writing smithi16030472-377 from 443850 to 1008216 tid 1 2024-09-21T18:07:47.578 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:47.583 INFO:tasks.rados.rados.0.smithi160.stdout:658: writing smithi16030472-377 from 1634861 to 2367717 tid 2 2024-09-21T18:07:47.597 INFO:tasks.rados.rados.0.smithi160.stdout:658: writing smithi16030472-377 from 2770646 to 3243850 tid 3 2024-09-21T18:07:47.598 INFO:tasks.rados.rados.0.smithi160.stdout:659: write oid 366 current snap is 0 2024-09-21T18:07:47.598 INFO:tasks.rados.rados.0.smithi160.stdout:659: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-21T18:07:47.598 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:47.611 INFO:tasks.rados.rados.0.smithi160.stdout:659: writing smithi16030472-366 from 460657 to 1100272 tid 1 2024-09-21T18:07:47.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:47.620 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:47.621 INFO:tasks.rados.rados.0.smithi160.stdout:659: writing smithi16030472-366 from 1693343 to 2169857 tid 2 2024-09-21T18:07:47.631 INFO:tasks.rados.rados.0.smithi160.stdout:659: writing smithi16030472-366 from 2681716 to 3260657 tid 3 2024-09-21T18:07:47.631 INFO:tasks.rados.rados.0.smithi160.stdout:660: read oid 432 snap -1 2024-09-21T18:07:47.631 INFO:tasks.rados.rados.0.smithi160.stdout:660: expect (ObjNum 431 snap 0 seq_num 431) 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:661: cache_flush oid 455 snap -1 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: finishing write tid 4 to smithi16030472-111 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: finishing write tid 5 to smithi16030472-111 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: oid 111 updating version 0 to 345 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: oid 111 updating version 345 to 349 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: oid 111 version 349 is already newer than 348 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: oid 111 updating version 349 to 350 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 111 v 350 (ObjNum 548 snap 0 seq_num 548) dirty exists 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: left oid 111 (ObjNum 548 snap 0 seq_num 548) 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:658: finishing write tid 1 to smithi16030472-377 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:658: finishing write tid 2 to smithi16030472-377 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:655: done (4 left) 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:662: cache_evict oid 363 snap -1 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:663: write (excl) oid 8 current snap is 0 2024-09-21T18:07:47.632 INFO:tasks.rados.rados.0.smithi160.stdout:663: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-21T18:07:47.636 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:47.630+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:47.638 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:47.638 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:47.642 INFO:tasks.rados.rados.0.smithi160.stdout:663: writing smithi16030472-8 from 477464 to 1192328 tid 1 2024-09-21T18:07:47.656 INFO:tasks.rados.rados.0.smithi160.stdout:663: writing smithi16030472-8 from 1751825 to 2488350 tid 2 2024-09-21T18:07:47.659 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:47.659 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:47.660 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:47.661 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:47.661 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:47.661 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:47.661 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:47.665 INFO:tasks.rados.rados.0.smithi160.stdout:663: writing smithi16030472-8 from 3225492 to 3277464 tid 3 2024-09-21T18:07:47.665 INFO:tasks.rados.rados.0.smithi160.stdout:664: write oid 484 current snap is 0 2024-09-21T18:07:47.665 INFO:tasks.rados.rados.0.smithi160.stdout:664: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-21T18:07:47.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:47.676 INFO:tasks.rados.rados.0.smithi160.stdout:664: writing smithi16030472-484 from 494271 to 1284384 tid 1 2024-09-21T18:07:47.680 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:47.683 INFO:tasks.rados.rados.0.smithi160.stdout:664: writing smithi16030472-484 from 1693954 to 2174137 tid 2 2024-09-21T18:07:47.690 INFO:tasks.rados.rados.0.smithi160.stdout:664: writing smithi16030472-484 from 2620209 to 3132338 tid 3 2024-09-21T18:07:47.690 INFO:tasks.rados.rados.0.smithi160.stdout:665: read oid 309 snap -1 2024-09-21T18:07:47.690 INFO:tasks.rados.rados.0.smithi160.stdout:665: expect (ObjNum 308 snap 0 seq_num 308) 2024-09-21T18:07:47.690 INFO:tasks.rados.rados.0.smithi160.stdout:666: cache_try_flush oid 70 snap -1 2024-09-21T18:07:47.691 INFO:tasks.rados.rados.0.smithi160.stdout:662: got (16) Device or resource busy 2024-09-21T18:07:47.691 INFO:tasks.rados.rados.0.smithi160.stdout:662: done (8 left) 2024-09-21T18:07:47.691 INFO:tasks.rados.rados.0.smithi160.stdout:667: read oid 196 snap -1 2024-09-21T18:07:47.691 INFO:tasks.rados.rados.0.smithi160.stdout:667: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-21T18:07:47.691 INFO:tasks.rados.rados.0.smithi160.stdout:668: delete oid 126 current snap is 0 2024-09-21T18:07:47.700 INFO:tasks.rados.rados.0.smithi160.stdout:665: expect (ObjNum 308 snap 0 seq_num 308) 2024-09-21T18:07:47.728 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:47.728 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:47.736 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:47.732+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:47.742 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:1245 3.1 3.9d89b61d (undecoded) ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e57)", 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:47.731657+0000", 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.066520836, 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.066545666000000003, 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "op must wait for map", 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731657+0000", 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731657+0000", 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731661+0000", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.754e-06 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731673+0000", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2747e-05 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731695+0000", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2133999999999998e-05 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731708+0000", 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2787e-05 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731785+0000", 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6540999999999998e-05 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:47.731789+0000", 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5970000000000002e-06 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:07:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 1 2024-09-21T18:07:47.803 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:47.809 INFO:tasks.rados.rados.0.smithi160.stdout:659: finishing write tid 1 to smithi16030472-366 2024-09-21T18:07:47.809 INFO:tasks.rados.rados.0.smithi160.stdout:659: finishing write tid 2 to smithi16030472-366 2024-09-21T18:07:47.809 INFO:tasks.rados.rados.0.smithi160.stdout:659: finishing write tid 3 to smithi16030472-366 2024-09-21T18:07:47.811 INFO:tasks.rados.rados.0.smithi160.stdout:659: finishing write tid 4 to smithi16030472-366 2024-09-21T18:07:47.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: finishing write tid 3 to smithi16030472-377 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: finishing write tid 4 to smithi16030472-377 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: finishing write tid 5 to smithi16030472-377 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: oid 377 updating version 0 to 413 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: oid 377 updating version 413 to 415 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: oid 377 version 415 is already newer than 412 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: oid 377 version 415 is already newer than 414 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 377 v 415 (ObjNum 550 snap 0 seq_num 550) dirty exists 2024-09-21T18:07:47.834 INFO:tasks.rados.rados.0.smithi160.stdout:658: left oid 377 (ObjNum 550 snap 0 seq_num 550) 2024-09-21T18:07:47.842 INFO:tasks.rados.rados.0.smithi160.stdout:666: got (2) No such file or directory 2024-09-21T18:07:47.858 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:47.856+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: finishing write tid 5 to smithi16030472-366 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: oid 366 updating version 0 to 352 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: oid 366 version 352 is already newer than 351 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: oid 366 updating version 352 to 354 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: oid 366 version 354 is already newer than 353 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 354 (ObjNum 551 snap 0 seq_num 551) dirty exists 2024-09-21T18:07:47.912 INFO:tasks.rados.rados.0.smithi160.stdout:659: left oid 366 (ObjNum 551 snap 0 seq_num 551) 2024-09-21T18:07:47.935 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:47.935 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:47.935 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:47.935 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:47.935 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:47.938 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:47.936+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:47.939 INFO:tasks.rados.rados.0.smithi160.stdout:660: expect (ObjNum 431 snap 0 seq_num 431) 2024-09-21T18:07:47.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:48.039 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:48.036+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:48.055 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2446378 2024-09-21T18:07:48.068 INFO:tasks.rados.rados.0.smithi160.stdout:663: finishing write tid 1 to smithi16030472-8 2024-09-21T18:07:48.068 INFO:tasks.rados.rados.0.smithi160.stdout:663: finishing write tid 2 to smithi16030472-8 2024-09-21T18:07:48.068 INFO:tasks.rados.rados.0.smithi160.stdout:663: finishing write tid 3 to smithi16030472-8 2024-09-21T18:07:48.068 INFO:tasks.rados.rados.0.smithi160.stdout:664: finishing write tid 1 to smithi16030472-484 2024-09-21T18:07:48.068 INFO:tasks.rados.rados.0.smithi160.stdout:667: expect (ObjNum 195 snap 0 seq_num 195) 2024-09-21T18:07:48.079 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2859033 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 34.907852746000003, 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:07:48.122 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:07:48.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:07:48.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:07:48.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:07:48.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:07:48.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.363915197000001, 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:48.129 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:48.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.134 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.202991733000001, 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.135 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:07:48.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:07:48.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.202317138000002, 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.140 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:07:48.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.202290600000001, 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.144 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.148 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.192456858, 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.149 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.152 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.180045069999998, 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.153 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:07:48.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.157 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.159 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.171154999999999, 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.160 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:07:48.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:07:48.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:07:48.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:07:48.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.170342983000001, 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.167 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.172 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.170234771000001, 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:48.173 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:07:48.174 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:07:48.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:07:48.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.069963625, 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:07:48.179 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.180 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:07:48.181 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.182 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.069067365999999, 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:07:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.068585753000001, 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.068133327999998, 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:07:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.068083002000002, 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:07:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:07:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 16.942350222000002, 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:07:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 16.938072165000001, 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:07:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:07:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:07:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:07:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:07:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:07:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 16.937701924999999, 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:07:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:07:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 16.937632194999999, 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:07:48.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:07:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:07:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 16.346055204999999, 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:48.235 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:48.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:48.238 INFO:tasks.rados.rados.0.smithi160.stdout:663: finishing write tid 4 to smithi16030472-8 2024-09-21T18:07:48.238 INFO:tasks.rados.rados.0.smithi160.stdout:663: finishing write tid 5 to smithi16030472-8 2024-09-21T18:07:48.238 INFO:tasks.rados.rados.0.smithi160.stdout:663: oid 8 updating version 0 to 432 2024-09-21T18:07:48.238 INFO:tasks.rados.rados.0.smithi160.stdout:663: oid 8 updating version 432 to 433 2024-09-21T18:07:48.238 INFO:tasks.rados.rados.0.smithi160.stdout:663: oid 8 updating version 433 to 435 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:663: oid 8 version 435 is already newer than 434 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 8 v 435 (ObjNum 552 snap 0 seq_num 552) dirty exists 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:663: left oid 8 (ObjNum 552 snap 0 seq_num 552) 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: finishing write tid 2 to smithi16030472-484 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: finishing write tid 3 to smithi16030472-484 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: finishing write tid 4 to smithi16030472-484 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: finishing write tid 5 to smithi16030472-484 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: oid 484 updating version 0 to 348 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: oid 484 updating version 348 to 349 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: oid 484 updating version 349 to 351 2024-09-21T18:07:48.239 INFO:tasks.rados.rados.0.smithi160.stdout:664: oid 484 version 351 is already newer than 350 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 351 (ObjNum 553 snap 0 seq_num 553) dirty exists 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:664: left oid 484 (ObjNum 553 snap 0 seq_num 553) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:658: done (9 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:659: done (8 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:660: done (7 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:663: done (6 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:664: done (5 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:665: done (4 left) 2024-09-21T18:07:48.240 INFO:tasks.rados.rados.0.smithi160.stdout:666: done (3 left) 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:667: done (2 left) 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:668: done (1 left) 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:669: write oid 114 current snap is 0 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:669: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:669: writing smithi16030472-114 from 527885 to 1184849 tid 1 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:669: writing smithi16030472-114 from 1927271 to 2411123 tid 2 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:669: writing smithi16030472-114 from 3191408 to 3327885 tid 3 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:670: copy_from oid 294 from oid 327 current snap is 0 2024-09-21T18:07:48.241 INFO:tasks.rados.rados.0.smithi160.stdout:661: got (0) Success 2024-09-21T18:07:48.242 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 0 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-21T18:07:48.242 INFO:tasks.rados.rados.0.smithi160.stdout:661: done (2 left) 2024-09-21T18:07:48.242 INFO:tasks.rados.rados.0.smithi160.stdout:671: cache_evict oid 75 snap -1 2024-09-21T18:07:48.242 INFO:tasks.rados.rados.0.smithi160.stdout:672: delete oid 407 current snap is 0 2024-09-21T18:07:48.242 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:48.233+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:48.338 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:48.334+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:48.374 INFO:tasks.rados.rados.0.smithi160.stdout:671: got (0) Success 2024-09-21T18:07:48.375 INFO:tasks.rados.rados.0.smithi160.stdout:671: done (3 left) 2024-09-21T18:07:48.376 INFO:tasks.rados.rados.0.smithi160.stdout:672: done (2 left) 2024-09-21T18:07:48.376 INFO:tasks.rados.rados.0.smithi160.stdout:673: delete oid 467 current snap is 0 2024-09-21T18:07:48.412 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:48.412 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:48.412 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:48.412 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:48.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:48.433 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:07:48.435 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:48.430+0000 7f66f2420640 -1 osd.7 57 handle_osd_map injecting map crc failure 2024-09-21T18:07:48.438 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:48.434+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:48.471 INFO:tasks.rados.rados.0.smithi160.stdout:669: finishing write tid 1 to smithi16030472-114 2024-09-21T18:07:48.539 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:48.536+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:48.551 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:48.552 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:48.552 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:48.552 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:48.552 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:48.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:48.585 INFO:tasks.rados.rados.0.smithi160.stdout:669: finishing write tid 2 to smithi16030472-114 2024-09-21T18:07:48.585 INFO:tasks.rados.rados.0.smithi160.stdout:669: finishing write tid 3 to smithi16030472-114 2024-09-21T18:07:48.587 INFO:tasks.rados.rados.0.smithi160.stdout:669: finishing write tid 4 to smithi16030472-114 2024-09-21T18:07:48.596 INFO:tasks.rados.rados.0.smithi160.stdout:673: done (2 left) 2024-09-21T18:07:48.596 INFO:tasks.rados.rados.0.smithi160.stdout:674: cache_try_flush oid 409 snap -1 2024-09-21T18:07:48.596 INFO:tasks.rados.rados.0.smithi160.stdout:675: write (excl) oid 441 current snap is 0 2024-09-21T18:07:48.596 INFO:tasks.rados.rados.0.smithi160.stdout:675: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-09-21T18:07:48.613 INFO:tasks.rados.rados.0.smithi160.stdout:675: writing smithi16030472-441 from 578306 to 1061017 tid 1 2024-09-21T18:07:48.630 INFO:tasks.rados.rados.0.smithi160.stdout:675: writing smithi16030472-441 from 1586364 to 2217543 tid 2 2024-09-21T18:07:48.640 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:48.637+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:48.645 INFO:tasks.rados.rados.0.smithi160.stdout:675: writing smithi16030472-441 from 2757324 to 3242166 tid 3 2024-09-21T18:07:48.645 INFO:tasks.rados.rados.0.smithi160.stdout:676: read oid 231 snap -1 2024-09-21T18:07:48.645 INFO:tasks.rados.rados.0.smithi160.stdout:676: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-21T18:07:48.645 INFO:tasks.rados.rados.0.smithi160.stdout:677: cache_flush oid 159 snap -1 2024-09-21T18:07:48.645 INFO:tasks.rados.rados.0.smithi160.stdout:678: write (excl) oid 311 current snap is 0 2024-09-21T18:07:48.646 INFO:tasks.rados.rados.0.smithi160.stdout:678: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-21T18:07:48.658 INFO:tasks.rados.rados.0.smithi160.stdout:678: writing smithi16030472-311 from 595113 to 1153073 tid 1 2024-09-21T18:07:48.668 INFO:tasks.rados.rados.0.smithi160.stdout:678: writing smithi16030472-311 from 1644846 to 2136036 tid 2 2024-09-21T18:07:48.679 INFO:tasks.rados.rados.0.smithi160.stdout:678: writing smithi16030472-311 from 2784747 to 3395113 tid 3 2024-09-21T18:07:48.679 INFO:tasks.rados.rados.0.smithi160.stdout:679: copy_from oid 385 from oid 88 current snap is 0 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:670: finishing copy_from to smithi16030472-294 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 229 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:670: finishing copy_from racing read to smithi16030472-294 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:675: finishing write tid 1 to smithi16030472-441 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:675: finishing write tid 2 to smithi16030472-441 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:670: done (7 left) 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:680: read oid 69 snap -1 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:680: expect deleted 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:681: cache_evict oid 328 snap -1 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:682: read oid 236 snap -1 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:682: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:683: read oid 109 snap -1 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:683: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:684: write (excl) oid 134 current snap is 0 2024-09-21T18:07:48.680 INFO:tasks.rados.rados.0.smithi160.stdout:684: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-21T18:07:48.689 INFO:tasks.rados.rados.0.smithi160.stdout:684: writing smithi16030472-134 from 611920 to 1245129 tid 1 2024-09-21T18:07:48.699 INFO:tasks.rados.rados.0.smithi160.stdout:684: writing smithi16030472-134 from 1703328 to 2338176 tid 2 2024-09-21T18:07:48.705 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:48.698+0000 7f3ce267c640 -1 osd.6 57 handle_osd_map injecting map crc failure 2024-09-21T18:07:48.708 INFO:tasks.rados.rados.0.smithi160.stdout:684: writing smithi16030472-134 from 2812170 to 3319556 tid 3 2024-09-21T18:07:48.709 INFO:tasks.rados.rados.0.smithi160.stdout:685: read oid 141 snap -1 2024-09-21T18:07:48.709 INFO:tasks.rados.rados.0.smithi160.stdout:685: expect (ObjNum 140 snap 0 seq_num 140) 2024-09-21T18:07:48.709 INFO:tasks.rados.rados.0.smithi160.stdout:686: delete oid 232 current snap is 0 2024-09-21T18:07:48.709 INFO:tasks.rados.rados.0.smithi160.stdout:681: got (0) Success 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: finishing write tid 5 to smithi16030472-114 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: oid 114 updating version 0 to 418 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: oid 114 version 418 is already newer than 416 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: oid 114 updating version 418 to 419 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: oid 114 version 419 is already newer than 417 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 419 (ObjNum 555 snap 0 seq_num 555) dirty exists 2024-09-21T18:07:48.710 INFO:tasks.rados.rados.0.smithi160.stdout:669: left oid 114 (ObjNum 555 snap 0 seq_num 555) 2024-09-21T18:07:48.741 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:48.737+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:48.742 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:48.742 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:48.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:48.742 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:07:48.742 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:48.751 INFO:tasks.rados.rados.0.smithi160.stdout:677: got (2) No such file or directory 2024-09-21T18:07:48.756 INFO:tasks.rados.rados.0.smithi160.stdout:683: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:07:48.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:48.842 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:48.838+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:48.872 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:48.877 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:48.877 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:48.877 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:07:48.877 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:48.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:48.913 INFO:tasks.rados.rados.0.smithi160.stdout:674: got (0) Success 2024-09-21T18:07:48.913 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 0 (ObjNum 408 snap 0 seq_num 408) dirty exists 2024-09-21T18:07:48.913 INFO:tasks.rados.rados.0.smithi160.stdout:676: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-21T18:07:48.913 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:48.943 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:48.938+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:48.955 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 488435 2024-09-21T18:07:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:49.021 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:07:49.021 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:07:49.021 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:07:49.021 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:684: finishing write tid 1 to smithi16030472-134 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:684: finishing write tid 2 to smithi16030472-134 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:675: finishing write tid 3 to smithi16030472-441 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:684: finishing write tid 3 to smithi16030472-134 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:684: finishing write tid 4 to smithi16030472-134 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:669: done (13 left) 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:674: done (12 left) 2024-09-21T18:07:49.035 INFO:tasks.rados.rados.0.smithi160.stdout:676: done (11 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:677: done (10 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:680: done (9 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:681: done (8 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:683: done (7 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:686: done (6 left) 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:687: copy_from oid 334 from oid 173 current snap is 0 2024-09-21T18:07:49.036 INFO:tasks.rados.rados.0.smithi160.stdout:688: delete oid 443 current snap is 0 2024-09-21T18:07:49.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:49.043 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:49.038+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:49.053 INFO:tasks.rados.rados.0.smithi160.stdout:675: finishing write tid 4 to smithi16030472-441 2024-09-21T18:07:49.053 INFO:tasks.rados.rados.0.smithi160.stdout:675: finishing write tid 5 to smithi16030472-441 2024-09-21T18:07:49.053 INFO:tasks.rados.rados.0.smithi160.stdout:675: oid 441 updating version 0 to 356 2024-09-21T18:07:49.053 INFO:tasks.rados.rados.0.smithi160.stdout:675: oid 441 version 356 is already newer than 355 2024-09-21T18:07:49.053 INFO:tasks.rados.rados.0.smithi160.stdout:675: oid 441 updating version 356 to 357 2024-09-21T18:07:49.054 INFO:tasks.rados.rados.0.smithi160.stdout:675: oid 441 updating version 357 to 358 2024-09-21T18:07:49.054 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 358 (ObjNum 558 snap 0 seq_num 558) dirty exists 2024-09-21T18:07:49.054 INFO:tasks.rados.rados.0.smithi160.stdout:675: left oid 441 (ObjNum 558 snap 0 seq_num 558) 2024-09-21T18:07:49.061 INFO:tasks.rados.rados.0.smithi160.stdout:678: finishing write tid 1 to smithi16030472-311 2024-09-21T18:07:49.062 INFO:tasks.rados.rados.0.smithi160.stdout:678: finishing write tid 2 to smithi16030472-311 2024-09-21T18:07:49.063 INFO:tasks.rados.rados.0.smithi160.stdout:678: finishing write tid 3 to smithi16030472-311 2024-09-21T18:07:49.065 INFO:tasks.rados.rados.0.smithi160.stdout:685: expect (ObjNum 140 snap 0 seq_num 140) 2024-09-21T18:07:49.085 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:49.142 INFO:tasks.rados.rados.0.smithi160.stdout:684: finishing write tid 5 to smithi16030472-134 2024-09-21T18:07:49.142 INFO:tasks.rados.rados.0.smithi160.stdout:684: oid 134 updating version 0 to 340 2024-09-21T18:07:49.142 INFO:tasks.rados.rados.0.smithi160.stdout:684: oid 134 version 340 is already newer than 339 2024-09-21T18:07:49.142 INFO:tasks.rados.rados.0.smithi160.stdout:684: oid 134 updating version 340 to 341 2024-09-21T18:07:49.143 INFO:tasks.rados.rados.0.smithi160.stdout:684: oid 134 updating version 341 to 342 2024-09-21T18:07:49.143 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 134 v 342 (ObjNum 560 snap 0 seq_num 560) dirty exists 2024-09-21T18:07:49.143 INFO:tasks.rados.rados.0.smithi160.stdout:684: left oid 134 (ObjNum 560 snap 0 seq_num 560) 2024-09-21T18:07:49.144 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:49.141+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:49.164 INFO:tasks.rados.rados.0.smithi160.stdout:678: finishing write tid 4 to smithi16030472-311 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 63.270738530000003, 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:49.168 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:07:49.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:07:49.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:49.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 62.185346140999997, 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:07:49.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:07:49.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:07:49.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:07:49.177 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:07:49.185 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:49.246 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:49.242+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: finishing write tid 5 to smithi16030472-311 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: oid 311 updating version 0 to 439 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: oid 311 version 439 is already newer than 436 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: oid 311 version 439 is already newer than 437 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: oid 311 version 439 is already newer than 438 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 311 v 439 (ObjNum 559 snap 0 seq_num 559) dirty exists 2024-09-21T18:07:49.265 INFO:tasks.rados.rados.0.smithi160.stdout:678: left oid 311 (ObjNum 559 snap 0 seq_num 559) 2024-09-21T18:07:49.277 INFO:tasks.rados.rados.0.smithi160.stdout:679: finishing copy_from racing read to smithi16030472-385 2024-09-21T18:07:49.278 INFO:tasks.rados.rados.0.smithi160.stdout:682: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:07:49.317 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:2671 3.0 3:0bfeb78a:::smithi16030472-334:head [copy-from ver 107 in=56b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e58)", 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:49.040422+0000", 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.284316403, 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.28436869199999998, 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:49.326 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040422+0000", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040422+0000", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040426+0000", 2024-09-21T18:07:49.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2529999999999998e-06 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040446+0000", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0040000000000001e-05 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040479+0000", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2319e-05 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.040497+0000", 2024-09-21T18:07:49.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8110000000000001e-05 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.140773+0000", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100276253 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.140923+0000", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000149784 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.280159+0000", 2024-09-21T18:07:49.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13923612199999999 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:49.287882+0000", 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0077236220000000003 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-09-21T18:07:49.330 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:49.341 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:49.382 INFO:tasks.rados.rados.0.smithi160.stdout:679: finishing copy_from to smithi16030472-385 2024-09-21T18:07:49.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 52 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-09-21T18:07:49.382 INFO:tasks.rados.rados.0.smithi160.stdout:687: finishing copy_from racing read to smithi16030472-334 2024-09-21T18:07:49.387 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:49.385+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:49.448 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:49.443+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:49.467 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:49.467 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:49.467 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:49.467 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:49.467 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:49.474 INFO:tasks.rados.rados.0.smithi160.stdout:687: finishing copy_from to smithi16030472-334 2024-09-21T18:07:49.474 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 334 v 107 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-09-21T18:07:49.482 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:49.549 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:49.542+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:675: done (7 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:678: done (6 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:679: done (5 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:682: done (4 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:684: done (3 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:685: done (2 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:687: done (1 left) 2024-09-21T18:07:49.576 INFO:tasks.rados.rados.0.smithi160.stdout:688: done (0 left) 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:689: cache_flush oid 412 snap -1 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:690: read oid 344 snap -1 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:690: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:691: cache_try_flush oid 314 snap -1 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:692: copy_from oid 484 from oid 214 current snap is 0 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:693: cache_evict oid 225 snap -1 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:694: delete oid 332 current snap is 0 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:691: got (0) Success 2024-09-21T18:07:49.577 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 314 v 0 (ObjNum 313 snap 0 seq_num 313) dirty exists 2024-09-21T18:07:49.651 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:49.646+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:49.654 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612900989, 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:07:49.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:07:49.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:49.660 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612661365000001, 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.665 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612625957999999, 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:07:49.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:07:49.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612592785, 2024-09-21T18:07:49.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:07:49.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612563224999999, 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:07:49.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:07:49.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:07:49.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612533569, 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:49.685 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.612047313000001, 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:07:49.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:07:49.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.61201372, 2024-09-21T18:07:49.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:07:49.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:07:49.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.511796027999999, 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:07:49.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.511197123999999, 2024-09-21T18:07:49.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:07:49.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:07:49.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:07:49.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.510482506999999, 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.707 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:07:49.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:07:49.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.714 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.501229774999999, 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:07:49.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.719 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.500965074, 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:07:49.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:07:49.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:07:49.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.726 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.500870384999999, 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.731 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.500829095, 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:07:49.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.737 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.489559192000002, 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:07:49.743 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.480442959000001, 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.744 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:07:49.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.750 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.479379592000001, 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:07:49.755 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.479339282000002, 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:07:49.756 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:07:49.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:07:49.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.763 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 17.716701440000001, 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:49.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:07:49.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:07:49.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:07:49.769 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.770 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.770 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.770 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:49.770 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:49.770 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:49.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:49.771 INFO:tasks.rados.rados.0.smithi160.stdout:693: got (16) Device or resource busy 2024-09-21T18:07:49.772 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:49.773 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:49.766+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:49.785 INFO:tasks.rados.rados.0.smithi160.stdout:690: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-21T18:07:49.785 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:49.857 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1924521 2024-09-21T18:07:49.873 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:49.867+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:49.898 INFO:tasks.rados.rados.0.smithi160.stdout:689: got (0) Success 2024-09-21T18:07:49.898 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 0 (ObjNum 409 snap 0 seq_num 409) dirty exists 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:692: finishing copy_from racing read to smithi16030472-484 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:692: finishing copy_from to smithi16030472-484 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 171 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:689: done (5 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:690: done (4 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:691: done (3 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:692: done (2 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:693: done (1 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:694: done (0 left) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:695: read oid 377 snap -1 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:695: expect (ObjNum 550 snap 0 seq_num 550) 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:696: cache_flush oid 238 snap -1 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:697: read oid 140 snap -1 2024-09-21T18:07:49.899 INFO:tasks.rados.rados.0.smithi160.stdout:697: expect (ObjNum 139 snap 0 seq_num 139) 2024-09-21T18:07:49.900 INFO:tasks.rados.rados.0.smithi160.stdout:698: cache_try_flush oid 29 snap -1 2024-09-21T18:07:49.900 INFO:tasks.rados.rados.0.smithi160.stdout:699: write oid 203 current snap is 0 2024-09-21T18:07:49.900 INFO:tasks.rados.rados.0.smithi160.stdout:699: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-09-21T18:07:49.909 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:49.909 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:49.909 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:49.909 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:49.921 INFO:tasks.rados.rados.0.smithi160.stdout:699: writing smithi16030472-203 from 679148 to 1329706 tid 1 2024-09-21T18:07:49.924 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:49.933 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:49.941 INFO:tasks.rados.rados.0.smithi160.stdout:699: writing smithi16030472-203 from 1937256 to 2579442 tid 2 2024-09-21T18:07:49.952 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:49.952 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:699: writing smithi16030472-203 from 3321862 to 3479148 tid 3 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:700: cache_flush oid 423 snap -1 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:696: got (2) No such file or directory 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:698: got (2) No such file or directory 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:695: expect (ObjNum 550 snap 0 seq_num 550) 2024-09-21T18:07:49.954 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:49.973 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:49.974 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:49.974 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:49.970+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:49.993 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:49.994 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:50.025 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:50.044 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:50.044 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:50.044 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:50.044 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:50.058 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:50.061 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:50.061 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:50.075 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:50.071+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:50.081 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:50.081 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:50.102 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:50.102 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:50.113 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1918802 2024-09-21T18:07:50.119 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:50.119 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:50.134 INFO:tasks.rados.rados.0.smithi160.stdout:697: expect (ObjNum 139 snap 0 seq_num 139) 2024-09-21T18:07:50.137 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:50.155 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.157 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:50.157 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:50.175 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:50.170+0000 7f66fb9f5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 26584) UID: 0 2024-09-21T18:07:50.176 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:50.176 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:50.180 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.186 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.186 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.192 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:50.192 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:50.201 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:50.205 INFO:tasks.rados.rados.0.smithi160.stdout:700: got (0) Success 2024-09-21T18:07:50.205 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 423 v 0 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:699: finishing write tid 1 to smithi16030472-203 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:695: done (5 left) 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:696: done (4 left) 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:697: done (3 left) 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:698: done (2 left) 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:700: done (1 left) 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:701: cache_evict oid 333 snap -1 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:702: cache_flush oid 486 snap -1 2024-09-21T18:07:50.206 INFO:tasks.rados.rados.0.smithi160.stdout:703: delete oid 306 current snap is 0 2024-09-21T18:07:50.207 INFO:tasks.rados.rados.0.smithi160.stdout:701: got (16) Device or resource busy 2024-09-21T18:07:50.207 INFO:tasks.rados.rados.0.smithi160.stdout:702: got (0) Success 2024-09-21T18:07:50.207 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 486 v 0 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:701: done (3 left) 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:702: done (2 left) 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:703: done (1 left) 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:704: cache_flush oid 446 snap -1 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:705: cache_try_flush oid 46 snap -1 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:706: read oid 266 snap -1 2024-09-21T18:07:50.209 INFO:tasks.rados.rados.0.smithi160.stdout:706: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-21T18:07:50.210 INFO:tasks.rados.rados.0.smithi160.stdout:707: write (excl) oid 102 current snap is 0 2024-09-21T18:07:50.210 INFO:tasks.rados.rados.0.smithi160.stdout:707: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-21T18:07:50.211 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:50.211 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:50.216 INFO:tasks.rados.rados.0.smithi160.stdout:707: writing smithi16030472-102 from 712762 to 1113818 tid 1 2024-09-21T18:07:50.222 INFO:tasks.rados.rados.0.smithi160.stdout:707: writing smithi16030472-102 from 1537867 to 2183722 tid 2 2024-09-21T18:07:50.233 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:50.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:50.241 INFO:tasks.rados.rados.0.smithi160.stdout:707: writing smithi16030472-102 from 2860355 to 3512762 tid 3 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:708: cache_evict oid 12 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:709: read oid 255 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:709: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:710: read oid 362 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:710: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:711: read oid 344 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:711: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:712: cache_try_flush oid 271 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:713: read oid 331 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:713: expect (ObjNum 330 snap 0 seq_num 330) 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:714: cache_try_flush oid 196 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:715: cache_evict oid 237 snap -1 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:716: write (excl) oid 308 current snap is 0 2024-09-21T18:07:50.242 INFO:tasks.rados.rados.0.smithi160.stdout:716: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-21T18:07:50.254 INFO:tasks.rados.rados.0.smithi160.stdout:716: writing smithi16030472-308 from 729569 to 1205874 tid 1 2024-09-21T18:07:50.255 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:50.255 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:50.266 INFO:tasks.rados.rados.0.smithi160.stdout:716: writing smithi16030472-308 from 1996349 to 2502215 tid 2 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:716: writing smithi16030472-308 from 3004131 to 3529569 tid 3 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:717: read oid 391 snap -1 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:717: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:718: cache_try_flush oid 69 snap -1 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:716: finishing write tid 1 to smithi16030472-308 2024-09-21T18:07:50.275 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:50.319 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:50.315+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:50.319 INFO:tasks.rados.rados.0.smithi160.stdout:706: expect (ObjNum 265 snap 0 seq_num 265) 2024-09-21T18:07:50.323 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1728032 2024-09-21T18:07:50.332 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.332 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.337 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.347 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:50.364 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.390 INFO:tasks.rados.rados.0.smithi160.stdout:706: done (15 left) 2024-09-21T18:07:50.390 INFO:tasks.rados.rados.0.smithi160.stdout:719: write oid 71 current snap is 0 2024-09-21T18:07:50.391 INFO:tasks.rados.rados.0.smithi160.stdout:719: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-21T18:07:50.391 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:50.389+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:50.401 INFO:tasks.rados.rados.0.smithi160.stdout:719: writing smithi16030472-71 from 746376 to 1297930 tid 1 2024-09-21T18:07:50.412 INFO:tasks.rados.rados.0.smithi160.stdout:719: writing smithi16030472-71 from 2054831 to 2704355 tid 2 2024-09-21T18:07:50.419 INFO:tasks.rados.rados.0.smithi160.stdout:719: writing smithi16030472-71 from 3315201 to 3546376 tid 3 2024-09-21T18:07:50.419 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:708: got (16) Device or resource busy 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:699: finishing write tid 2 to smithi16030472-203 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:699: finishing write tid 3 to smithi16030472-203 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:699: finishing write tid 4 to smithi16030472-203 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:699: finishing write tid 5 to smithi16030472-203 2024-09-21T18:07:50.422 INFO:tasks.rados.rados.0.smithi160.stdout:699: oid 203 updating version 0 to 423 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:699: oid 203 updating version 423 to 425 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:699: oid 203 updating version 425 to 426 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:699: oid 203 version 426 is already newer than 424 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 426 (ObjNum 564 snap 0 seq_num 564) dirty exists 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:699: left oid 203 (ObjNum 564 snap 0 seq_num 564) 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:699: done (15 left) 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:708: done (14 left) 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:720: cache_evict oid 367 snap -1 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:714: got (2) No such file or directory 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:714: done (14 left) 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:721: copy_from oid 318 from oid 432 current snap is 0 2024-09-21T18:07:50.423 INFO:tasks.rados.rados.0.smithi160.stdout:722: delete oid 98 current snap is 0 2024-09-21T18:07:50.424 INFO:tasks.rados.rados.0.smithi160.stdout:720: got (0) Success 2024-09-21T18:07:50.450 INFO:tasks.rados.rados.0.smithi160.stdout:707: finishing write tid 1 to smithi16030472-102 2024-09-21T18:07:50.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:07:50.463 INFO:tasks.rados.rados.0.smithi160.stdout:707: finishing write tid 2 to smithi16030472-102 2024-09-21T18:07:50.463 INFO:tasks.rados.rados.0.smithi160.stdout:707: finishing write tid 3 to smithi16030472-102 2024-09-21T18:07:50.477 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.477 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.491 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:50.503 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:50.499+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:50.530 INFO:tasks.rados.rados.0.smithi160.stdout:719: finishing write tid 1 to smithi16030472-71 2024-09-21T18:07:50.533 INFO:tasks.rados.rados.0.smithi160.stdout:711: expect (ObjNum 343 snap 0 seq_num 343) 2024-09-21T18:07:50.533 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.578 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:50.571+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:50.662 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.663 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.677 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:50.678 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:50.673+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:712: got (2) No such file or directory 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:715: got (16) Device or resource busy 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:716: finishing write tid 2 to smithi16030472-308 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:716: finishing write tid 3 to smithi16030472-308 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:707: finishing write tid 4 to smithi16030472-102 2024-09-21T18:07:50.703 INFO:tasks.rados.rados.0.smithi160.stdout:707: finishing write tid 5 to smithi16030472-102 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:707: oid 102 updating version 0 to 428 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:707: oid 102 updating version 428 to 429 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:707: oid 102 version 429 is already newer than 427 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:707: oid 102 updating version 429 to 430 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 102 v 430 (ObjNum 566 snap 0 seq_num 566) dirty exists 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:707: left oid 102 (ObjNum 566 snap 0 seq_num 566) 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:710: expect (ObjNum 361 snap 0 seq_num 361) 2024-09-21T18:07:50.704 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.770 INFO:tasks.rados.rados.0.smithi160.stdout:719: finishing write tid 2 to smithi16030472-71 2024-09-21T18:07:50.770 INFO:tasks.rados.rados.0.smithi160.stdout:719: finishing write tid 3 to smithi16030472-71 2024-09-21T18:07:50.770 INFO:tasks.rados.rados.0.smithi160.stdout:719: finishing write tid 4 to smithi16030472-71 2024-09-21T18:07:50.770 INFO:tasks.rados.rados.0.smithi160.stdout:719: finishing write tid 5 to smithi16030472-71 2024-09-21T18:07:50.770 INFO:tasks.rados.rados.0.smithi160.stdout:719: oid 71 updating version 0 to 336 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:719: oid 71 version 336 is already newer than 334 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:719: oid 71 version 336 is already newer than 335 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:719: oid 71 updating version 336 to 337 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 71 v 337 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:719: left oid 71 (ObjNum 568 snap 0 seq_num 568) 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: finishing write tid 4 to smithi16030472-308 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: finishing write tid 5 to smithi16030472-308 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: oid 308 updating version 0 to 362 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: oid 308 version 362 is already newer than 359 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: oid 308 version 362 is already newer than 360 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: oid 308 version 362 is already newer than 361 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 362 (ObjNum 567 snap 0 seq_num 567) dirty exists 2024-09-21T18:07:50.771 INFO:tasks.rados.rados.0.smithi160.stdout:716: left oid 308 (ObjNum 567 snap 0 seq_num 567) 2024-09-21T18:07:50.780 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:50.775+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:707: done (15 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:710: done (14 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:711: done (13 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:712: done (12 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:715: done (11 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:716: done (10 left) 2024-09-21T18:07:50.783 INFO:tasks.rados.rados.0.smithi160.stdout:719: done (9 left) 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:720: done (8 left) 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:722: done (7 left) 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:723: copy_from oid 137 from oid 194 current snap is 0 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:724: cache_evict oid 402 snap -1 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:725: read oid 390 snap -1 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:725: expect (ObjNum 389 snap 0 seq_num 389) 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:726: write (excl) oid 487 current snap is 0 2024-09-21T18:07:50.784 INFO:tasks.rados.rados.0.smithi160.stdout:726: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2024-09-21T18:07:50.796 INFO:tasks.rados.rados.0.smithi160.stdout:726: writing smithi16030472-487 from 779990 to 1482042 tid 1 2024-09-21T18:07:50.804 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.804 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.807 INFO:tasks.rados.rados.0.smithi160.stdout:726: writing smithi16030472-487 from 2055442 to 2708635 tid 2 2024-09-21T18:07:50.815 INFO:tasks.rados.rados.0.smithi160.stdout:726: writing smithi16030472-487 from 3253694 to 3579990 tid 3 2024-09-21T18:07:50.815 INFO:tasks.rados.rados.0.smithi160.stdout:727: copy_from oid 92 from oid 193 current snap is 0 2024-09-21T18:07:50.815 INFO:tasks.rados.rados.0.smithi160.stdout:713: expect (ObjNum 330 snap 0 seq_num 330) 2024-09-21T18:07:50.815 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:50.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:50.881 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:50.874+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:50.900 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 394729 2024-09-21T18:07:50.949 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:50.949 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:50.964 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:50.979 INFO:tasks.rados.rados.0.smithi160.stdout:724: got (0) Success 2024-09-21T18:07:50.979 INFO:tasks.rados.rados.0.smithi160.stdout:713: done (11 left) 2024-09-21T18:07:50.979 INFO:tasks.rados.rados.0.smithi160.stdout:724: done (10 left) 2024-09-21T18:07:50.979 INFO:tasks.rados.rados.0.smithi160.stdout:728: write (excl) oid 86 current snap is 0 2024-09-21T18:07:50.979 INFO:tasks.rados.rados.0.smithi160.stdout:728: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2024-09-21T18:07:50.981 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:50.978+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:50.989 INFO:tasks.rados.rados.0.smithi160.stdout:728: writing smithi16030472-86 from 796797 to 1290451 tid 1 2024-09-21T18:07:51.001 INFO:tasks.rados.rados.0.smithi160.stdout:728: writing smithi16030472-86 from 1830277 to 2627128 tid 2 2024-09-21T18:07:51.010 INFO:tasks.rados.rados.0.smithi160.stdout:728: writing smithi16030472-86 from 3397470 to 3596797 tid 3 2024-09-21T18:07:51.010 INFO:tasks.rados.rados.0.smithi160.stdout:729: cache_flush oid 67 snap -1 2024-09-21T18:07:51.011 INFO:tasks.rados.rados.0.smithi160.stdout:730: write (excl) oid 206 current snap is 0 2024-09-21T18:07:51.011 INFO:tasks.rados.rados.0.smithi160.stdout:730: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-21T18:07:51.019 INFO:tasks.rados.rados.0.smithi160.stdout:730: writing smithi16030472-206 from 413604 to 982507 tid 1 2024-09-21T18:07:51.030 INFO:tasks.rados.rados.0.smithi160.stdout:730: writing smithi16030472-206 from 1488759 to 2145621 tid 2 2024-09-21T18:07:51.039 INFO:tasks.rados.rados.0.smithi160.stdout:730: writing smithi16030472-206 from 2624893 to 3216602 tid 3 2024-09-21T18:07:51.039 INFO:tasks.rados.rados.0.smithi160.stdout:731: write (excl) oid 472 current snap is 0 2024-09-21T18:07:51.039 INFO:tasks.rados.rados.0.smithi160.stdout:731: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-09-21T18:07:51.047 INFO:tasks.rados.rados.0.smithi160.stdout:731: writing smithi16030472-472 from 430411 to 1074563 tid 1 2024-09-21T18:07:51.055 INFO:tasks.rados.rados.0.smithi160.stdout:731: writing smithi16030472-472 from 1830888 to 2231408 tid 2 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:731: writing smithi16030472-472 from 2935963 to 3338944 tid 3 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:732: read oid 2 snap -1 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:732: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:733: delete oid 37 current snap is 0 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:729: got (0) Success 2024-09-21T18:07:51.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 67 v 0 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-09-21T18:07:51.064 INFO:tasks.rados.rados.0.smithi160.stdout:730: finishing write tid 1 to smithi16030472-206 2024-09-21T18:07:51.064 INFO:tasks.rados.rados.0.smithi160.stdout:705: got (0) Success 2024-09-21T18:07:51.064 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-21T18:07:51.082 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:51.077+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:51.089 INFO:tasks.rados.rados.0.smithi160.stdout:725: expect (ObjNum 389 snap 0 seq_num 389) 2024-09-21T18:07:51.089 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:51.093 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.093 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:51.180 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 996795 2024-09-21T18:07:51.183 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:51.178+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:51.199 INFO:tasks.rados.rados.0.smithi160.stdout:723: finishing copy_from racing read to smithi16030472-137 2024-09-21T18:07:51.199 INFO:tasks.rados.rados.0.smithi160.stdout:717: expect (ObjNum 390 snap 0 seq_num 390) 2024-09-21T18:07:51.204 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 870430 2024-09-21T18:07:51.242 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.242 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.257 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:704: got (0) Success 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 0 (ObjNum 538 snap 0 seq_num 538) dirty exists 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:718: got (0) Success 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 0 (ObjNum 305 snap 0 seq_num 3836895405) clean dne 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:721: finishing copy_from racing read to smithi16030472-318 2024-09-21T18:07:51.266 INFO:tasks.rados.rados.0.smithi160.stdout:709: expect (ObjNum 254 snap 0 seq_num 254) 2024-09-21T18:07:51.276 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1859449 2024-09-21T18:07:51.285 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:51.279+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:51.285 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1061325 2024-09-21T18:07:51.298 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:51.322 INFO:tasks.rados.rados.0.smithi160.stdout:721: finishing copy_from to smithi16030472-318 2024-09-21T18:07:51.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 318 v 364 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-21T18:07:51.357 INFO:tasks.rados.rados.0.smithi160.stdout:730: finishing write tid 2 to smithi16030472-206 2024-09-21T18:07:51.357 INFO:tasks.rados.rados.0.smithi160.stdout:723: finishing copy_from to smithi16030472-137 2024-09-21T18:07:51.358 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 137 v 383 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:07:51.358 INFO:tasks.rados.rados.0.smithi160.stdout:730: finishing write tid 3 to smithi16030472-206 2024-09-21T18:07:51.358 INFO:tasks.rados.rados.0.smithi160.stdout:730: finishing write tid 4 to smithi16030472-206 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: finishing write tid 5 to smithi16030472-206 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: oid 206 updating version 0 to 442 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: oid 206 updating version 442 to 443 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: oid 206 version 443 is already newer than 440 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: oid 206 version 443 is already newer than 441 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 206 v 443 (ObjNum 572 snap 0 seq_num 572) dirty exists 2024-09-21T18:07:51.359 INFO:tasks.rados.rados.0.smithi160.stdout:730: left oid 206 (ObjNum 572 snap 0 seq_num 572) 2024-09-21T18:07:51.374 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.375 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.386 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:51.381+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:51.388 INFO:tasks.rados.rados.0.smithi160.stdout:728: finishing write tid 1 to smithi16030472-86 2024-09-21T18:07:51.388 INFO:tasks.rados.rados.0.smithi160.stdout:728: finishing write tid 2 to smithi16030472-86 2024-09-21T18:07:51.388 INFO:tasks.rados.rados.0.smithi160.stdout:728: finishing write tid 3 to smithi16030472-86 2024-09-21T18:07:51.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:51.396 INFO:tasks.rados.rados.0.smithi160.stdout:732: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:07:51.399 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1061183 2024-09-21T18:07:51.415 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:07:51.457 INFO:tasks.rados.rados.0.smithi160.stdout:726: finishing write tid 1 to smithi16030472-487 2024-09-21T18:07:51.458 INFO:tasks.rados.rados.0.smithi160.stdout:726: finishing write tid 2 to smithi16030472-487 2024-09-21T18:07:51.458 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:07:51.454+0000 7f66f041c640 -1 osd.7 58 handle_osd_map injecting map crc failure 2024-09-21T18:07:51.459 INFO:tasks.rados.rados.0.smithi160.stdout:726: finishing write tid 3 to smithi16030472-487 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: finishing write tid 4 to smithi16030472-487 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: finishing write tid 5 to smithi16030472-487 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: oid 487 updating version 0 to 430 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: oid 487 version 430 is already newer than 428 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: oid 487 updating version 430 to 431 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: oid 487 version 431 is already newer than 429 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 487 v 431 (ObjNum 570 snap 0 seq_num 570) dirty exists 2024-09-21T18:07:51.460 INFO:tasks.rados.rados.0.smithi160.stdout:726: left oid 487 (ObjNum 570 snap 0 seq_num 570) 2024-09-21T18:07:51.486 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:51.481+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:51.517 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.517 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.532 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:704: done (15 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:705: done (14 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:709: done (13 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:717: done (12 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:718: done (11 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:721: done (10 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:723: done (9 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:725: done (8 left) 2024-09-21T18:07:51.544 INFO:tasks.rados.rados.0.smithi160.stdout:726: done (7 left) 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:729: done (6 left) 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:730: done (5 left) 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:732: done (4 left) 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:733: done (3 left) 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:734: write oid 74 current snap is 0 2024-09-21T18:07:51.545 INFO:tasks.rados.rados.0.smithi160.stdout:734: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-21T18:07:51.551 INFO:tasks.rados.rados.0.smithi160.stdout:734: writing smithi16030472-74 from 464025 to 1258675 tid 1 2024-09-21T18:07:51.558 INFO:tasks.rados.rados.0.smithi160.stdout:734: writing smithi16030472-74 from 1947852 to 2352041 tid 2 2024-09-21T18:07:51.564 INFO:tasks.rados.rados.0.smithi160.stdout:734: writing smithi16030472-74 from 2990809 to 3416334 tid 3 2024-09-21T18:07:51.564 INFO:tasks.rados.rados.0.smithi160.stdout:735: cache_flush oid 464 snap -1 2024-09-21T18:07:51.564 INFO:tasks.rados.rados.0.smithi160.stdout:728: finishing write tid 4 to smithi16030472-86 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: finishing write tid 5 to smithi16030472-86 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: oid 86 updating version 0 to 432 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: oid 86 version 432 is already newer than 431 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: oid 86 updating version 432 to 433 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: oid 86 updating version 433 to 434 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 86 v 434 (ObjNum 571 snap 0 seq_num 571) dirty exists 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: left oid 86 (ObjNum 571 snap 0 seq_num 571) 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:731: finishing write tid 1 to smithi16030472-472 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:728: done (4 left) 2024-09-21T18:07:51.565 INFO:tasks.rados.rados.0.smithi160.stdout:736: delete oid 485 current snap is 0 2024-09-21T18:07:51.566 INFO:tasks.rados.rados.0.smithi160.stdout:735: got (0) Success 2024-09-21T18:07:51.566 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 0 (ObjNum 463 snap 0 seq_num 463) dirty exists 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: finishing write tid 2 to smithi16030472-472 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: finishing write tid 3 to smithi16030472-472 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: finishing write tid 4 to smithi16030472-472 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: finishing write tid 5 to smithi16030472-472 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: oid 472 updating version 0 to 438 2024-09-21T18:07:51.572 INFO:tasks.rados.rados.0.smithi160.stdout:731: oid 472 version 438 is already newer than 435 2024-09-21T18:07:51.573 INFO:tasks.rados.rados.0.smithi160.stdout:731: oid 472 version 438 is already newer than 437 2024-09-21T18:07:51.573 INFO:tasks.rados.rados.0.smithi160.stdout:731: oid 472 version 438 is already newer than 436 2024-09-21T18:07:51.573 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 472 v 438 (ObjNum 573 snap 0 seq_num 573) dirty exists 2024-09-21T18:07:51.573 INFO:tasks.rados.rados.0.smithi160.stdout:731: left oid 472 (ObjNum 573 snap 0 seq_num 573) 2024-09-21T18:07:51.618 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:51.613+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:51.676 INFO:tasks.rados.rados.0.smithi160.stdout:734: finishing write tid 1 to smithi16030472-74 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:727: finishing copy_from racing read to smithi16030472-92 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:731: done (4 left) 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:735: done (3 left) 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:736: done (2 left) 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:737: cache_try_flush oid 300 snap -1 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:738: write (excl) oid 303 current snap is 0 2024-09-21T18:07:51.684 INFO:tasks.rados.rados.0.smithi160.stdout:738: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-21T18:07:51.687 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:51.685+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:51.691 INFO:tasks.rados.rados.0.smithi160.stdout:738: writing smithi16030472-303 from 497639 to 1042787 tid 1 2024-09-21T18:07:51.699 INFO:tasks.rados.rados.0.smithi160.stdout:738: writing smithi16030472-303 from 1548463 to 1956321 tid 2 2024-09-21T18:07:51.701 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.701 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.708 INFO:tasks.rados.rados.0.smithi160.stdout:738: writing smithi16030472-303 from 2529302 to 3093724 tid 3 2024-09-21T18:07:51.712 INFO:tasks.rados.rados.0.smithi160.stdout:738: writing smithi16030472-303 from 3528143 to 3697639 tid 4 2024-09-21T18:07:51.712 INFO:tasks.rados.rados.0.smithi160.stdout:739: write oid 445 current snap is 0 2024-09-21T18:07:51.712 INFO:tasks.rados.rados.0.smithi160.stdout:739: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-21T18:07:51.715 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:51.722 INFO:tasks.rados.rados.0.smithi160.stdout:739: writing smithi16030472-445 from 514446 to 1251196 tid 1 2024-09-21T18:07:51.731 INFO:tasks.rados.rados.0.smithi160.stdout:739: writing smithi16030472-445 from 1723298 to 2391167 tid 2 2024-09-21T18:07:51.741 INFO:tasks.rados.rados.0.smithi160.stdout:739: writing smithi16030472-445 from 3073078 to 3714446 tid 3 2024-09-21T18:07:51.741 INFO:tasks.rados.rados.0.smithi160.stdout:740: write (excl) oid 424 current snap is 0 2024-09-21T18:07:51.741 INFO:tasks.rados.rados.0.smithi160.stdout:740: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-21T18:07:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:740: writing smithi16030472-424 from 531253 to 943252 tid 1 2024-09-21T18:07:51.753 INFO:tasks.rados.rados.0.smithi160.stdout:740: writing smithi16030472-424 from 1381780 to 1793307 tid 2 2024-09-21T18:07:51.760 INFO:tasks.rados.rados.0.smithi160.stdout:740: writing smithi16030472-424 from 2300501 to 2887467 tid 3 2024-09-21T18:07:51.765 INFO:tasks.rados.rados.0.smithi160.stdout:740: writing smithi16030472-424 from 3376974 to 3731253 tid 4 2024-09-21T18:07:51.765 INFO:tasks.rados.rados.0.smithi160.stdout:741: copy_from oid 333 from oid 189 current snap is 0 2024-09-21T18:07:51.765 INFO:tasks.rados.rados.0.smithi160.stdout:742: write (excl) oid 203 current snap is 0 2024-09-21T18:07:51.765 INFO:tasks.rados.rados.0.smithi160.stdout:742: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-21T18:07:51.771 INFO:tasks.rados.rados.0.smithi160.stdout:742: writing smithi16030472-203 from 548060 to 1035308 tid 1 2024-09-21T18:07:51.777 INFO:tasks.rados.rados.0.smithi160.stdout:742: writing smithi16030472-203 from 1440262 to 2111800 tid 2 2024-09-21T18:07:51.786 INFO:tasks.rados.rados.0.smithi160.stdout:742: writing smithi16030472-203 from 2844277 to 3642515 tid 3 2024-09-21T18:07:51.786 INFO:tasks.rados.rados.0.smithi160.stdout:743: read oid 8 snap -1 2024-09-21T18:07:51.786 INFO:tasks.rados.rados.0.smithi160.stdout:743: expect (ObjNum 552 snap 0 seq_num 552) 2024-09-21T18:07:51.786 INFO:tasks.rados.rados.0.smithi160.stdout:744: write (excl) oid 219 current snap is 0 2024-09-21T18:07:51.786 INFO:tasks.rados.rados.0.smithi160.stdout:744: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-21T18:07:51.788 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:51.785+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:51.792 INFO:tasks.rados.rados.0.smithi160.stdout:744: writing smithi16030472-219 from 564867 to 1127364 tid 1 2024-09-21T18:07:51.799 INFO:tasks.rados.rados.0.smithi160.stdout:744: writing smithi16030472-219 from 1898744 to 2313940 tid 2 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:744: writing smithi16030472-219 from 2755347 to 3481210 tid 3 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:745: read oid 388 snap -1 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:745: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:746: cache_flush oid 95 snap -1 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:741: finishing copy_from racing read to smithi16030472-333 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:747: cache_try_flush oid 360 snap -1 2024-09-21T18:07:51.806 INFO:tasks.rados.rados.0.smithi160.stdout:741: finishing copy_from to smithi16030472-333 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 116 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:741: done (12 left) 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:748: cache_evict oid 234 snap -1 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:749: read oid 486 snap -1 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:749: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:750: read oid 18 snap -1 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:750: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout:751: cache_flush oid 422 snap -1 2024-09-21T18:07:51.807 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:51.814 INFO:tasks.rados.rados.0.smithi160.stdout:727: finishing copy_from to smithi16030472-92 2024-09-21T18:07:51.814 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 92 v 126 (ObjNum 192 snap 0 seq_num 192) dirty exists 2024-09-21T18:07:51.814 INFO:tasks.rados.rados.0.smithi160.stdout:727: done (15 left) 2024-09-21T18:07:51.814 INFO:tasks.rados.rados.0.smithi160.stdout:752: copy_from oid 42 from oid 497 current snap is 0 2024-09-21T18:07:51.814 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:51.881 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:51.881 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:51.889 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:51.885+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:51.897 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:07:51.959 INFO:tasks.rados.rados.0.smithi160.stdout:737: got (0) Success 2024-09-21T18:07:51.959 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 300 v 0 (ObjNum 299 snap 0 seq_num 299) dirty exists 2024-09-21T18:07:51.959 INFO:tasks.rados.rados.0.smithi160.stdout:734: finishing write tid 2 to smithi16030472-74 2024-09-21T18:07:51.960 INFO:tasks.rados.rados.0.smithi160.stdout:734: finishing write tid 3 to smithi16030472-74 2024-09-21T18:07:51.960 INFO:tasks.rados.rados.0.smithi160.stdout:734: finishing write tid 4 to smithi16030472-74 2024-09-21T18:07:51.960 INFO:tasks.rados.rados.0.smithi160.stdout:734: finishing write tid 5 to smithi16030472-74 2024-09-21T18:07:51.960 INFO:tasks.rados.rados.0.smithi160.stdout:734: oid 74 updating version 0 to 365 2024-09-21T18:07:51.960 INFO:tasks.rados.rados.0.smithi160.stdout:734: oid 74 updating version 365 to 366 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:734: oid 74 updating version 366 to 367 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:734: oid 74 version 367 is already newer than 364 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 74 v 367 (ObjNum 575 snap 0 seq_num 575) dirty exists 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:734: left oid 74 (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:739: finishing write tid 1 to smithi16030472-445 2024-09-21T18:07:51.961 INFO:tasks.rados.rados.0.smithi160.stdout:739: finishing write tid 2 to smithi16030472-445 2024-09-21T18:07:51.962 INFO:tasks.rados.rados.0.smithi160.stdout:734: done (15 left) 2024-09-21T18:07:51.962 INFO:tasks.rados.rados.0.smithi160.stdout:737: done (14 left) 2024-09-21T18:07:51.962 INFO:tasks.rados.rados.0.smithi160.stdout:753: delete oid 136 current snap is 0 2024-09-21T18:07:51.984 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 1 to smithi16030472-303 2024-09-21T18:07:51.990 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:51.987+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:51.997 INFO:tasks.rados.rados.0.smithi160.stdout:743: expect (ObjNum 552 snap 0 seq_num 552) 2024-09-21T18:07:51.998 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.022 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.022 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:07:52.092 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:52.085+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:52.101 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.163 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.163 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:07:52.193 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:52.189+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:739: finishing write tid 3 to smithi16030472-445 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:747: got (0) Success 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 0 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 2 to smithi16030472-303 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 3 to smithi16030472-303 2024-09-21T18:07:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:751: got (2) No such file or directory 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: finishing write tid 1 to smithi16030472-203 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: finishing write tid 2 to smithi16030472-203 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: finishing write tid 3 to smithi16030472-203 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: finishing write tid 4 to smithi16030472-203 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: finishing write tid 5 to smithi16030472-203 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: oid 203 updating version 0 to 441 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: oid 203 version 441 is already newer than 440 2024-09-21T18:07:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:742: oid 203 updating version 441 to 442 2024-09-21T18:07:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:742: oid 203 version 442 is already newer than 439 2024-09-21T18:07:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 442 (ObjNum 580 snap 0 seq_num 580) dirty exists 2024-09-21T18:07:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:742: left oid 203 (ObjNum 580 snap 0 seq_num 580) 2024-09-21T18:07:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 1 to smithi16030472-424 2024-09-21T18:07:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:744: finishing write tid 1 to smithi16030472-219 2024-09-21T18:07:52.200 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 4 to smithi16030472-303 2024-09-21T18:07:52.228 INFO:tasks.rados.rados.0.smithi160.stdout:739: finishing write tid 4 to smithi16030472-445 2024-09-21T18:07:52.270 INFO:tasks.rados.rados.0.smithi160.stdout:744: finishing write tid 2 to smithi16030472-219 2024-09-21T18:07:52.270 INFO:tasks.rados.rados.0.smithi160.stdout:744: finishing write tid 3 to smithi16030472-219 2024-09-21T18:07:52.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 5 to smithi16030472-303 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: finishing write tid 6 to smithi16030472-303 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: oid 303 updating version 0 to 434 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: oid 303 updating version 434 to 435 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: oid 303 version 435 is already newer than 428 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: oid 303 version 435 is already newer than 433 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: oid 303 updating version 435 to 436 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 303 v 436 (ObjNum 577 snap 0 seq_num 577) dirty exists 2024-09-21T18:07:52.304 INFO:tasks.rados.rados.0.smithi160.stdout:738: left oid 303 (ObjNum 577 snap 0 seq_num 577) 2024-09-21T18:07:52.306 INFO:tasks.rados.rados.0.smithi160.stdout:745: expect (ObjNum 387 snap 0 seq_num 387) 2024-09-21T18:07:52.319 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:52.315+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:52.344 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.344 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.356 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:07:52.357 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.400 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.401 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:52.397+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:752: finishing copy_from racing read to smithi16030472-42 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: finishing write tid 5 to smithi16030472-445 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: oid 445 updating version 0 to 369 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: oid 445 version 369 is already newer than 368 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: oid 445 updating version 369 to 371 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: oid 445 version 371 is already newer than 370 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 371 (ObjNum 578 snap 0 seq_num 578) dirty exists 2024-09-21T18:07:52.430 INFO:tasks.rados.rados.0.smithi160.stdout:739: left oid 445 (ObjNum 578 snap 0 seq_num 578) 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:748: got (0) Success 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:744: finishing write tid 4 to smithi16030472-219 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:744: finishing write tid 5 to smithi16030472-219 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:744: oid 219 updating version 0 to 445 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:744: oid 219 version 445 is already newer than 444 2024-09-21T18:07:52.431 INFO:tasks.rados.rados.0.smithi160.stdout:744: oid 219 version 445 is already newer than 443 2024-09-21T18:07:52.432 INFO:tasks.rados.rados.0.smithi160.stdout:744: oid 219 updating version 445 to 446 2024-09-21T18:07:52.432 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 219 v 446 (ObjNum 581 snap 0 seq_num 581) dirty exists 2024-09-21T18:07:52.432 INFO:tasks.rados.rados.0.smithi160.stdout:744: left oid 219 (ObjNum 581 snap 0 seq_num 581) 2024-09-21T18:07:52.432 INFO:tasks.rados.rados.0.smithi160.stdout:749: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:07:52.433 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1398175 2024-09-21T18:07:52.434 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:52.435 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:52.455 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:52.475 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:52.476 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.476 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.485 INFO:tasks.rados.rados.0.smithi160.stdout:752: finishing copy_from to smithi16030472-42 2024-09-21T18:07:52.485 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 348 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:07:52.485 INFO:tasks.rados.rados.0.smithi160.stdout:738: done (14 left) 2024-09-21T18:07:52.485 INFO:tasks.rados.rados.0.smithi160.stdout:739: done (13 left) 2024-09-21T18:07:52.485 INFO:tasks.rados.rados.0.smithi160.stdout:742: done (12 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:743: done (11 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:744: done (10 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:745: done (9 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:747: done (8 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:748: done (7 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:749: done (6 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:751: done (5 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:752: done (4 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:753: done (3 left) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:754: cache_flush oid 116 snap -1 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 2 to smithi16030472-424 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 3 to smithi16030472-424 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:750: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:07:52.486 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.491 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:07:52.494 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:52.494 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:52.491+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:52.551 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 533766 2024-09-21T18:07:52.578 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:52.578 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:52.589 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 4 to smithi16030472-424 2024-09-21T18:07:52.589 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 5 to smithi16030472-424 2024-09-21T18:07:52.589 INFO:tasks.rados.rados.0.smithi160.stdout:750: done (3 left) 2024-09-21T18:07:52.589 INFO:tasks.rados.rados.0.smithi160.stdout:755: write oid 136 current snap is 0 2024-09-21T18:07:52.589 INFO:tasks.rados.rados.0.smithi160.stdout:755: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-21T18:07:52.595 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:52.591+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:52.596 INFO:tasks.rados.rados.0.smithi160.stdout:755: writing smithi16030472-136 from 598481 to 1311476 tid 1 2024-09-21T18:07:52.598 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:52.593+0000 7f9bb588f640 -1 osd.10 59 handle_osd_map injecting map crc failure 2024-09-21T18:07:52.602 INFO:tasks.rados.rados.0.smithi160.stdout:755: writing smithi16030472-136 from 1899355 to 2318220 tid 2 2024-09-21T18:07:52.609 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:52.610 INFO:tasks.rados.rados.0.smithi160.stdout:755: writing smithi16030472-136 from 3093840 to 3798481 tid 3 2024-09-21T18:07:52.610 INFO:tasks.rados.rados.0.smithi160.stdout:756: read oid 267 snap -1 2024-09-21T18:07:52.610 INFO:tasks.rados.rados.0.smithi160.stdout:756: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:07:52.610 INFO:tasks.rados.rados.0.smithi160.stdout:757: delete oid 128 current snap is 0 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: finishing write tid 6 to smithi16030472-424 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: oid 424 updating version 0 to 375 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: oid 424 version 375 is already newer than 373 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: oid 424 version 375 is already newer than 357 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: oid 424 version 375 is already newer than 372 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: oid 424 version 375 is already newer than 374 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 375 (ObjNum 579 snap 0 seq_num 579) dirty exists 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:740: left oid 424 (ObjNum 579 snap 0 seq_num 579) 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:746: got (0) Success 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 95 v 0 (ObjNum 94 snap 0 seq_num 94) dirty exists 2024-09-21T18:07:52.611 INFO:tasks.rados.rados.0.smithi160.stdout:754: got (2) No such file or directory 2024-09-21T18:07:52.637 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:52.637 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:52.659 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.659 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.665 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:52.665 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:52.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:07:52.689 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:52.689 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:52.715 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:52.716 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:52.717 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:52.713+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:52.739 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:52.739 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:52.766 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:52.766 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:52.791 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:52.791 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:52.796 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:52.796 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:52.797 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:52.793+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:52.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:07:52.818 INFO:tasks.rados.rados.0.smithi160.stdout:756: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:07:52.819 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:52.820 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:52.845 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:52.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:52.845 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2099684 2024-09-21T18:07:52.854 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:52.870 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:52.871 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:52.929 INFO:tasks.rados.rados.0.smithi160.stdout:755: finishing write tid 1 to smithi16030472-136 2024-09-21T18:07:52.929 INFO:tasks.rados.rados.0.smithi160.stdout:755: finishing write tid 2 to smithi16030472-136 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: finishing write tid 3 to smithi16030472-136 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: finishing write tid 4 to smithi16030472-136 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: finishing write tid 5 to smithi16030472-136 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: oid 136 updating version 0 to 441 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: oid 136 version 441 is already newer than 438 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: oid 136 version 441 is already newer than 440 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: oid 136 version 441 is already newer than 439 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 136 v 441 (ObjNum 583 snap 0 seq_num 583) dirty exists 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: left oid 136 (ObjNum 583 snap 0 seq_num 583) 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:740: done (5 left) 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:746: done (4 left) 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:754: done (3 left) 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:755: done (2 left) 2024-09-21T18:07:52.930 INFO:tasks.rados.rados.0.smithi160.stdout:756: done (1 left) 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:757: done (0 left) 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:758: read oid 357 snap -1 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:758: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:759: read oid 212 snap -1 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:759: expect (ObjNum 509 snap 0 seq_num 509) 2024-09-21T18:07:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:760: delete oid 228 current snap is 0 2024-09-21T18:07:52.939 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:52.935+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.977 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:236 3.1 e45/31 3:96516620:::smithi16030472-58:head v 45'30, mlcod=45'24)", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 38.289920960000003, 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.202914602, 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684354+0000", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2959999999999999e-06 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684626+0000", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027191299999999998 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684653+0000", 2024-09-21T18:07:52.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7195999999999998e-05 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684667+0000", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3767e-05 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787798+0000", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103131111 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787831+0000", 2024-09-21T18:07:52.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3229000000000003e-05 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887202+0000", 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099371371 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887265+0000", 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2718999999999994e-05 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:07:52.981 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 34.135726849000001, 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:07:52.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:07:52.984 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.985 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.724084555000001, 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.986 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:52.987 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.988 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:07:52.989 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.721253683, 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.990 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:07:52.991 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:07:52.992 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:07:52.993 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.717280774999999, 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.994 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.995 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.996 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:52.997 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.714163824, 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:52.998 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:07:52.999 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.000 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.001 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.712232820000001, 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.002 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:07:53.003 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:07:53.004 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.709126060999999, 2024-09-21T18:07:53.006 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.007 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.008 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:07:53.009 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.010 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.705968377000001, 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.011 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:07:53.012 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:07:53.013 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:07:53.014 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.703735326, 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.015 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.016 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.017 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:07:53.018 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.701483653, 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:07:53.019 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.020 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.021 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.022 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:07:53.023 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.608906064999999, 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.024 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.025 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:07:53.026 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.027 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 26.832236688999998, 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.028 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:07:53.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:07:53.030 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1098 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'211, mlcod=49'204)", 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 26.725739646000001, 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.135100678, 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.032 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248535+0000", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6830000000000001e-06 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248818+0000", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028246800000000002 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248848+0000", 2024-09-21T18:07:53.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0128999999999999e-05 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248863+0000", 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5302999999999999e-05 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.380608+0000", 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.131744374 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.380649+0000", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1430000000000001e-05 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.383553+0000", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029036890000000001 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.383632+0000", 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9602000000000001e-05 2024-09-21T18:07:53.035 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.822729105000001, 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.036 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.037 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:07:53.038 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.039 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:07:53.040 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.715899126, 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.041 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.042 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:07:53.043 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.044 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.382527573999999, 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.045 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.046 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.047 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.048 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.049 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2367 3.1 e55/31 3:92a56fa6:::smithi16030472-339:head v 55'417, mlcod=55'415)", 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:43.572987+0000", 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.4012838260000002, 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.112913323, 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.572987+0000", 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.050 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.572987+0000", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.572991+0000", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.89e-06 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.574973+0000", 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0019814530000000002 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.051 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.575013+0000", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0531999999999998e-05 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.575033+0000", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9273e-05 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.675334+0000", 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100301748 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.052 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.675372+0000", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8022999999999998e-05 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.685799+0000", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.010426944 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.685901+0000", 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010145999999999999 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2368 3.1 e55/31 3:92a56fa6:::smithi16030472-339:head v 55'418, mlcod=55'415)", 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:43.575075+0000", 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.3991965559999997, 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11088474800000001, 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.575075+0000", 2024-09-21T18:07:53.054 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.575075+0000", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.575076+0000", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5829999999999999e-06 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.576304+0000", 2024-09-21T18:07:53.055 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0012281219999999999 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.576355+0000", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0794999999999997e-05 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.576390+0000", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4635999999999998e-05 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.677967+0000", 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101576952 2024-09-21T18:07:53.056 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.678001+0000", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3958000000000003e-05 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.685909+0000", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0079084600000000008 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:43.685959+0000", 2024-09-21T18:07:53.057 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0241999999999998e-05 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 8.6105361049999996, 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.058 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:53.059 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:53.060 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:07:53.061 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:53.062 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:53.063 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:07:53.064 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:53.057+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:53.136 INFO:tasks.rados.rados.0.smithi160.stdout:758: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:07:53.137 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:53.164 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:53.161+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:53.205 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1164045 2024-09-21T18:07:53.209 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:53.209 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:53.209 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:53.209 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:53.223 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:07:53.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:53.261+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:53.278 INFO:tasks.rados.rados.0.smithi160.stdout:759: expect (ObjNum 509 snap 0 seq_num 509) 2024-09-21T18:07:53.278 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:53.345 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:53.345 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:53.360 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:07:53.366 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:53.359+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:53.377 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2492038 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:758: done (2 left) 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:759: done (1 left) 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:760: done (0 left) 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:761: read oid 145 snap -1 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:761: expect (ObjNum 144 snap 0 seq_num 144) 2024-09-21T18:07:53.378 INFO:tasks.rados.rados.0.smithi160.stdout:762: cache_try_flush oid 165 snap -1 2024-09-21T18:07:53.379 INFO:tasks.rados.rados.0.smithi160.stdout:763: read oid 465 snap -1 2024-09-21T18:07:53.379 INFO:tasks.rados.rados.0.smithi160.stdout:763: expect (ObjNum 464 snap 0 seq_num 464) 2024-09-21T18:07:53.379 INFO:tasks.rados.rados.0.smithi160.stdout:764: cache_try_flush oid 17 snap -1 2024-09-21T18:07:53.379 INFO:tasks.rados.rados.0.smithi160.stdout:765: write (excl) oid 225 current snap is 0 2024-09-21T18:07:53.379 INFO:tasks.rados.rados.0.smithi160.stdout:765: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-09-21T18:07:53.389 INFO:tasks.rados.rados.0.smithi160.stdout:765: writing smithi16030472-225 from 648902 to 1303997 tid 1 2024-09-21T18:07:53.401 INFO:tasks.rados.rados.0.smithi160.stdout:765: writing smithi16030472-225 from 2074801 to 2640993 tid 2 2024-09-21T18:07:53.413 INFO:tasks.rados.rados.0.smithi160.stdout:765: writing smithi16030472-225 from 3176109 to 3848902 tid 3 2024-09-21T18:07:53.413 INFO:tasks.rados.rados.0.smithi160.stdout:766: write oid 299 current snap is 0 2024-09-21T18:07:53.413 INFO:tasks.rados.rados.0.smithi160.stdout:766: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-21T18:07:53.423 INFO:tasks.rados.rados.0.smithi160.stdout:766: writing smithi16030472-299 from 665709 to 1396053 tid 1 2024-09-21T18:07:53.431 INFO:tasks.rados.rados.0.smithi160.stdout:766: writing smithi16030472-299 from 2133283 to 2559486 tid 2 2024-09-21T18:07:53.438 INFO:tasks.rados.rados.0.smithi160.stdout:766: writing smithi16030472-299 from 3203532 to 3713380 tid 3 2024-09-21T18:07:53.438 INFO:tasks.rados.rados.0.smithi160.stdout:767: read oid 416 snap -1 2024-09-21T18:07:53.438 INFO:tasks.rados.rados.0.smithi160.stdout:767: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:07:53.438 INFO:tasks.rados.rados.0.smithi160.stdout:768: delete oid 68 current snap is 0 2024-09-21T18:07:53.467 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:53.461+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:53.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:07:53.485 INFO:tasks.rados.rados.0.smithi160.stdout:763: expect (ObjNum 464 snap 0 seq_num 464) 2024-09-21T18:07:53.509 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:53.509 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:53.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:53.509 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:53.509 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:53.523 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:07:53.568 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:53.563+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:53.605 INFO:tasks.rados.rados.0.smithi160.stdout:763: done (7 left) 2024-09-21T18:07:53.605 INFO:tasks.rados.rados.0.smithi160.stdout:768: done (6 left) 2024-09-21T18:07:53.605 INFO:tasks.rados.rados.0.smithi160.stdout:769: delete oid 185 current snap is 0 2024-09-21T18:07:53.668 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:53.663+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:53.691 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:53.691 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:53.691 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:07:53.691 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:53.705 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T18:07:53.768 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:53.763+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:53.785 INFO:tasks.rados.rados.0.smithi160.stdout:762: got (0) Success 2024-09-21T18:07:53.785 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 165 v 0 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-21T18:07:53.869 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:53.865+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:53.878 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:53.878 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:53.878 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:53.878 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:53.878 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:53.889 INFO:tasks.rados.rados.0.smithi160.stdout:765: finishing write tid 1 to smithi16030472-225 2024-09-21T18:07:53.889 INFO:tasks.rados.rados.0.smithi160.stdout:765: finishing write tid 2 to smithi16030472-225 2024-09-21T18:07:53.892 INFO:tasks.rados.rados.0.smithi160.stdout:765: finishing write tid 3 to smithi16030472-225 2024-09-21T18:07:53.896 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T18:07:53.897 INFO:tasks.rados.rados.0.smithi160.stdout:766: finishing write tid 1 to smithi16030472-299 2024-09-21T18:07:53.900 INFO:tasks.rados.rados.0.smithi160.stdout:767: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:07:53.969 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:53.965+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:765: finishing write tid 4 to smithi16030472-225 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:762: done (6 left) 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:767: done (5 left) 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:769: done (4 left) 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:770: read oid 185 snap -1 2024-09-21T18:07:53.996 INFO:tasks.rados.rados.0.smithi160.stdout:770: expect deleted 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:771: read oid 340 snap -1 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:771: expect (ObjNum 339 snap 0 seq_num 339) 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:772: cache_try_flush oid 360 snap -1 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:773: cache_flush oid 278 snap -1 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: finishing write tid 5 to smithi16030472-225 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: oid 225 updating version 0 to 447 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: oid 225 version 447 is already newer than 445 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: oid 225 version 447 is already newer than 444 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: oid 225 version 447 is already newer than 446 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 447 (ObjNum 586 snap 0 seq_num 586) dirty exists 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: left oid 225 (ObjNum 586 snap 0 seq_num 586) 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:765: done (7 left) 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:774: read oid 90 snap -1 2024-09-21T18:07:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:774: expect (ObjNum 371 snap 0 seq_num 371) 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:775: copy_from oid 307 from oid 449 current snap is 0 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:776: cache_flush oid 22 snap -1 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:777: copy_from oid 467 from oid 377 current snap is 0 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:778: cache_flush oid 25 snap -1 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:779: write (excl) oid 249 current snap is 0 2024-09-21T18:07:53.998 INFO:tasks.rados.rados.0.smithi160.stdout:779: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-09-21T18:07:54.005 INFO:tasks.rados.rados.0.smithi160.stdout:779: writing smithi16030472-249 from 716130 to 1272221 tid 1 2024-09-21T18:07:54.009 INFO:tasks.rados.rados.0.smithi160.stdout:779: writing smithi16030472-249 from 1792376 to 2365906 tid 2 2024-09-21T18:07:54.014 INFO:tasks.rados.rados.0.smithi160.stdout:779: writing smithi16030472-249 from 2769448 to 3229465 tid 3 2024-09-21T18:07:54.018 INFO:tasks.rados.rados.0.smithi160.stdout:779: writing smithi16030472-249 from 3821956 to 3916130 tid 4 2024-09-21T18:07:54.018 INFO:tasks.rados.rados.0.smithi160.stdout:780: write (excl) oid 180 current snap is 0 2024-09-21T18:07:54.018 INFO:tasks.rados.rados.0.smithi160.stdout:780: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-21T18:07:54.025 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:07:54.025 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:54.026 INFO:tasks.rados.rados.0.smithi160.stdout:780: writing smithi16030472-180 from 732937 to 1364277 tid 1 2024-09-21T18:07:54.031 INFO:tasks.rados.rados.0.smithi160.stdout:780: writing smithi16030472-180 from 1850858 to 2284399 tid 2 2024-09-21T18:07:54.038 INFO:tasks.rados.rados.0.smithi160.stdout:780: writing smithi16030472-180 from 2796871 to 3468160 tid 3 2024-09-21T18:07:54.040 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:780: writing smithi16030472-180 from 3888195 to 3932937 tid 4 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:781: copy_from oid 114 from oid 109 current snap is 0 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:782: read oid 270 snap -1 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:782: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:761: expect (ObjNum 144 snap 0 seq_num 144) 2024-09-21T18:07:54.041 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:54.086 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:54.082+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:54.105 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:772: got (0) Success 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 0 (ObjNum 359 snap 0 seq_num 359) dirty exists 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:766: finishing write tid 2 to smithi16030472-299 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:778: got (2) No such file or directory 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 1 to smithi16030472-249 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:761: done (15 left) 2024-09-21T18:07:54.134 INFO:tasks.rados.rados.0.smithi160.stdout:772: done (14 left) 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:778: done (13 left) 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:783: copy_from oid 70 from oid 339 current snap is 0 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:766: finishing write tid 3 to smithi16030472-299 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:784: cache_evict oid 244 snap -1 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:785: write oid 302 current snap is 0 2024-09-21T18:07:54.135 INFO:tasks.rados.rados.0.smithi160.stdout:785: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-21T18:07:54.148 INFO:tasks.rados.rados.0.smithi160.stdout:785: writing smithi16030472-302 from 749744 to 1456333 tid 1 2024-09-21T18:07:54.157 INFO:tasks.rados.rados.0.smithi160.stdout:785: writing smithi16030472-302 from 1909340 to 2486539 tid 2 2024-09-21T18:07:54.164 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:54.164 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:54.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2867 4.1 4:aa88bef2:::smithi16030472-299:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.782429+0000", 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.37889510100000001, 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37893964400000002, 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2867 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782429+0000", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782429+0000", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782429+0000", 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-07 2024-09-21T18:07:54.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782431+0000", 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0590000000000001e-06 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782446+0000", 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5398000000000001e-05 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.782453+0000", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3099999999999997e-06 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.888161+0000", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.105708567 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.888270+0000", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010904900000000001 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.112665+0000", 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22439505500000001 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.112836+0000", 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000170817 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2909 4.1 4.78fa2739 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.133295+0000", 2024-09-21T18:07:54.169 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.028028804000000001, 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028195794999999999, 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2909 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133295+0000", 2024-09-21T18:07:54.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133295+0000", 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133296+0000", 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7899999999999997e-07 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133300+0000", 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9600000000000002e-06 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133310+0000", 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0763e-05 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133318+0000", 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0210000000000002e-06 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.172 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2910 4.1 4.16633e61 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.147456+0000", 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.013867841000000001, 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.014110217, 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2910 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.173 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147456+0000", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147456+0000", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147458+0000", 2024-09-21T18:07:54.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.368e-06 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147669+0000", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210812 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147686+0000", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7244e-05 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.147696+0000", 2024-09-21T18:07:54.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0200999999999999e-05 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2911 4.1 4.16633e61 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.156193+0000", 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0051303340000000003, 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0054441159999999997, 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.176 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2911 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156193+0000", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156193+0000", 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156195+0000", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.513e-06 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156450+0000", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025488399999999998 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156465+0000", 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5179e-05 2024-09-21T18:07:54.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.156474+0000", 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1269999999999992e-06 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2873 4.3 4:f7e9dc57:::smithi16030472-185:head [sparse-read 0~0,omap-get-vals-by-keys in=4b,omap-get-keys in=12b,omap-get-vals in=16b,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995072+0000", 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16625129599999999, 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166653143, 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.179 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2873 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995072+0000", 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995072+0000", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995078+0000", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1440000000000002e-06 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995085+0000", 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3200000000000002e-06 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995098+0000", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3278999999999999e-05 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995107+0000", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7879999999999995e-06 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995298+0000", 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019087600000000001 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995415+0000", 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000117489 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2874 4.3 4:f7e9dc57:::smithi16030472-185:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995142+0000", 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16618163399999999, 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667465300000001, 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.183 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2874 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995142+0000", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995142+0000", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995143+0000", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6199999999999996e-07 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995147+0000", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1160000000000001e-06 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995158+0000", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0488e-05 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995164+0000", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8229999999999997e-06 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995497+0000", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000332293 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995561+0000", 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4392000000000003e-05 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2875 4.3 4:f7e9dc57:::smithi16030472-185:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995174+0000", 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16614959100000001, 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166732871, 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2875 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995174+0000", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995174+0000", 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995175+0000", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5799999999999998e-07 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995178+0000", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.134e-06 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995184+0000", 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3210000000000001e-06 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995190+0000", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4990000000000002e-06 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995643+0000", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00045347200000000001 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995731+0000", 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7658000000000007e-05 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2886 4.3 4:faae7397:::smithi16030472-22:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995236+0000", 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16608726600000001, 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166761415, 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2886 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995236+0000", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995236+0000", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995237+0000", 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.02e-07 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995240+0000", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1599999999999998e-06 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995247+0000", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9580000000000001e-06 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995274+0000", 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6633999999999999e-05 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.095866+0000", 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10059214399999999 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.096045+0000", 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017926500000000001 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2887 4.3 4:e43034e2:::smithi16030472-467:head [copy-from ver 415 in=56b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.194 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995307+0000", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.16601669499999999, 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166783612, 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2887 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995307+0000", 2024-09-21T18:07:54.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995307+0000", 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995308+0000", 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4900000000000005e-07 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995313+0000", 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7489999999999998e-06 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995321+0000", 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2349999999999999e-06 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995327+0000", 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1600000000000003e-06 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.096272+0000", 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100945321 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2888 4.3 4:e43034e2:::smithi16030472-467:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.995336+0000", 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.165987933, 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.166836976, 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.198 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2888 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995336+0000", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995336+0000", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995337+0000", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8199999999999999e-07 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995340+0000", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.089e-06 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995347+0000", 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.5039999999999996e-06 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.995353+0000", 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.434e-06 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.096697+0000", 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101344225 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.201 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2857 4.3 4:fd62100b:::smithi16030472-17:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:53.677652+0000", 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.483671197, 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.48460230100000001, 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.202 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2857 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677652+0000", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677652+0000", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677656+0000", 2024-09-21T18:07:54.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677669+0000", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3274e-05 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677689+0000", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9882e-05 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.677706+0000", 2024-09-21T18:07:54.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6322999999999999e-05 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.678313+0000", 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00060761599999999995 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:53.678509+0000", 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019544699999999999 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2902 4.3 4:f63a4439:::smithi16030472-114:head [copy-from ver 68 in=56b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.039793+0000", 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121530633, 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12255126299999999, 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2902 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039793+0000", 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039793+0000", 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039797+0000", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.216e-06 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039810+0000", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2884999999999999e-05 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039840+0000", 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0158000000000002e-05 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039855+0000", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4616e-05 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.096974+0000", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057119367999999997 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097123+0000", 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000148899 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.211 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2903 4.3 4:f63a4439:::smithi16030472-114:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.039882+0000", 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121441325, 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.1225484, 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.212 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2903 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039882+0000", 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.214 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039882+0000", 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039883+0000", 2024-09-21T18:07:54.215 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4499999999999996e-07 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039886+0000", 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.723e-06 2024-09-21T18:07:54.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039894+0000", 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8259999999999993e-06 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.039900+0000", 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9290000000000003e-06 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097293+0000", 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057393049000000002 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097354+0000", 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1060000000000002e-05 2024-09-21T18:07:54.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1354 4.3 4.16d53733 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.220 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.097296+0000", 2024-09-21T18:07:54.221 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.064027392000000002, 2024-09-21T18:07:54.221 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.065220126000000003, 2024-09-21T18:07:54.221 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.221 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097296+0000", 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097296+0000", 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097300+0000", 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9029999999999997e-06 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097314+0000", 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4233000000000001e-05 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097710+0000", 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039547699999999998 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.097736+0000", 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6596000000000001e-05 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2907 4.1 4.1ad5cf4d (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.224 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.133116+0000", 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.028207871999999998, 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029461838000000001, 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2907 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.225 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133116+0000", 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133116+0000", 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133120+0000", 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.692e-06 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133133+0000", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3081999999999999e-05 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133156+0000", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3527999999999999e-05 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133169+0000", 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2532e-05 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2908 4.1 4.1ad5cf4d (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e60)", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.133196+0000", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.028128067, 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029458858000000001, 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2908 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.228 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133196+0000", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133196+0000", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133197+0000", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0100000000000003e-07 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.229 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133200+0000", 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5159999999999999e-06 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133207+0000", 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4560000000000003e-06 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.133213+0000", 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9859999999999999e-06 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.230 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:54.231 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 16 2024-09-21T18:07:54.231 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:54.231 INFO:tasks.rados.rados.0.smithi160.stdout:785: writing smithi16030472-302 from 3224294 to 3706855 tid 3 2024-09-21T18:07:54.231 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:54.232 INFO:tasks.rados.rados.0.smithi160.stdout:766: finishing write tid 4 to smithi16030472-299 2024-09-21T18:07:54.233 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:07:54.233 INFO:tasks.rados.rados.0.smithi160.stdout:771: expect (ObjNum 339 snap 0 seq_num 339) 2024-09-21T18:07:54.233 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:54.233 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:54.226+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:54.251 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2428850 2024-09-21T18:07:54.277 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2706315 2024-09-21T18:07:54.293 INFO:tasks.rados.rados.0.smithi160.stdout:764: got (0) Success 2024-09-21T18:07:54.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 17 v 0 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:07:54.293 INFO:tasks.rados.rados.0.smithi160.stdout:774: expect (ObjNum 371 snap 0 seq_num 371) 2024-09-21T18:07:54.294 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 542105 2024-09-21T18:07:54.333 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:54.327+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:54.343 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1381528 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:781: finishing copy_from racing read to smithi16030472-114 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: finishing write tid 5 to smithi16030472-299 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: oid 299 updating version 0 to 436 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: oid 299 version 436 is already newer than 435 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: oid 299 updating version 436 to 448 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: oid 299 updating version 448 to 449 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 299 v 449 (ObjNum 587 snap 0 seq_num 587) dirty exists 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:766: left oid 299 (ObjNum 587 snap 0 seq_num 587) 2024-09-21T18:07:54.357 INFO:tasks.rados.rados.0.smithi160.stdout:764: done (15 left) 2024-09-21T18:07:54.358 INFO:tasks.rados.rados.0.smithi160.stdout:766: done (14 left) 2024-09-21T18:07:54.358 INFO:tasks.rados.rados.0.smithi160.stdout:771: done (13 left) 2024-09-21T18:07:54.358 INFO:tasks.rados.rados.0.smithi160.stdout:774: done (12 left) 2024-09-21T18:07:54.358 INFO:tasks.rados.rados.0.smithi160.stdout:786: write oid 237 current snap is 0 2024-09-21T18:07:54.358 INFO:tasks.rados.rados.0.smithi160.stdout:786: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-21T18:07:54.364 INFO:tasks.rados.rados.0.smithi160.stdout:786: writing smithi16030472-237 from 766551 to 1264742 tid 1 2024-09-21T18:07:54.371 INFO:tasks.rados.rados.0.smithi160.stdout:786: writing smithi16030472-237 from 1684175 to 2405032 tid 2 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:786: writing smithi16030472-237 from 2851717 to 3545550 tid 3 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:787: cache_evict oid 119 snap -1 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:788: cache_flush oid 222 snap -1 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:789: read oid 232 snap -1 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:789: expect deleted 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:784: got (2) No such file or directory 2024-09-21T18:07:54.378 INFO:tasks.rados.rados.0.smithi160.stdout:784: done (15 left) 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:790: delete oid 164 current snap is 0 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:773: got (0) Success 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 278 v 0 (ObjNum 522 snap 0 seq_num 522) dirty exists 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:775: finishing copy_from to smithi16030472-307 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 325 (ObjNum 448 snap 0 seq_num 448) dirty exists 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 2 to smithi16030472-249 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 3 to smithi16030472-249 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:785: finishing write tid 1 to smithi16030472-302 2024-09-21T18:07:54.379 INFO:tasks.rados.rados.0.smithi160.stdout:775: finishing copy_from racing read to smithi16030472-307 2024-09-21T18:07:54.383 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:54.383 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:54.383 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:54.383 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:54.383 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:54.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:07:54.434 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:54.427+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:54.466 INFO:tasks.rados.rados.0.smithi160.stdout:786: finishing write tid 1 to smithi16030472-237 2024-09-21T18:07:54.475 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:787: got (16) Device or resource busy 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:788: got (2) No such file or directory 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:781: finishing copy_from to smithi16030472-114 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 68 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:785: finishing write tid 2 to smithi16030472-302 2024-09-21T18:07:54.478 INFO:tasks.rados.rados.0.smithi160.stdout:783: finishing copy_from racing read to smithi16030472-70 2024-09-21T18:07:54.487 INFO:tasks.rados.rados.0.smithi160.stdout:786: finishing write tid 2 to smithi16030472-237 2024-09-21T18:07:54.522 INFO:tasks.rados.rados.0.smithi160.stdout:776: got (0) Success 2024-09-21T18:07:54.522 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:07:54.534 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:54.530+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:54.570 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.421354604000001, 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.571 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.572 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.573 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.574 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:07:54.575 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.576 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:54.577 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.421308295000003, 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.578 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.579 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.580 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.581 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.582 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.419212381999998, 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.583 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.584 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:07:54.585 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.586 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:07:54.587 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:07:54.588 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.589 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.418549267000003, 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.590 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.591 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:07:54.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.418512767999999, 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:07:54.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:07:54.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.284874596000002, 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.602 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:07:54.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:07:54.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:07:54.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.284162805000001, 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:07:54.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 41.284125242000002, 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:07:54.614 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:07:54.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:07:54.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.486739242999999, 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:07:54.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:07:54.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:07:54.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.623 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.486544696999999, 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:07:54.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:07:54.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.628 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.486258309, 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.629 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:07:54.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:07:54.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.486242228999998, 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.633 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.636 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.457314976999999, 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:07:54.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.451095901999999, 2024-09-21T18:07:54.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.643 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:07:54.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:07:54.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.445994306999999, 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:07:54.650 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.651 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:07:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:07:54.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:07:54.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:07:54.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:07:54.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.445660359000001, 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:07:54.659 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.445639880000002, 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:07:54.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:07:54.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.667 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.359311352999999, 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:07:54.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:07:54.671 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.350851166999998, 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.672 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:07:54.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:07:54.675 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.344923462000001, 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:54.676 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:07:54.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:07:54.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.680 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:54.681 INFO:tasks.rados.rados.0.smithi160.stdout:785: finishing write tid 3 to smithi16030472-302 2024-09-21T18:07:54.681 INFO:tasks.rados.rados.0.smithi160.stdout:785: finishing write tid 4 to smithi16030472-302 2024-09-21T18:07:54.682 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:07:54.683 INFO:tasks.rados.rados.0.smithi160.stdout:786: finishing write tid 3 to smithi16030472-237 2024-09-21T18:07:54.683 INFO:tasks.rados.rados.0.smithi160.stdout:786: finishing write tid 4 to smithi16030472-237 2024-09-21T18:07:54.683 INFO:tasks.rados.rados.0.smithi160.stdout:786: finishing write tid 5 to smithi16030472-237 2024-09-21T18:07:54.683 INFO:tasks.rados.rados.0.smithi160.stdout:786: oid 237 updating version 0 to 446 2024-09-21T18:07:54.683 INFO:tasks.rados.rados.0.smithi160.stdout:786: oid 237 version 446 is already newer than 444 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:786: oid 237 version 446 is already newer than 445 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:786: oid 237 version 446 is already newer than 443 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 237 v 446 (ObjNum 593 snap 0 seq_num 593) dirty exists 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:786: left oid 237 (ObjNum 593 snap 0 seq_num 593) 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 4 to smithi16030472-249 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:783: finishing copy_from to smithi16030472-70 2024-09-21T18:07:54.684 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 419 (ObjNum 516 snap 0 seq_num 516) dirty exists 2024-09-21T18:07:54.684 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:54.677+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:54.746 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 5 to smithi16030472-249 2024-09-21T18:07:54.783 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:54.779+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:54.823 INFO:tasks.rados.rados.0.smithi160.stdout:777: finishing copy_from racing read to smithi16030472-467 2024-09-21T18:07:54.826 INFO:tasks.rados.rados.0.smithi160.stdout:777: finishing copy_from to smithi16030472-467 2024-09-21T18:07:54.826 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 467 v 415 (ObjNum 550 snap 0 seq_num 550) dirty exists 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: finishing write tid 6 to smithi16030472-249 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: oid 249 updating version 0 to 359 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: oid 249 updating version 359 to 379 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: oid 249 version 379 is already newer than 378 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: oid 249 updating version 379 to 380 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: oid 249 version 380 is already newer than 377 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 249 v 380 (ObjNum 590 snap 0 seq_num 590) dirty exists 2024-09-21T18:07:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:779: left oid 249 (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:07:54.848 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 1 to smithi16030472-180 2024-09-21T18:07:54.848 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 2 to smithi16030472-180 2024-09-21T18:07:54.848 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 3 to smithi16030472-180 2024-09-21T18:07:54.848 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 4 to smithi16030472-180 2024-09-21T18:07:54.848 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 5 to smithi16030472-180 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: finishing write tid 6 to smithi16030472-180 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: oid 180 updating version 0 to 383 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: oid 180 version 383 is already newer than 382 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: oid 180 updating version 383 to 384 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: oid 180 version 384 is already newer than 381 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: oid 180 updating version 384 to 385 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 385 (ObjNum 591 snap 0 seq_num 591) dirty exists 2024-09-21T18:07:54.849 INFO:tasks.rados.rados.0.smithi160.stdout:780: left oid 180 (ObjNum 591 snap 0 seq_num 591) 2024-09-21T18:07:54.856 INFO:tasks.rados.rados.0.smithi160.stdout:782: expect (ObjNum 269 snap 0 seq_num 269) 2024-09-21T18:07:54.857 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:54.858 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1374 3.3 3:ea88361e:::smithi16030472-164:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e61)", 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.521586+0000", 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.33562304999999998, 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33567810199999998, 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521586+0000", 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521586+0000", 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521587+0000", 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9299999999999997e-07 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521592+0000", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0520000000000001e-06 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521600+0000", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0690000000000002e-06 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521607+0000", 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0439999999999998e-06 2024-09-21T18:07:54.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.831159+0000", 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30955220100000003 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.831326+0000", 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000167384 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2925 3.3 3:ea88361e:::smithi16030472-164:head [stat out=16b,delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e61)", 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.521460+0000", 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.335748834, 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33590598799999999, 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521460+0000", 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521460+0000", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521462+0000", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6449999999999999e-06 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521467+0000", 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0860000000000001e-06 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521479+0000", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255e-05 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521488+0000", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6119999999999999e-06 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.829020+0000", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30753168199999997 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:54.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.829368+0000", 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034785099999999999 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.829711+0000", 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034323399999999999 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.831655+0000", 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0019438369999999999 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.866 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1373 3.3 3:ea88361e:::smithi16030472-164:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e61)", 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:54.521535+0000", 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.33567410800000003, 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33593376400000002, 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521535+0000", 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521535+0000", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521536+0000", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0139999999999999e-06 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521540+0000", 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1420000000000003e-06 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521550+0000", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6080000000000002e-06 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.521557+0000", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.153e-06 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.831004+0000", 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30944743699999999 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:54.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:54.831082+0000", 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7871e-05 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 3 2024-09-21T18:07:54.870 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:54.873 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:07:54.884 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:54.881+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:54.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:07:54.957 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:54.985 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:54.981+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:54.985 INFO:tasks.rados.rados.0.smithi160.stdout:785: finishing write tid 5 to smithi16030472-302 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:785: oid 302 updating version 0 to 439 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:785: oid 302 version 439 is already newer than 437 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:785: oid 302 version 439 is already newer than 438 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:785: oid 302 updating version 439 to 440 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 302 v 440 (ObjNum 592 snap 0 seq_num 592) dirty exists 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:785: left oid 302 (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:770: done (15 left) 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:773: done (14 left) 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:775: done (13 left) 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:776: done (12 left) 2024-09-21T18:07:54.986 INFO:tasks.rados.rados.0.smithi160.stdout:777: done (11 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:779: done (10 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:780: done (9 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:781: done (8 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:782: done (7 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:783: done (6 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:785: done (5 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:786: done (4 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:787: done (3 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:788: done (2 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:789: done (1 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:790: done (0 left) 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:791: cache_flush oid 251 snap -1 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:792: write oid 371 current snap is 0 2024-09-21T18:07:54.987 INFO:tasks.rados.rados.0.smithi160.stdout:792: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-21T18:07:54.992 INFO:tasks.rados.rados.0.smithi160.stdout:792: writing smithi16030472-371 from 400165 to 1048854 tid 1 2024-09-21T18:07:54.999 INFO:tasks.rados.rados.0.smithi160.stdout:792: writing smithi16030472-371 from 1684786 to 2000165 tid 2 2024-09-21T18:07:54.999 INFO:tasks.rados.rados.0.smithi160.stdout:793: cache_flush oid 419 snap -1 2024-09-21T18:07:54.999 INFO:tasks.rados.rados.0.smithi160.stdout:794: delete oid 111 current snap is 0 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:794: done (3 left) 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:795: cache_evict oid 411 snap -1 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:796: read oid 407 snap -1 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:796: expect deleted 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:797: cache_evict oid 245 snap -1 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:798: cache_evict oid 441 snap -1 2024-09-21T18:07:55.003 INFO:tasks.rados.rados.0.smithi160.stdout:799: write (excl) oid 116 current snap is 0 2024-09-21T18:07:55.004 INFO:tasks.rados.rados.0.smithi160.stdout:799: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-21T18:07:55.011 INFO:tasks.rados.rados.0.smithi160.stdout:799: writing smithi16030472-116 from 433779 to 1232966 tid 1 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:799: writing smithi16030472-116 from 1801750 to 2033779 tid 2 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:800: cache_try_flush oid 481 snap -1 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:801: read oid 132 snap -1 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:801: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:802: cache_flush oid 159 snap -1 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:803: cache_evict oid 87 snap -1 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:804: copy_from oid 321 from oid 199 current snap is 0 2024-09-21T18:07:55.017 INFO:tasks.rados.rados.0.smithi160.stdout:805: cache_evict oid 286 snap -1 2024-09-21T18:07:55.018 INFO:tasks.rados.rados.0.smithi160.stdout:806: copy_from oid 20 from oid 93 current snap is 0 2024-09-21T18:07:55.018 INFO:tasks.rados.rados.0.smithi160.stdout:807: cache_try_flush oid 140 snap -1 2024-09-21T18:07:55.018 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:55.020 INFO:tasks.rados.rados.0.smithi160.stdout:803: got (2) No such file or directory 2024-09-21T18:07:55.020 INFO:tasks.rados.rados.0.smithi160.stdout:803: done (15 left) 2024-09-21T18:07:55.021 INFO:tasks.rados.rados.0.smithi160.stdout:808: cache_flush oid 297 snap -1 2024-09-21T18:07:55.021 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:55.051 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:55.049+0000 7f0b6a212640 -1 osd.9 60 handle_osd_map injecting map crc failure 2024-09-21T18:07:55.052 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:07:55.053 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:55.067 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:07:55.076 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:55.076 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:55.085 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:55.079+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:55.100 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:55.101 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:55.104 INFO:tasks.rados.rados.0.smithi160.stdout:795: got (0) Success 2024-09-21T18:07:55.104 INFO:tasks.rados.rados.0.smithi160.stdout:795: done (15 left) 2024-09-21T18:07:55.104 INFO:tasks.rados.rados.0.smithi160.stdout:809: delete oid 44 current snap is 0 2024-09-21T18:07:55.121 INFO:tasks.rados.rados.0.smithi160.stdout:807: got (2) No such file or directory 2024-09-21T18:07:55.121 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:55.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:55.138 INFO:tasks.rados.rados.0.smithi160.stdout:806: finishing copy_from racing read to smithi16030472-20 2024-09-21T18:07:55.143 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:55.143 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:55.164 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:55.164 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:55.186 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:55.181+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:55.186 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:55.187 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:55.204 INFO:tasks.rados.rados.0.smithi160.stdout:798: got (16) Device or resource busy 2024-09-21T18:07:55.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-09-21T18:07:55.209 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T18:07:55.221 INFO:tasks.rados.rados.0.smithi160.stdout:806: finishing copy_from to smithi16030472-20 2024-09-21T18:07:55.221 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 20 v 59 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-21T18:07:55.229 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:55.229 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:55.231 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.554438814999997, 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.232 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.233 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.235 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.554368453000002, 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:55.236 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:07:55.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:07:55.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.542567018, 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.239 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:07:55.240 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:07:55.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.534700635, 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.243 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:07:55.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.534314991000002, 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:07:55.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:07:55.251 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 36.534269541999997, 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.252 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.486371624999997, 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.261 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 35.486263983000001, 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.262 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:07:55.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:07:55.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.265 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 29.910481367999999, 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 29.910429433000001, 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 29.143725045, 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:07:55.277 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:07:55.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:07:55.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.282 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.283 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 28.914621787000002, 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:07:55.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.289 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 28.914193796999999, 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 28.914147228000001, 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.296 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:07:55.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.413982118, 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:07:55.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:07:55.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:07:55.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 24.413478168000001, 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.313 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.405946371999999, 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:07:55.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.405872051999999, 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 17.004068806999999, 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.325 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:07:55.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 16.901718919, 2024-09-21T18:07:55.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:07:55.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.334 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:07:55.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:07:55.335 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:07:55.336 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:55.337 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:796: done (15 left) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:798: done (14 left) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:806: done (13 left) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:807: done (12 left) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:809: done (11 left) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:810: cache_try_flush oid 109 snap -1 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:811: read oid 342 snap -1 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:811: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-21T18:07:55.337 INFO:tasks.rados.rados.0.smithi160.stdout:812: write (excl) oid 135 current snap is 0 2024-09-21T18:07:55.338 INFO:tasks.rados.rados.0.smithi160.stdout:812: seq_num 599 ranges {467393=549685,1802361=265032} 2024-09-21T18:07:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:55.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:812: writing smithi16030472-135 from 467393 to 1017078 tid 1 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:812: writing smithi16030472-135 from 1802361 to 2067393 tid 2 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:813: read oid 490 snap -1 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:813: expect deleted 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:814: write oid 82 current snap is 0 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:814: seq_num 600 ranges {484200=624934,1860843=223357} 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:814: writing smithi16030472-82 from 484200 to 1109134 tid 1 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:814: writing smithi16030472-82 from 1860843 to 2084200 tid 2 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:813: done (15 left) 2024-09-21T18:07:55.340 INFO:tasks.rados.rados.0.smithi160.stdout:815: cache_evict oid 2 snap -1 2024-09-21T18:07:55.341 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:55.341 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:55.329+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:55.438 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:55.429+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:55.456 INFO:tasks.rados.rados.0.smithi160.stdout:811: expect (ObjNum 341 snap 0 seq_num 341) 2024-09-21T18:07:55.474 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:55.474 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:55.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:07:55.523 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:55.531 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:55.531 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:55.539 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:55.537+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:55.549 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:55.549 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:55.570 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:55.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:55.592 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:55.614 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:55.615 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:791: got (2) No such file or directory 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:793: got (2) No such file or directory 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:797: got (16) Device or resource busy 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:792: finishing write tid 1 to smithi16030472-371 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:792: finishing write tid 2 to smithi16030472-371 2024-09-21T18:07:55.616 INFO:tasks.rados.rados.0.smithi160.stdout:792: finishing write tid 3 to smithi16030472-371 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: finishing write tid 4 to smithi16030472-371 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: oid 371 updating version 0 to 453 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: oid 371 updating version 453 to 454 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: oid 371 version 454 is already newer than 452 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 454 (ObjNum 595 snap 0 seq_num 595) dirty exists 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: left oid 371 (ObjNum 595 snap 0 seq_num 595) 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:805: got (0) Success 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:810: got (0) Success 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 0 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:800: got (0) Success 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 0 (ObjNum 480 snap 0 seq_num 480) dirty exists 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:799: finishing write tid 1 to smithi16030472-116 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:799: finishing write tid 2 to smithi16030472-116 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:812: finishing write tid 1 to smithi16030472-135 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:791: done (15 left) 2024-09-21T18:07:55.617 INFO:tasks.rados.rados.0.smithi160.stdout:792: done (14 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:793: done (13 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:797: done (12 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:800: done (11 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:805: done (10 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:810: done (9 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:811: done (8 left) 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:816: cache_try_flush oid 167 snap -1 2024-09-21T18:07:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:801: expect (ObjNum 131 snap 0 seq_num 131) 2024-09-21T18:07:55.619 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:55.619 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:55.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:07:55.640 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:55.637+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:55.663 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2095884 2024-09-21T18:07:55.664 INFO:tasks.rados.rados.0.smithi160.stdout:802: got (2) No such file or directory 2024-09-21T18:07:55.664 INFO:tasks.rados.rados.0.smithi160.stdout:808: got (2) No such file or directory 2024-09-21T18:07:55.664 INFO:tasks.rados.rados.0.smithi160.stdout:812: finishing write tid 2 to smithi16030472-135 2024-09-21T18:07:55.664 INFO:tasks.rados.rados.0.smithi160.stdout:816: got (2) No such file or directory 2024-09-21T18:07:55.664 INFO:tasks.rados.rados.0.smithi160.stdout:815: got (0) Success 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:804: finishing copy_from racing read to smithi16030472-321 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: finishing write tid 3 to smithi16030472-135 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: finishing write tid 4 to smithi16030472-135 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: oid 135 updating version 0 to 456 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: oid 135 updating version 456 to 457 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: oid 135 version 457 is already newer than 455 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 135 v 457 (ObjNum 599 snap 0 seq_num 599) dirty exists 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: left oid 135 (ObjNum 599 snap 0 seq_num 599) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:801: done (8 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:802: done (7 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:808: done (6 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:812: done (5 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:815: done (4 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:816: done (3 left) 2024-09-21T18:07:55.665 INFO:tasks.rados.rados.0.smithi160.stdout:817: read oid 108 snap -1 2024-09-21T18:07:55.666 INFO:tasks.rados.rados.0.smithi160.stdout:817: expect (ObjNum 107 snap 0 seq_num 107) 2024-09-21T18:07:55.666 INFO:tasks.rados.rados.0.smithi160.stdout:818: delete oid 255 current snap is 0 2024-09-21T18:07:55.668 INFO:tasks.rados.rados.0.smithi160.stdout:818: done (4 left) 2024-09-21T18:07:55.668 INFO:tasks.rados.rados.0.smithi160.stdout:819: copy_from oid 33 from oid 251 current snap is 0 2024-09-21T18:07:55.668 INFO:tasks.rados.rados.0.smithi160.stdout:820: write oid 19 current snap is 0 2024-09-21T18:07:55.669 INFO:tasks.rados.rados.0.smithi160.stdout:820: seq_num 602 ranges {517814=491785,1577807=540007} 2024-09-21T18:07:55.685 INFO:tasks.rados.rados.0.smithi160.stdout:820: writing smithi16030472-19 from 517814 to 1009599 tid 1 2024-09-21T18:07:55.703 INFO:tasks.rados.rados.0.smithi160.stdout:820: writing smithi16030472-19 from 1577807 to 2117814 tid 2 2024-09-21T18:07:55.703 INFO:tasks.rados.rados.0.smithi160.stdout:821: cache_flush oid 301 snap -1 2024-09-21T18:07:55.703 INFO:tasks.rados.rados.0.smithi160.stdout:822: write (excl) oid 451 current snap is 0 2024-09-21T18:07:55.703 INFO:tasks.rados.rados.0.smithi160.stdout:822: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-21T18:07:55.725 INFO:tasks.rados.rados.0.smithi160.stdout:822: writing smithi16030472-451 from 534621 to 1101655 tid 1 2024-09-21T18:07:55.733 INFO:tasks.rados.rados.0.smithi160.stdout:822: writing smithi16030472-451 from 1636289 to 2134621 tid 2 2024-09-21T18:07:55.734 INFO:tasks.rados.rados.0.smithi160.stdout:823: cache_try_flush oid 341 snap -1 2024-09-21T18:07:55.734 INFO:tasks.rados.rados.0.smithi160.stdout:824: delete oid 93 current snap is 0 2024-09-21T18:07:55.736 INFO:tasks.rados.rados.0.smithi160.stdout:814: finishing write tid 1 to smithi16030472-82 2024-09-21T18:07:55.737 INFO:tasks.rados.rados.0.smithi160.stdout:804: finishing copy_from to smithi16030472-321 2024-09-21T18:07:55.737 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 321 v 107 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:07:55.737 INFO:tasks.rados.rados.0.smithi160.stdout:799: finishing write tid 3 to smithi16030472-116 2024-09-21T18:07:55.741 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:55.734+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:55.770 INFO:tasks.rados.rados.0.smithi160.stdout:817: expect (ObjNum 107 snap 0 seq_num 107) 2024-09-21T18:07:55.781 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2530252 2024-09-21T18:07:55.803 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:55.804 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:55.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:07:55.863 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:55.859+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:799: finishing write tid 4 to smithi16030472-116 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:799: oid 116 updating version 0 to 443 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:799: oid 116 version 443 is already newer than 442 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:799: oid 116 updating version 443 to 451 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 451 (ObjNum 597 snap 0 seq_num 597) dirty exists 2024-09-21T18:07:55.880 INFO:tasks.rados.rados.0.smithi160.stdout:799: left oid 116 (ObjNum 597 snap 0 seq_num 597) 2024-09-21T18:07:55.938 INFO:tasks.rados.rados.0.smithi160.stdout:799: done (9 left) 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:804: done (8 left) 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:817: done (7 left) 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:824: done (6 left) 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:825: copy_from oid 197 from oid 469 current snap is 0 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:826: read oid 446 snap -1 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:826: expect (ObjNum 538 snap 0 seq_num 538) 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:827: write oid 173 current snap is 0 2024-09-21T18:07:55.939 INFO:tasks.rados.rados.0.smithi160.stdout:827: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-21T18:07:55.939 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:55.940 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:55.942 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:55.938+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:55.951 INFO:tasks.rados.rados.0.smithi160.stdout:827: writing smithi16030472-173 from 568235 to 1285767 tid 1 2024-09-21T18:07:55.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:07:55.961 INFO:tasks.rados.rados.0.smithi160.stdout:827: writing smithi16030472-173 from 1753253 to 2168235 tid 2 2024-09-21T18:07:55.961 INFO:tasks.rados.rados.0.smithi160.stdout:828: cache_flush oid 277 snap -1 2024-09-21T18:07:55.961 INFO:tasks.rados.rados.0.smithi160.stdout:829: read oid 310 snap -1 2024-09-21T18:07:55.961 INFO:tasks.rados.rados.0.smithi160.stdout:829: expect (ObjNum 309 snap 0 seq_num 309) 2024-09-21T18:07:55.961 INFO:tasks.rados.rados.0.smithi160.stdout:830: delete oid 296 current snap is 0 2024-09-21T18:07:56.040 INFO:tasks.rados.rados.0.smithi160.stdout:823: got (0) Success 2024-09-21T18:07:56.040 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 0 (ObjNum 340 snap 0 seq_num 340) dirty exists 2024-09-21T18:07:56.043 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:56.041+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:56.049 INFO:tasks.rados.rados.0.smithi160.stdout:826: expect (ObjNum 538 snap 0 seq_num 538) 2024-09-21T18:07:56.130 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.130 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.144 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:56.141+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:56.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:07:56.147 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 219923 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: finishing write tid 2 to smithi16030472-82 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: finishing write tid 3 to smithi16030472-82 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: finishing write tid 4 to smithi16030472-82 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: oid 82 updating version 0 to 450 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: oid 82 updating version 450 to 452 2024-09-21T18:07:56.196 INFO:tasks.rados.rados.0.smithi160.stdout:814: oid 82 updating version 452 to 453 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 453 (ObjNum 600 snap 0 seq_num 600) dirty exists 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:814: left oid 82 (ObjNum 600 snap 0 seq_num 600) 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:821: got (2) No such file or directory 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:828: got (2) No such file or directory 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:822: finishing write tid 1 to smithi16030472-451 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:822: finishing write tid 2 to smithi16030472-451 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:819: finishing copy_from racing read to smithi16030472-33 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:822: finishing write tid 3 to smithi16030472-451 2024-09-21T18:07:56.197 INFO:tasks.rados.rados.0.smithi160.stdout:829: expect (ObjNum 309 snap 0 seq_num 309) 2024-09-21T18:07:56.216 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 343358 2024-09-21T18:07:56.245 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:56.237+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:56.268 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.268 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:820: finishing write tid 1 to smithi16030472-19 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:822: finishing write tid 4 to smithi16030472-451 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:822: oid 451 updating version 0 to 457 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:822: oid 451 updating version 457 to 459 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:822: oid 451 version 459 is already newer than 458 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 451 v 459 (ObjNum 603 snap 0 seq_num 603) dirty exists 2024-09-21T18:07:56.282 INFO:tasks.rados.rados.0.smithi160.stdout:822: left oid 451 (ObjNum 603 snap 0 seq_num 603) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: finishing write tid 2 to smithi16030472-19 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: finishing write tid 3 to smithi16030472-19 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: finishing write tid 4 to smithi16030472-19 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: oid 19 updating version 0 to 456 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: oid 19 version 456 is already newer than 454 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: oid 19 version 456 is already newer than 455 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 19 v 456 (ObjNum 602 snap 0 seq_num 602) dirty exists 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: left oid 19 (ObjNum 602 snap 0 seq_num 602) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:819: finishing copy_from to smithi16030472-33 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 33 v 181 (ObjNum 250 snap 0 seq_num 250) dirty exists 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:827: finishing write tid 1 to smithi16030472-173 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:827: finishing write tid 2 to smithi16030472-173 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:814: done (11 left) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:819: done (10 left) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:820: done (9 left) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:821: done (8 left) 2024-09-21T18:07:56.283 INFO:tasks.rados.rados.0.smithi160.stdout:822: done (7 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:823: done (6 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:826: done (5 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:828: done (4 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:829: done (3 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:830: done (2 left) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:831: read oid 97 snap -1 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:831: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:832: cache_try_flush oid 413 snap -1 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:833: write oid 215 current snap is 0 2024-09-21T18:07:56.284 INFO:tasks.rados.rados.0.smithi160.stdout:833: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-21T18:07:56.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:07:56.298 INFO:tasks.rados.rados.0.smithi160.stdout:833: writing smithi16030472-215 from 601849 to 1069879 tid 1 2024-09-21T18:07:56.305 INFO:tasks.thrashosds.thrasher:Recovered, killing an osd 2024-09-21T18:07:56.306 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15] 2024-09-21T18:07:56.306 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-21T18:07:56.306 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:07:56.315 INFO:tasks.rados.rados.0.smithi160.stdout:833: writing smithi16030472-215 from 1753864 to 2201849 tid 2 2024-09-21T18:07:56.315 INFO:tasks.rados.rados.0.smithi160.stdout:834: delete oid 185 current snap is 0 2024-09-21T18:07:56.346 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:56.341+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:56.380 INFO:tasks.ceph.osd.7:Stopped 2024-09-21T18:07:56.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 7 2024-09-21T18:07:56.412 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.412 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:07:56.446 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:56.441+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:56.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:07:56.547 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:56.545+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: finishing write tid 3 to smithi16030472-173 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: finishing write tid 4 to smithi16030472-173 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: oid 173 updating version 0 to 463 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: oid 173 version 463 is already newer than 462 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: oid 173 version 463 is already newer than 461 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 463 (ObjNum 605 snap 0 seq_num 605) dirty exists 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:827: left oid 173 (ObjNum 605 snap 0 seq_num 605) 2024-09-21T18:07:56.585 INFO:tasks.rados.rados.0.smithi160.stdout:825: finishing copy_from racing read to smithi16030472-197 2024-09-21T18:07:56.586 INFO:tasks.rados.rados.0.smithi160.stdout:825: finishing copy_from to smithi16030472-197 2024-09-21T18:07:56.586 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 336 (ObjNum 468 snap 0 seq_num 468) dirty exists 2024-09-21T18:07:56.586 INFO:tasks.rados.rados.0.smithi160.stdout:832: got (0) Success 2024-09-21T18:07:56.586 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 0 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-21T18:07:56.588 INFO:tasks.rados.rados.0.smithi160.stdout:831: expect (ObjNum 96 snap 0 seq_num 96) 2024-09-21T18:07:56.602 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.602 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.616 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 515099 2024-09-21T18:07:56.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:07:56.647 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:56.642+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:56.687 INFO:tasks.rados.rados.0.smithi160.stdout:833: finishing write tid 1 to smithi16030472-215 2024-09-21T18:07:56.723 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:07:56.719+0000 7f1072a45640 -1 mon.a@0(leader).osd e62 definitely_dead 0 2024-09-21T18:07:56.724 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7 is already down. 2024-09-21T18:07:56.737 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.737 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.744 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-21T18:07:56.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-09-21T18:07:56.748 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:07:56.745+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:07:56.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:07:56.849 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:56.842+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:56.889 INFO:tasks.rados.rados.0.smithi160.stdout:833: finishing write tid 2 to smithi16030472-215 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:825: done (5 left) 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:827: done (4 left) 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:831: done (3 left) 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:832: done (2 left) 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:834: done (1 left) 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:835: copy_from oid 359 from oid 303 current snap is 0 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:836: cache_evict oid 220 snap -1 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:837: write (excl) oid 445 current snap is 0 2024-09-21T18:07:56.890 INFO:tasks.rados.rados.0.smithi160.stdout:837: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-21T18:07:56.912 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:56.912 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:56.912 INFO:tasks.rados.rados.0.smithi160.stdout:837: writing smithi16030472-445 from 635463 to 1370344 tid 1 2024-09-21T18:07:56.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:07:56.927 INFO:tasks.rados.rados.0.smithi160.stdout:837: writing smithi16030472-445 from 1987181 to 2235463 tid 2 2024-09-21T18:07:56.928 INFO:tasks.rados.rados.0.smithi160.stdout:838: write oid 21 current snap is 0 2024-09-21T18:07:56.928 INFO:tasks.rados.rados.0.smithi160.stdout:838: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-21T18:07:56.942 INFO:tasks.rados.rados.0.smithi160.stdout:838: writing smithi16030472-21 from 652270 to 1062400 tid 1 2024-09-21T18:07:56.950 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:56.945+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:56.954 INFO:tasks.rados.rados.0.smithi160.stdout:838: writing smithi16030472-21 from 1529310 to 2023177 tid 2 2024-09-21T18:07:56.955 INFO:tasks.rados.rados.0.smithi160.stdout:839: cache_flush oid 94 snap -1 2024-09-21T18:07:56.955 INFO:tasks.rados.rados.0.smithi160.stdout:840: write oid 449 current snap is 0 2024-09-21T18:07:56.955 INFO:tasks.rados.rados.0.smithi160.stdout:840: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-21T18:07:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:840: writing smithi16030472-449 from 669077 to 1154456 tid 1 2024-09-21T18:07:56.975 INFO:tasks.rados.rados.0.smithi160.stdout:840: writing smithi16030472-449 from 1587792 to 2269077 tid 2 2024-09-21T18:07:56.975 INFO:tasks.rados.rados.0.smithi160.stdout:841: cache_flush oid 229 snap -1 2024-09-21T18:07:56.975 INFO:tasks.rados.rados.0.smithi160.stdout:842: copy_from oid 442 from oid 275 current snap is 0 2024-09-21T18:07:56.975 INFO:tasks.rados.rados.0.smithi160.stdout:843: delete oid 310 current snap is 0 2024-09-21T18:07:56.979 INFO:tasks.rados.rados.0.smithi160.stdout:843: done (9 left) 2024-09-21T18:07:56.979 INFO:tasks.rados.rados.0.smithi160.stdout:844: delete oid 437 current snap is 0 2024-09-21T18:07:57.028 INFO:tasks.rados.rados.0.smithi160.stdout:837: finishing write tid 1 to smithi16030472-445 2024-09-21T18:07:57.028 INFO:tasks.rados.rados.0.smithi160.stdout:837: finishing write tid 2 to smithi16030472-445 2024-09-21T18:07:57.036 INFO:tasks.rados.rados.0.smithi160.stdout:837: finishing write tid 3 to smithi16030472-445 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:837: finishing write tid 4 to smithi16030472-445 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:837: oid 445 updating version 0 to 388 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:837: oid 445 updating version 388 to 389 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:837: oid 445 version 389 is already newer than 387 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 389 (ObjNum 609 snap 0 seq_num 609) dirty exists 2024-09-21T18:07:57.037 INFO:tasks.rados.rados.0.smithi160.stdout:837: left oid 445 (ObjNum 609 snap 0 seq_num 609) 2024-09-21T18:07:57.050 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.050 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.050 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:07:57.045+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:07:57.052 INFO:tasks.rados.rados.0.smithi160.stdout:833: finishing write tid 3 to smithi16030472-215 2024-09-21T18:07:57.052 INFO:tasks.rados.rados.0.smithi160.stdout:838: finishing write tid 1 to smithi16030472-21 2024-09-21T18:07:57.054 INFO:tasks.rados.rados.0.smithi160.stdout:833: finishing write tid 4 to smithi16030472-215 2024-09-21T18:07:57.054 INFO:tasks.rados.rados.0.smithi160.stdout:833: oid 215 updating version 0 to 466 2024-09-21T18:07:57.054 INFO:tasks.rados.rados.0.smithi160.stdout:833: oid 215 version 466 is already newer than 465 2024-09-21T18:07:57.054 INFO:tasks.rados.rados.0.smithi160.stdout:833: oid 215 version 466 is already newer than 464 2024-09-21T18:07:57.055 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 215 v 466 (ObjNum 607 snap 0 seq_num 607) dirty exists 2024-09-21T18:07:57.055 INFO:tasks.rados.rados.0.smithi160.stdout:833: left oid 215 (ObjNum 607 snap 0 seq_num 607) 2024-09-21T18:07:57.061 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:07:57.055+0000 7f1072a45640 -1 mon.a@0(leader).osd e62 definitely_dead 0 2024-09-21T18:07:57.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:07:57.092 INFO:tasks.rados.rados.0.smithi160.stdout:836: got (0) Success 2024-09-21T18:07:57.151 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:07:57.149+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:07:57.165 INFO:tasks.rados.rados.0.smithi160.stdout:841: got (0) Success 2024-09-21T18:07:57.165 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 229 v 0 (ObjNum 228 snap 0 seq_num 228) dirty exists 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:833: done (9 left) 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:836: done (8 left) 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:837: done (7 left) 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:841: done (6 left) 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:844: done (5 left) 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:845: cache_evict oid 332 snap -1 2024-09-21T18:07:57.184 INFO:tasks.rados.rados.0.smithi160.stdout:846: copy_from oid 377 from oid 199 current snap is 0 2024-09-21T18:07:57.185 INFO:tasks.rados.rados.0.smithi160.stdout:847: read oid 222 snap -1 2024-09-21T18:07:57.185 INFO:tasks.rados.rados.0.smithi160.stdout:847: expect (ObjNum 221 snap 0 seq_num 221) 2024-09-21T18:07:57.185 INFO:tasks.rados.rados.0.smithi160.stdout:848: delete oid 314 current snap is 0 2024-09-21T18:07:57.188 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.189 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:07:57.253 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:57.247+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:57.285 INFO:tasks.rados.rados.0.smithi160.stdout:845: got (0) Success 2024-09-21T18:07:57.338 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.338 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:07:57.353 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:57.347+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:57.385 INFO:tasks.rados.rados.0.smithi160.stdout:835: finishing copy_from to smithi16030472-359 2024-09-21T18:07:57.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 436 (ObjNum 577 snap 0 seq_num 577) dirty exists 2024-09-21T18:07:57.385 INFO:tasks.rados.rados.0.smithi160.stdout:835: finishing copy_from racing read to smithi16030472-359 2024-09-21T18:07:57.391 INFO:tasks.rados.rados.0.smithi160.stdout:840: finishing write tid 1 to smithi16030472-449 2024-09-21T18:07:57.391 INFO:tasks.rados.rados.0.smithi160.stdout:840: finishing write tid 2 to smithi16030472-449 2024-09-21T18:07:57.391 INFO:tasks.rados.rados.0.smithi160.stdout:838: finishing write tid 2 to smithi16030472-21 2024-09-21T18:07:57.454 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:57.449+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:57.480 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.480 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.490 INFO:tasks.rados.rados.0.smithi160.stdout:838: finishing write tid 3 to smithi16030472-21 2024-09-21T18:07:57.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:07:57.497 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:07:57.497 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.7. 2024-09-21T18:07:57.513 INFO:tasks.rados.rados.0.smithi160.stdout:847: expect (ObjNum 221 snap 0 seq_num 221) 2024-09-21T18:07:57.531 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-21T18:07:57.532 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:07:57.532 DEBUG:teuthology.orchestra.run.smithi160:> type ceph-objectstore-tool 2024-09-21T18:07:57.537 INFO:teuthology.orchestra.run.smithi160.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T18:07:57.538 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-21T18:07:57.582 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:57.578+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:57.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:07:57.661 INFO:tasks.rados.rados.0.smithi160.stdout:846: finishing copy_from racing read to smithi16030472-377 2024-09-21T18:07:57.661 INFO:tasks.rados.rados.0.smithi160.stdout:840: finishing write tid 3 to smithi16030472-449 2024-09-21T18:07:57.661 INFO:tasks.rados.rados.0.smithi160.stdout:840: finishing write tid 4 to smithi16030472-449 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:840: oid 449 updating version 0 to 467 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:840: oid 449 updating version 467 to 468 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:840: oid 449 updating version 468 to 469 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 469 (ObjNum 611 snap 0 seq_num 611) dirty exists 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:840: left oid 449 (ObjNum 611 snap 0 seq_num 611) 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:842: finishing copy_from racing read to smithi16030472-442 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:839: got (0) Success 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 0 (ObjNum 93 snap 0 seq_num 93) dirty exists 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:838: finishing write tid 4 to smithi16030472-21 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:838: oid 21 updating version 0 to 458 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:838: oid 21 updating version 458 to 459 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:838: oid 21 updating version 459 to 460 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 460 (ObjNum 610 snap 0 seq_num 610) dirty exists 2024-09-21T18:07:57.662 INFO:tasks.rados.rados.0.smithi160.stdout:838: left oid 21 (ObjNum 610 snap 0 seq_num 610) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:842: finishing copy_from to smithi16030472-442 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 199 (ObjNum 274 snap 0 seq_num 274) dirty exists 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:846: finishing copy_from to smithi16030472-377 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 377 v 107 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:835: done (8 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:838: done (7 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:839: done (6 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:840: done (5 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:842: done (4 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:845: done (3 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:846: done (2 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:847: done (1 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:848: done (0 left) 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:849: write oid 69 current snap is 0 2024-09-21T18:07:57.663 INFO:tasks.rados.rados.0.smithi160.stdout:849: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-21T18:07:57.665 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.665 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.680 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:07:57.681 INFO:tasks.rados.rados.0.smithi160.stdout:849: writing smithi16030472-69 from 736305 to 1522680 tid 1 2024-09-21T18:07:57.683 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:57.679+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:849: writing smithi16030472-69 from 2105367 to 2336305 tid 2 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:850: cache_evict oid 461 snap -1 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:849: finishing write tid 1 to smithi16030472-69 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:851: cache_evict oid 86 snap -1 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:852: cache_flush oid 179 snap -1 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:853: cache_try_flush oid 126 snap -1 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:854: write (excl) oid 292 current snap is 0 2024-09-21T18:07:57.691 INFO:tasks.rados.rados.0.smithi160.stdout:854: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-21T18:07:57.706 INFO:tasks.rados.rados.0.smithi160.stdout:854: writing smithi16030472-292 from 753112 to 1331089 tid 1 2024-09-21T18:07:57.716 INFO:tasks.rados.rados.0.smithi160.stdout:854: writing smithi16030472-292 from 1880202 to 2353112 tid 2 2024-09-21T18:07:57.716 INFO:tasks.rados.rados.0.smithi160.stdout:855: read oid 174 snap -1 2024-09-21T18:07:57.717 INFO:tasks.rados.rados.0.smithi160.stdout:855: expect (ObjNum 173 snap 0 seq_num 173) 2024-09-21T18:07:57.717 INFO:tasks.rados.rados.0.smithi160.stdout:856: delete oid 31 current snap is 0 2024-09-21T18:07:57.717 INFO:tasks.rados.rados.0.smithi160.stdout:850: got (16) Device or resource busy 2024-09-21T18:07:57.756 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:57.749+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:57.791 INFO:tasks.rados.rados.0.smithi160.stdout:851: got (0) Success 2024-09-21T18:07:57.799 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:07:57.799 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:07:57.802 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:07:57.802 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:07:57.814 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:07:57.821 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:07:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:07:57.843 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:07:57.843 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:07:57.857 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:57.850+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:57.865 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:07:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:07:57.886 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:07:57.886 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:07:57.893 INFO:tasks.rados.rados.0.smithi160.stdout:852: got (0) Success 2024-09-21T18:07:57.893 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 0 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-21T18:07:57.893 INFO:tasks.rados.rados.0.smithi160.stdout:849: finishing write tid 2 to smithi16030472-69 2024-09-21T18:07:57.893 INFO:tasks.rados.rados.0.smithi160.stdout:849: finishing write tid 3 to smithi16030472-69 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:849: finishing write tid 4 to smithi16030472-69 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:849: oid 69 updating version 0 to 451 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:849: oid 69 updating version 451 to 452 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:849: oid 69 version 452 is already newer than 450 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 452 (ObjNum 615 snap 0 seq_num 615) dirty exists 2024-09-21T18:07:57.894 INFO:tasks.rados.rados.0.smithi160.stdout:849: left oid 69 (ObjNum 615 snap 0 seq_num 615) 2024-09-21T18:07:57.909 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:07:57.909 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:07:57.927 INFO:teuthology.orchestra.run.smithi160.stdout:2.7 2024-09-21T18:07:57.931 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:07:57.932 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:07:57.932 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:07:57.934 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:57.935 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:57.935 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:57.949 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:07:57.952 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:07:57.952 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:07:57.957 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:07:57.951+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:07:57.972 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:07:57.972 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:07:58.018 INFO:tasks.rados.rados.0.smithi160.stdout:855: expect (ObjNum 173 snap 0 seq_num 173) 2024-09-21T18:07:58.018 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:58.021 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:07:58.021 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:07:58.047 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:07:58.048 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:07:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:58.058 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:58.055+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:58.071 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:07:58.072 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 73.770550702999998, 2024-09-21T18:07:58.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 73.769336644000006, 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 73.767985723999999, 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.082 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:07:58.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:07:58.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:07:58.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 67.480405086999994, 2024-09-21T18:07:58.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:07:58.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:07:58.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.090 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 45.381812457000002, 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:07:58.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 45.381335247000003, 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:07:58.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 45.380900294, 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.099 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:07:58.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1028 3.2 e49/31 3:5a53c113:::smithi16030472-229:head v 49'122, mlcod=49'116)", 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:25.498789+0000", 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 32.570813369, 2024-09-21T18:07:58.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20736566000000001, 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.498789+0000", 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.498789+0000", 2024-09-21T18:07:58.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.498792+0000", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7499999999999999e-06 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.499702+0000", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00091011200000000003 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.499727+0000", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4783e-05 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.599868+0000", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10014120999999999 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.700122+0000", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100254278 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.700164+0000", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1965999999999998e-05 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.705910+0000", 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057455869999999999 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:25.706155+0000", 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024497400000000002 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.081165847000001, 2024-09-21T18:07:58.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:07:58.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.980596455000001, 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:07:58.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:07:58.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:07:58.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.980238388, 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:07:58.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:07:58.121 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.980201555000001, 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:07:58.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:07:58.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:07:58.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.979741823000001, 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.129 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.979310296000001, 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:07:58.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:07:58.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.978373681000001, 2024-09-21T18:07:58.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.978334998000001, 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:07:58.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:07:58.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:07:58.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.977048346, 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.874903882000002, 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:07:58.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:07:58.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.874180088999999, 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:07:58.151 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.155 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 18.748469849999999, 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:07:58.159 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.160 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.160 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.160 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.160 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.160 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:58.161 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:07:58.162 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:07:58.162 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:07:58.163 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:07:58.163 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:07:58.163 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:07:58.157+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 487679 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:854: finishing write tid 1 to smithi16030472-292 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:854: finishing write tid 2 to smithi16030472-292 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:849: done (7 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:850: done (6 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:851: done (5 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:852: done (4 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:855: done (3 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:856: done (2 left) 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:857: copy_from oid 216 from oid 173 current snap is 0 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:854: finishing write tid 3 to smithi16030472-292 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:854: finishing write tid 4 to smithi16030472-292 2024-09-21T18:07:58.164 INFO:tasks.rados.rados.0.smithi160.stdout:854: oid 292 updating version 0 to 455 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:854: oid 292 version 455 is already newer than 454 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:854: oid 292 version 455 is already newer than 453 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 292 v 455 (ObjNum 616 snap 0 seq_num 616) dirty exists 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:854: left oid 292 (ObjNum 616 snap 0 seq_num 616) 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:854: done (2 left) 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:858: cache_evict oid 31 snap -1 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:859: copy_from oid 136 from oid 433 current snap is 0 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:860: read oid 355 snap -1 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:860: expect (ObjNum 354 snap 0 seq_num 354) 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:861: cache_try_flush oid 145 snap -1 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:862: cache_try_flush oid 6 snap -1 2024-09-21T18:07:58.165 INFO:tasks.rados.rados.0.smithi160.stdout:863: read oid 281 snap -1 2024-09-21T18:07:58.166 INFO:tasks.rados.rados.0.smithi160.stdout:863: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-21T18:07:58.166 INFO:tasks.rados.rados.0.smithi160.stdout:864: cache_flush oid 445 snap -1 2024-09-21T18:07:58.166 INFO:tasks.rados.rados.0.smithi160.stdout:865: delete oid 253 current snap is 0 2024-09-21T18:07:58.206 INFO:tasks.rados.rados.0.smithi160.stdout:853: got (0) Success 2024-09-21T18:07:58.206 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 126 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:07:58.206 INFO:tasks.rados.rados.0.smithi160.stdout:861: got (0) Success 2024-09-21T18:07:58.207 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 145 v 0 (ObjNum 144 snap 0 seq_num 144) dirty exists 2024-09-21T18:07:58.225 INFO:tasks.rados.rados.0.smithi160.stdout:858: got (16) Device or resource busy 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:853: done (9 left) 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:858: done (8 left) 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:861: done (7 left) 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:865: done (6 left) 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:866: read oid 275 snap -1 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:866: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:07:58.229 INFO:tasks.rados.rados.0.smithi160.stdout:867: delete oid 322 current snap is 0 2024-09-21T18:07:58.231 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:07:58.232 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:07:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:07:58.236 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:07:58.262 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:58.261+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:58.335 INFO:tasks.rados.rados.0.smithi160.stdout:867: done (7 left) 2024-09-21T18:07:58.335 INFO:tasks.rados.rados.0.smithi160.stdout:868: delete oid 19 current snap is 0 2024-09-21T18:07:58.339 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:58.340 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:58.340 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:58.340 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:58.354 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:07:58.363 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:07:58.359+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:07:58.426 INFO:tasks.rados.rados.0.smithi160.stdout:859: finishing copy_from racing read to smithi16030472-136 2024-09-21T18:07:58.436 INFO:tasks.rados.rados.0.smithi160.stdout:857: finishing copy_from racing read to smithi16030472-216 2024-09-21T18:07:58.437 INFO:tasks.rados.rados.0.smithi160.stdout:860: expect (ObjNum 354 snap 0 seq_num 354) 2024-09-21T18:07:58.437 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 508589 2024-09-21T18:07:58.463 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:58.461+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:58.473 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 2.7 --file /var/log/ceph/exp.2.7.7 2024-09-21T18:07:58.478 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:58.478 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:58.478 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:58.478 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:58.478 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:58.493 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:07:58.557 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 567394 2024-09-21T18:07:58.565 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:58.557+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:58.595 INFO:tasks.rados.rados.0.smithi160.stdout:866: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:07:58.664 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.629718449999999, 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:07:58.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.626107899000001, 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:07:58.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.623301359999999, 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:07:58.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 27.619553159999999, 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.677 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:07:58.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:07:58.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.998702355999999, 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.681 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.897848283999998, 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.685 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:07:58.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.897092004000001, 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:07:58.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:07:58.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:07:58.695 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.896700447000001, 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:07:58.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.699 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.896399323000001, 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:07:58.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:07:58.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.895675473000001, 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.704 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:07:58.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:07:58.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:07:58.708 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.794778634, 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:07:58.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.712 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.794684351000001, 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:07:58.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:07:58.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.793771155999998, 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:07:58.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:07:58.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.793201467999999, 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:07:58.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:07:58.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:07:58.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.792473984000001, 2024-09-21T18:07:58.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:07:58.731 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 26.726292075, 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.735 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.470246270000001, 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:07:58.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:07:58.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:07:58.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:07:58.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.469963472, 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.469884487000002, 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.744 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:07:58.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:07:58.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:07:58.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.467266156000001, 2024-09-21T18:07:58.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:07:58.752 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:857: finishing copy_from to smithi16030472-216 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 216 v 463 (ObjNum 605 snap 0 seq_num 605) dirty exists 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:863: expect (ObjNum 280 snap 0 seq_num 280) 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:58.755 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:58.750+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:58.755 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:07:58.761 INFO:tasks.rados.rados.0.smithi160.stdout:859: finishing copy_from to smithi16030472-136 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 136 v 425 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:857: done (7 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:859: done (6 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:860: done (5 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:863: done (4 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:866: done (3 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:868: done (2 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:869: cache_flush oid 99 snap -1 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:864: got (0) Success 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 0 (ObjNum 609 snap 0 seq_num 609) dirty exists 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:862: got (0) Success 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:862: done (2 left) 2024-09-21T18:07:58.762 INFO:tasks.rados.rados.0.smithi160.stdout:864: done (1 left) 2024-09-21T18:07:58.763 INFO:tasks.rados.rados.0.smithi160.stdout:870: cache_evict oid 467 snap -1 2024-09-21T18:07:58.763 INFO:tasks.rados.rados.0.smithi160.stdout:871: cache_evict oid 372 snap -1 2024-09-21T18:07:58.763 INFO:tasks.rados.rados.0.smithi160.stdout:872: cache_flush oid 489 snap -1 2024-09-21T18:07:58.763 INFO:tasks.rados.rados.0.smithi160.stdout:873: write (excl) oid 379 current snap is 0 2024-09-21T18:07:58.763 INFO:tasks.rados.rados.0.smithi160.stdout:873: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-21T18:07:58.767 INFO:tasks.rados.rados.0.smithi160.stdout:873: writing smithi16030472-379 from 437147 to 991369 tid 1 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:873: writing smithi16030472-379 from 1656259 to 2312129 tid 2 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:874: cache_flush oid 244 snap -1 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:872: got (0) Success 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 489 v 0 (ObjNum 488 snap 0 seq_num 488) dirty exists 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:872: done (5 left) 2024-09-21T18:07:58.775 INFO:tasks.rados.rados.0.smithi160.stdout:875: delete oid 483 current snap is 0 2024-09-21T18:07:58.851 INFO:teuthology.orchestra.run.smithi160.stderr:Exporting 2.7 info 2.7( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) 2024-09-21T18:07:58.852 INFO:teuthology.orchestra.run.smithi160.stderr:Export successful 2024-09-21T18:07:58.854 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:07:58.850+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:07:58.865 INFO:tasks.rados.rados.0.smithi160.stdout:871: got (0) Success 2024-09-21T18:07:58.877 INFO:tasks.rados.rados.0.smithi160.stdout:874: got (2) No such file or directory 2024-09-21T18:07:58.894 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:58.894 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:58.894 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:07:58.894 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:58.910 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:07:58.955 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:07:58.953+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:07:58.984 INFO:tasks.rados.rados.0.smithi160.stdout:871: done (5 left) 2024-09-21T18:07:58.984 INFO:tasks.rados.rados.0.smithi160.stdout:874: done (4 left) 2024-09-21T18:07:58.984 INFO:tasks.rados.rados.0.smithi160.stdout:875: done (3 left) 2024-09-21T18:07:58.984 INFO:tasks.rados.rados.0.smithi160.stdout:876: cache_evict oid 255 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:877: cache_flush oid 226 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:878: read oid 192 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:878: expect (ObjNum 191 snap 0 seq_num 191) 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:879: read oid 112 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:879: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:880: cache_flush oid 36 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:881: cache_try_flush oid 470 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:882: read oid 20 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:882: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:883: read oid 425 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:883: expect (ObjNum 424 snap 0 seq_num 424) 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:884: read oid 175 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:884: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:885: cache_evict oid 477 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:886: cache_try_flush oid 408 snap -1 2024-09-21T18:07:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:887: cache_evict oid 455 snap -1 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:888: cache_evict oid 260 snap -1 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:876: got (16) Device or resource busy 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:876: done (15 left) 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:889: read oid 420 snap -1 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:889: expect (ObjNum 419 snap 0 seq_num 419) 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:877: got (2) No such file or directory 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:877: done (15 left) 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout:890: cache_try_flush oid 325 snap -1 2024-09-21T18:07:58.986 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:58.994 INFO:tasks.rados.rados.0.smithi160.stdout:879: expect (ObjNum 111 snap 0 seq_num 111) 2024-09-21T18:07:59.020 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3715852 2024-09-21T18:07:59.030 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:59.030 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:07:59.030 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:07:59.031 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:07:59.031 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:59.045 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:07:59.056 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:07:59.050+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:07:59.099 INFO:tasks.rados.rados.0.smithi160.stdout:887: got (0) Success 2024-09-21T18:07:59.099 INFO:tasks.rados.rados.0.smithi160.stdout:882: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-21T18:07:59.099 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.157 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:59.153+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:59.159 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:870: got (0) Success 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:880: got (0) Success 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 0 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:885: got (2) No such file or directory 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:881: got (0) Success 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 470 v 0 (ObjNum 469 snap 0 seq_num 469) dirty exists 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:869: got (0) Success 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 0 (ObjNum 98 snap 0 seq_num 98) dirty exists 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:890: got (2) No such file or directory 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:886: got (0) Success 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 0 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:873: finishing write tid 1 to smithi16030472-379 2024-09-21T18:07:59.201 INFO:tasks.rados.rados.0.smithi160.stdout:869: done (15 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:870: done (14 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:879: done (13 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:880: done (12 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:881: done (11 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:882: done (10 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:885: done (9 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:886: done (8 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:887: done (7 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:890: done (6 left) 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:891: cache_try_flush oid 9 snap -1 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:892: cache_try_flush oid 417 snap -1 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:893: cache_evict oid 80 snap -1 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:894: cache_try_flush oid 295 snap -1 2024-09-21T18:07:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:895: cache_flush oid 116 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:896: cache_try_flush oid 328 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:897: read oid 306 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:897: expect deleted 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:898: read oid 272 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:898: expect (ObjNum 520 snap 0 seq_num 520) 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:899: read oid 284 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:899: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:900: read oid 226 snap -1 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:900: expect (ObjNum 225 snap 0 seq_num 225) 2024-09-21T18:07:59.203 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:59.214 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:07:59.214 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:07:59.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:07:59.214 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:07:59.215 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:07:59.227 INFO:tasks.rados.rados.0.smithi160.stdout:884: expect (ObjNum 174 snap 0 seq_num 174) 2024-09-21T18:07:59.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:07:59.236 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2719333 2024-09-21T18:07:59.244 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.274 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:59.269+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:59.280 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 761250 2024-09-21T18:07:59.306 INFO:tasks.rados.rados.0.smithi160.stdout:884: done (15 left) 2024-09-21T18:07:59.306 INFO:tasks.rados.rados.0.smithi160.stdout:901: copy_from oid 234 from oid 332 current snap is 0 2024-09-21T18:07:59.306 INFO:tasks.rados.rados.0.smithi160.stdout:893: got (16) Device or resource busy 2024-09-21T18:07:59.306 INFO:tasks.rados.rados.0.smithi160.stdout:888: got (2) No such file or directory 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:888: done (15 left) 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:893: done (14 left) 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:902: copy_from oid 195 from oid 365 current snap is 0 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:903: cache_flush oid 135 snap -1 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:896: got (0) Success 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:07:59.307 INFO:tasks.rados.rados.0.smithi160.stdout:878: expect (ObjNum 191 snap 0 seq_num 191) 2024-09-21T18:07:59.359 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:07:59.355+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:07:59.361 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.11.0:746 4.1 e64/33)", 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:59.317178+0000", 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.044330427999999998, 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.044348278999999997, 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317178+0000", 2024-09-21T18:07:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317178+0000", 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317178+0000", 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7000000000000001e-07 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317183+0000", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.1970000000000004e-06 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317200+0000", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7042e-05 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:59.317212+0000", 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1843e-05 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:07:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:07:59.365 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.375 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:07:59.389 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 2.7 2024-09-21T18:07:59.406 INFO:tasks.rados.rados.0.smithi160.stdout:883: expect (ObjNum 424 snap 0 seq_num 424) 2024-09-21T18:07:59.418 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1959467 2024-09-21T18:07:59.435 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 825831 2024-09-21T18:07:59.459 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.460 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:59.455+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:59.492 INFO:tasks.rados.rados.0.smithi160.stdout:873: finishing write tid 2 to smithi16030472-379 2024-09-21T18:07:59.493 INFO:tasks.rados.rados.0.smithi160.stdout:873: finishing write tid 3 to smithi16030472-379 2024-09-21T18:07:59.493 INFO:tasks.rados.rados.0.smithi160.stdout:898: expect (ObjNum 520 snap 0 seq_num 520) 2024-09-21T18:07:59.514 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2071647 2024-09-21T18:07:59.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:07:59.538 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:59.538 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:59.538 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:07:59.538 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:07:59.538 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:59.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:07:59.559 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.560 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:07:59.557+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:07:59.598 INFO:tasks.rados.rados.0.smithi160.stdout:873: finishing write tid 4 to smithi16030472-379 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:873: oid 379 updating version 0 to 390 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:873: oid 379 updating version 390 to 392 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:873: oid 379 version 392 is already newer than 391 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 379 v 392 (ObjNum 621 snap 0 seq_num 621) dirty exists 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:873: left oid 379 (ObjNum 621 snap 0 seq_num 621) 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:895: got (0) Success 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 0 (ObjNum 597 snap 0 seq_num 597) dirty exists 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:889: expect (ObjNum 419 snap 0 seq_num 419) 2024-09-21T18:07:59.599 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.661 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:07:59.655+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:07:59.674 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:59.674 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 41.341910425999998, 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.502058429999998, 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:07:59.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:59.683 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.398337982000001, 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:07:59.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.397563239, 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:07:59.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.690 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.396915219999997, 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.692 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:07:59.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:07:59.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:07:59.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.383043524999998, 2024-09-21T18:07:59.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:07:59.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.380102508999997, 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:07:59.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.377085661000002, 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.704 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:07:59.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:07:59.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.374874079000001, 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.708 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:07:59.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:59.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.269536987000002, 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:59.716 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 34.578114612999997, 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 34.575084613000001, 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:07:59.720 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:07:59.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 29.608520650999999, 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.724 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:07:59.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 27.941685216, 2024-09-21T18:07:59.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:07:59.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:07:59.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:07:59.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:07:59.732 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.638194134999999, 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:07:59.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:07:59.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:07:59.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.635897102000001, 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:07:59.739 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:07:59.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.633846653999999, 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:07:59.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.631691292999999, 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.745 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:07:59.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:07:59.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:07:59.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.630928534999999, 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.52902349, 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:07:59.754 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:07:59.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:07:59.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:07:59.758 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:903: got (0) Success 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 135 v 0 (ObjNum 599 snap 0 seq_num 599) dirty exists 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:892: got (0) Success 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 0 (ObjNum 502 snap 0 seq_num 502) dirty exists 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:899: expect (ObjNum 283 snap 0 seq_num 283) 2024-09-21T18:07:59.759 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.760 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:07:59.761 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:07:59.753+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:07:59.761 INFO:tasks.rados.rados.0.smithi160.stdout:902: finishing copy_from racing read to smithi16030472-195 2024-09-21T18:07:59.761 INFO:tasks.rados.rados.0.smithi160.stdout:901: finishing copy_from to smithi16030472-234 2024-09-21T18:07:59.761 INFO:tasks.rados.rados.0.smithi160.stdout:901: got expected ENOENT (src dne) 2024-09-21T18:07:59.762 INFO:tasks.rados.rados.0.smithi160.stdout:900: expect (ObjNum 225 snap 0 seq_num 225) 2024-09-21T18:07:59.762 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:07:59.849 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2507085 2024-09-21T18:07:59.862 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:07:59.857+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:902: finishing copy_from to smithi16030472-195 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 242 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:894: got (0) Success 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 294 snap 0 seq_num 294) dirty exists 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:891: got (0) Success 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 9 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:07:59.867 INFO:tasks.rados.rados.0.smithi160.stdout:901: finishing copy_from racing read to smithi16030472-234 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:873: done (15 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:878: done (14 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:883: done (13 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:889: done (12 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:891: done (11 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:892: done (10 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:894: done (9 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:895: done (8 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:896: done (7 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:897: done (6 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:898: done (5 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:899: done (4 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:900: done (3 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:901: done (2 left) 2024-09-21T18:07:59.868 INFO:tasks.rados.rados.0.smithi160.stdout:902: done (1 left) 2024-09-21T18:07:59.869 INFO:tasks.rados.rados.0.smithi160.stdout:903: done (0 left) 2024-09-21T18:07:59.869 INFO:tasks.rados.rados.0.smithi160.stdout:904: write oid 255 current snap is 0 2024-09-21T18:07:59.869 INFO:tasks.rados.rados.0.smithi160.stdout:904: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-21T18:07:59.886 INFO:tasks.rados.rados.0.smithi160.stdout:904: writing smithi16030472-255 from 470761 to 1175481 tid 1 2024-09-21T18:07:59.886 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:07:59.886 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:07:59.886 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:07:59.886 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:07:59.901 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:07:59.905 INFO:tasks.rados.rados.0.smithi160.stdout:904: writing smithi16030472-255 from 1656870 to 2316409 tid 2 2024-09-21T18:07:59.905 INFO:tasks.rados.rados.0.smithi160.stdout:905: read oid 181 snap -1 2024-09-21T18:07:59.905 INFO:tasks.rados.rados.0.smithi160.stdout:905: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-21T18:07:59.905 INFO:tasks.rados.rados.0.smithi160.stdout:906: write (excl) oid 37 current snap is 0 2024-09-21T18:07:59.905 INFO:tasks.rados.rados.0.smithi160.stdout:906: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-21T18:07:59.920 INFO:tasks.rados.rados.0.smithi160.stdout:906: writing smithi16030472-37 from 487568 to 983890 tid 1 2024-09-21T18:07:59.932 INFO:tasks.rados.rados.0.smithi160.stdout:906: writing smithi16030472-37 from 1431705 to 1951255 tid 2 2024-09-21T18:07:59.932 INFO:tasks.rados.rados.0.smithi160.stdout:907: copy_from oid 15 from oid 390 current snap is 0 2024-09-21T18:07:59.932 INFO:tasks.rados.rados.0.smithi160.stdout:904: finishing write tid 1 to smithi16030472-255 2024-09-21T18:07:59.932 INFO:tasks.rados.rados.0.smithi160.stdout:908: write (excl) oid 268 current snap is 0 2024-09-21T18:07:59.932 INFO:tasks.rados.rados.0.smithi160.stdout:908: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-21T18:07:59.943 INFO:tasks.rados.rados.0.smithi160.stdout:908: writing smithi16030472-268 from 504375 to 1075946 tid 1 2024-09-21T18:07:59.954 INFO:tasks.rados.rados.0.smithi160.stdout:908: writing smithi16030472-268 from 1490187 to 2153395 tid 2 2024-09-21T18:07:59.954 INFO:tasks.rados.rados.0.smithi160.stdout:909: write oid 227 current snap is 0 2024-09-21T18:07:59.954 INFO:tasks.rados.rados.0.smithi160.stdout:909: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-21T18:07:59.962 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:07:59.958+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:07:59.966 INFO:tasks.rados.rados.0.smithi160.stdout:909: writing smithi16030472-227 from 521182 to 1168002 tid 1 2024-09-21T18:07:59.978 INFO:tasks.rados.rados.0.smithi160.stdout:909: writing smithi16030472-227 from 1948669 to 2471888 tid 2 2024-09-21T18:07:59.978 INFO:tasks.rados.rados.0.smithi160.stdout:910: cache_try_flush oid 491 snap -1 2024-09-21T18:07:59.978 INFO:tasks.rados.rados.0.smithi160.stdout:906: finishing write tid 1 to smithi16030472-37 2024-09-21T18:07:59.979 INFO:tasks.rados.rados.0.smithi160.stdout:911: cache_flush oid 61 snap -1 2024-09-21T18:07:59.979 INFO:tasks.rados.rados.0.smithi160.stdout:912: read oid 477 snap -1 2024-09-21T18:07:59.979 INFO:tasks.rados.rados.0.smithi160.stdout:912: expect (ObjNum 476 snap 0 seq_num 476) 2024-09-21T18:07:59.979 INFO:tasks.rados.rados.0.smithi160.stdout:913: cache_try_flush oid 153 snap -1 2024-09-21T18:07:59.979 INFO:tasks.rados.rados.0.smithi160.stdout:914: delete oid 394 current snap is 0 2024-09-21T18:08:00.060 INFO:tasks.rados.rados.0.smithi160.stdout:911: got (2) No such file or directory 2024-09-21T18:08:00.063 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:00.058+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:00.071 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:00.071 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:00.071 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:00.071 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:00.071 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:00.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:904: finishing write tid 2 to smithi16030472-255 2024-09-21T18:08:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:904: finishing write tid 3 to smithi16030472-255 2024-09-21T18:08:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:904: finishing write tid 4 to smithi16030472-255 2024-09-21T18:08:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:904: oid 255 updating version 0 to 457 2024-09-21T18:08:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:904: oid 255 updating version 457 to 459 2024-09-21T18:08:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:904: oid 255 version 459 is already newer than 458 2024-09-21T18:08:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 255 v 459 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:08:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:904: left oid 255 (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:08:00.164 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:00.159+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:00.180 INFO:tasks.rados.rados.0.smithi160.stdout:905: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-21T18:08:00.180 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:00.210 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:00.211 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:00.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:00.211 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:08:00.211 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:00.223 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:00.225 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:00.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:00.265 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:00.261+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:907: finishing copy_from racing read to smithi16030472-15 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:910: got (0) Success 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:909: finishing write tid 1 to smithi16030472-227 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:906: finishing write tid 2 to smithi16030472-37 2024-09-21T18:08:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:906: finishing write tid 3 to smithi16030472-37 2024-09-21T18:08:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:904: done (10 left) 2024-09-21T18:08:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:905: done (9 left) 2024-09-21T18:08:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:910: done (8 left) 2024-09-21T18:08:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:911: done (7 left) 2024-09-21T18:08:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:914: done (6 left) 2024-09-21T18:08:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:915: delete oid 63 current snap is 0 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3156 4.2 e65/33)", 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:00.290851+0000", 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.057641380999999998, 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.057676013999999998, 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "no flag points reached", 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:00.290851+0000", 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:00.290851+0000", 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:00.290884+0000", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3185000000000002e-05 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:00.291459+0000", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00057506400000000002 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:00.291472+0000", 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3018e-05 2024-09-21T18:08:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:08:00.352 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:00.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:00.366 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:00.365+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:906: finishing write tid 4 to smithi16030472-37 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:906: oid 37 updating version 0 to 465 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:906: oid 37 version 465 is already newer than 464 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:906: oid 37 version 465 is already newer than 463 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 37 v 465 (ObjNum 624 snap 0 seq_num 624) dirty exists 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:906: left oid 37 (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:907: finishing copy_from to smithi16030472-15 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 271 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:908: finishing write tid 1 to smithi16030472-268 2024-09-21T18:08:00.386 INFO:tasks.rados.rados.0.smithi160.stdout:913: got (2) No such file or directory 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: finishing write tid 2 to smithi16030472-268 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: finishing write tid 3 to smithi16030472-268 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: finishing write tid 4 to smithi16030472-268 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: oid 268 updating version 0 to 475 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: oid 268 updating version 475 to 476 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: oid 268 version 476 is already newer than 474 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 268 v 476 (ObjNum 625 snap 0 seq_num 625) dirty exists 2024-09-21T18:08:00.387 INFO:tasks.rados.rados.0.smithi160.stdout:908: left oid 268 (ObjNum 625 snap 0 seq_num 625) 2024-09-21T18:08:00.389 INFO:tasks.rados.rados.0.smithi160.stdout:912: expect (ObjNum 476 snap 0 seq_num 476) 2024-09-21T18:08:00.394 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:00.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:00.409 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:00.414 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:00.415 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:00.433 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:00.433 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: finishing write tid 2 to smithi16030472-227 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: finishing write tid 3 to smithi16030472-227 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: finishing write tid 4 to smithi16030472-227 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: oid 227 updating version 0 to 393 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: oid 227 version 393 is already newer than 391 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:909: oid 227 updating version 393 to 394 2024-09-21T18:08:00.450 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 227 v 394 (ObjNum 626 snap 0 seq_num 626) dirty exists 2024-09-21T18:08:00.451 INFO:tasks.rados.rados.0.smithi160.stdout:909: left oid 227 (ObjNum 626 snap 0 seq_num 626) 2024-09-21T18:08:00.454 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:00.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:00.467 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:00.461+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:00.473 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:00.488 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:00.488 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:00.488 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:00.488 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:00.488 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:00.495 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:00.495 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:00.502 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:00.515 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:08:00.519 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:00.519 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:00.520 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:00.540 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:00.540 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:00.561 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:00.561 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:00.568 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:00.565+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:00.578 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:00.599 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:00.599 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:00.616 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:00.616 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:00.618 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:00.619 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:00.633 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:00.637 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:00.637 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:00.654 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:00.654 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:00.669 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:00.665+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:00.673 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:00.673 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:00.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:00.677 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:00.689 INFO:tasks.rados.rados.0.smithi160.stdout:906: done (6 left) 2024-09-21T18:08:00.689 INFO:tasks.rados.rados.0.smithi160.stdout:907: done (5 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:908: done (4 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:909: done (3 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:912: done (2 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:913: done (1 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:915: done (0 left) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:916: read oid 102 snap -1 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:916: expect (ObjNum 566 snap 0 seq_num 566) 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:917: write (excl) oid 442 current snap is 0 2024-09-21T18:08:00.690 INFO:tasks.rados.rados.0.smithi160.stdout:917: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-21T18:08:00.703 INFO:tasks.rados.rados.0.smithi160.stdout:917: writing smithi16030472-442 from 571603 to 1044170 tid 1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:917: writing smithi16030472-442 from 1607762 to 2278308 tid 2 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:918: cache_flush oid 469 snap -1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:919: cache_flush oid 495 snap -1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:920: read oid 385 snap -1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:920: expect (ObjNum 87 snap 0 seq_num 87) 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:921: cache_evict oid 92 snap -1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:922: cache_try_flush oid 353 snap -1 2024-09-21T18:08:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:923: cache_try_flush oid 144 snap -1 2024-09-21T18:08:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:924: copy_from oid 482 from oid 365 current snap is 0 2024-09-21T18:08:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:925: read oid 357 snap -1 2024-09-21T18:08:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:925: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:08:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:926: write (excl) oid 284 current snap is 0 2024-09-21T18:08:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:926: seq_num 630 ranges {588410=547816,1666244=414204,2484603=103807} 2024-09-21T18:08:00.730 INFO:tasks.rados.rados.0.smithi160.stdout:926: writing smithi16030472-284 from 588410 to 1136226 tid 1 2024-09-21T18:08:00.739 INFO:tasks.rados.rados.0.smithi160.stdout:926: writing smithi16030472-284 from 1666244 to 2080448 tid 2 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:926: writing smithi16030472-284 from 2484603 to 2588410 tid 3 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:927: read oid 353 snap -1 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:927: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:928: cache_try_flush oid 160 snap -1 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:921: got (16) Device or resource busy 2024-09-21T18:08:00.745 INFO:tasks.rados.rados.0.smithi160.stdout:923: got (2) No such file or directory 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:924: finishing copy_from racing read to smithi16030472-482 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:921: done (12 left) 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:923: done (11 left) 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:929: cache_flush oid 347 snap -1 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:930: write (excl) oid 83 current snap is 0 2024-09-21T18:08:00.746 INFO:tasks.rados.rados.0.smithi160.stdout:930: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-21T18:08:00.754 INFO:tasks.rados.rados.0.smithi160.stdout:930: writing smithi16030472-83 from 605217 to 1344635 tid 1 2024-09-21T18:08:00.758 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:00.758 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:00.764 INFO:tasks.rados.rados.0.smithi160.stdout:930: writing smithi16030472-83 from 2124726 to 2605217 tid 2 2024-09-21T18:08:00.764 INFO:tasks.rados.rados.0.smithi160.stdout:931: write oid 462 current snap is 0 2024-09-21T18:08:00.764 INFO:tasks.rados.rados.0.smithi160.stdout:931: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-21T18:08:00.770 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:00.765+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:00.772 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:00.773 INFO:tasks.rados.rados.0.smithi160.stdout:931: writing smithi16030472-462 from 622024 to 1036691 tid 1 2024-09-21T18:08:00.781 INFO:tasks.rados.rados.0.smithi160.stdout:931: writing smithi16030472-462 from 1783208 to 2201081 tid 2 2024-09-21T18:08:00.781 INFO:tasks.rados.rados.0.smithi160.stdout:932: read oid 387 snap -1 2024-09-21T18:08:00.781 INFO:tasks.rados.rados.0.smithi160.stdout:932: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-21T18:08:00.781 INFO:tasks.rados.rados.0.smithi160.stdout:933: cache_evict oid 431 snap -1 2024-09-21T18:08:00.782 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:00.871 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:00.866+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:00.897 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:00.897 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:00.913 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:00.930 INFO:tasks.rados.rados.0.smithi160.stdout:927: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-21T18:08:00.931 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:00.948 INFO:teuthology.orchestra.run.smithi160.stdout: marking collection for removal 2024-09-21T18:08:00.949 INFO:teuthology.orchestra.run.smithi160.stdout:setting '_remove' omap key 2024-09-21T18:08:00.949 INFO:teuthology.orchestra.run.smithi160.stdout:finish_remove_pgs 2.7_head removing 2.7 2024-09-21T18:08:00.952 INFO:teuthology.orchestra.run.smithi160.stdout:Remove successful 2024-09-21T18:08:00.972 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:00.965+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:01.008 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3510105 2024-09-21T18:08:01.034 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.034 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.050 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:01.061 INFO:tasks.rados.rados.0.smithi160.stdout:928: got (2) No such file or directory 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:929: got (2) No such file or directory 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:927: done (15 left) 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:928: done (14 left) 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:929: done (13 left) 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:934: delete oid 458 current snap is 0 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:924: finishing copy_from to smithi16030472-482 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 482 v 242 (ObjNum 364 snap 0 seq_num 364) dirty exists 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:926: finishing write tid 1 to smithi16030472-284 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:930: finishing write tid 1 to smithi16030472-83 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:930: finishing write tid 2 to smithi16030472-83 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:930: finishing write tid 3 to smithi16030472-83 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:930: finishing write tid 4 to smithi16030472-83 2024-09-21T18:08:01.062 INFO:tasks.rados.rados.0.smithi160.stdout:930: oid 83 updating version 0 to 396 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:930: oid 83 version 396 is already newer than 395 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:930: oid 83 updating version 396 to 397 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 83 v 397 (ObjNum 631 snap 0 seq_num 631) dirty exists 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:930: left oid 83 (ObjNum 631 snap 0 seq_num 631) 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: finishing write tid 2 to smithi16030472-284 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: finishing write tid 3 to smithi16030472-284 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: finishing write tid 4 to smithi16030472-284 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: finishing write tid 5 to smithi16030472-284 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: oid 284 updating version 0 to 460 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: oid 284 updating version 460 to 461 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: oid 284 updating version 461 to 462 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:926: oid 284 updating version 462 to 463 2024-09-21T18:08:01.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 284 v 463 (ObjNum 630 snap 0 seq_num 630) dirty exists 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:926: left oid 284 (ObjNum 630 snap 0 seq_num 630) 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: finishing write tid 1 to smithi16030472-462 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: finishing write tid 2 to smithi16030472-462 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: finishing write tid 3 to smithi16030472-462 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: finishing write tid 4 to smithi16030472-462 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: oid 462 updating version 0 to 464 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: oid 462 updating version 464 to 465 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: oid 462 updating version 465 to 466 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 462 v 466 (ObjNum 632 snap 0 seq_num 632) dirty exists 2024-09-21T18:08:01.064 INFO:tasks.rados.rados.0.smithi160.stdout:931: left oid 462 (ObjNum 632 snap 0 seq_num 632) 2024-09-21T18:08:01.074 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:01.069+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:01.162 INFO:tasks.rados.rados.0.smithi160.stdout:918: got (0) Success 2024-09-21T18:08:01.163 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 469 v 0 (ObjNum 468 snap 0 seq_num 468) dirty exists 2024-09-21T18:08:01.165 INFO:tasks.rados.rados.0.smithi160.stdout:932: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-21T18:08:01.165 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:01.174 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.174 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.174 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:01.173+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:01.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:01.235 INFO:tasks.rados.rados.0.smithi160.stdout:919: got (0) Success 2024-09-21T18:08:01.236 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 0 (ObjNum 494 snap 0 seq_num 494) dirty exists 2024-09-21T18:08:01.236 INFO:tasks.rados.rados.0.smithi160.stdout:916: expect (ObjNum 566 snap 0 seq_num 566) 2024-09-21T18:08:01.275 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:01.270+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:01.288 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 681914 2024-09-21T18:08:01.349 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.349 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.364 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:01.376 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:01.373+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:01.378 INFO:tasks.rados.rados.0.smithi160.stdout:920: expect (ObjNum 87 snap 0 seq_num 87) 2024-09-21T18:08:01.467 INFO:tasks.rados.rados.0.smithi160.stdout:933: got (0) Success 2024-09-21T18:08:01.467 INFO:tasks.rados.rados.0.smithi160.stdout:917: finishing write tid 1 to smithi16030472-442 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:916: done (13 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:918: done (12 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:919: done (11 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:920: done (10 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:924: done (9 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:926: done (8 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:930: done (7 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:931: done (6 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:932: done (5 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:933: done (4 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:934: done (3 left) 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:935: write oid 162 current snap is 0 2024-09-21T18:08:01.468 INFO:tasks.rados.rados.0.smithi160.stdout:935: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-21T18:08:01.474 INFO:tasks.rados.rados.0.smithi160.stdout:935: writing smithi16030472-162 from 655638 to 1220803 tid 1 2024-09-21T18:08:01.477 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:01.473+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:01.480 INFO:tasks.rados.rados.0.smithi160.stdout:935: writing smithi16030472-162 from 1900172 to 2321714 tid 2 2024-09-21T18:08:01.480 INFO:tasks.rados.rados.0.smithi160.stdout:936: copy_from oid 235 from oid 78 current snap is 0 2024-09-21T18:08:01.480 INFO:tasks.rados.rados.0.smithi160.stdout:917: finishing write tid 2 to smithi16030472-442 2024-09-21T18:08:01.480 INFO:tasks.rados.rados.0.smithi160.stdout:917: finishing write tid 3 to smithi16030472-442 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:917: finishing write tid 4 to smithi16030472-442 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:917: oid 442 updating version 0 to 467 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:917: oid 442 version 467 is already newer than 466 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:917: oid 442 updating version 467 to 468 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 468 (ObjNum 629 snap 0 seq_num 629) dirty exists 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:917: left oid 442 (ObjNum 629 snap 0 seq_num 629) 2024-09-21T18:08:01.481 INFO:tasks.rados.rados.0.smithi160.stdout:925: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:08:01.494 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.495 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.7.7 2024-09-21T18:08:01.510 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:01.557 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:01.578 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:01.575+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:01.603 INFO:tasks.rados.rados.0.smithi160.stdout:917: done (4 left) 2024-09-21T18:08:01.603 INFO:tasks.rados.rados.0.smithi160.stdout:925: done (3 left) 2024-09-21T18:08:01.603 INFO:tasks.rados.rados.0.smithi160.stdout:937: delete oid 92 current snap is 0 2024-09-21T18:08:01.603 INFO:tasks.rados.rados.0.smithi160.stdout:935: finishing write tid 1 to smithi16030472-162 2024-09-21T18:08:01.633 INFO:tasks.rados.rados.0.smithi160.stdout:922: got (0) Success 2024-09-21T18:08:01.634 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 353 v 0 (ObjNum 352 snap 0 seq_num 352) dirty exists 2024-09-21T18:08:01.650 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:01.645+0000 7f440780b640 -1 osd.13 64 handle_osd_map injecting map crc failure 2024-09-21T18:08:01.663 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.663 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.676 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:01.718 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:01.714+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:01.780 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:01.775+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:01.789 INFO:tasks.rados.rados.0.smithi160.stdout:936: finishing copy_from racing read to smithi16030472-235 2024-09-21T18:08:01.789 INFO:tasks.rados.rados.0.smithi160.stdout:936: finishing copy_from to smithi16030472-235 2024-09-21T18:08:01.789 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 235 v 53 (ObjNum 77 snap 0 seq_num 77) dirty exists 2024-09-21T18:08:01.791 INFO:tasks.rados.rados.0.smithi160.stdout:935: finishing write tid 2 to smithi16030472-162 2024-09-21T18:08:01.793 INFO:tasks.rados.rados.0.smithi160.stdout:935: finishing write tid 3 to smithi16030472-162 2024-09-21T18:08:01.793 INFO:tasks.rados.rados.0.smithi160.stdout:935: finishing write tid 4 to smithi16030472-162 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:935: oid 162 updating version 0 to 468 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:935: oid 162 version 468 is already newer than 467 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:935: oid 162 updating version 468 to 469 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 162 v 469 (ObjNum 634 snap 0 seq_num 634) dirty exists 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:935: left oid 162 (ObjNum 634 snap 0 seq_num 634) 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:922: done (3 left) 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:935: done (2 left) 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:936: done (1 left) 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:937: done (0 left) 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:938: write oid 92 current snap is 0 2024-09-21T18:08:01.794 INFO:tasks.rados.rados.0.smithi160.stdout:938: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-21T18:08:01.804 INFO:tasks.rados.rados.0.smithi160.stdout:938: writing smithi16030472-92 from 689252 to 1404915 tid 1 2024-09-21T18:08:01.812 INFO:tasks.rados.rados.0.smithi160.stdout:938: writing smithi16030472-92 from 1900783 to 2325994 tid 2 2024-09-21T18:08:01.812 INFO:tasks.rados.rados.0.smithi160.stdout:939: delete oid 456 current snap is 0 2024-09-21T18:08:01.838 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.838 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.852 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:01.881 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:01.877+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:01.916 INFO:tasks.rados.rados.0.smithi160.stdout:939: done (1 left) 2024-09-21T18:08:01.916 INFO:tasks.rados.rados.0.smithi160.stdout:940: cache_flush oid 191 snap -1 2024-09-21T18:08:01.916 INFO:tasks.rados.rados.0.smithi160.stdout:941: cache_try_flush oid 119 snap -1 2024-09-21T18:08:01.916 INFO:tasks.rados.rados.0.smithi160.stdout:942: delete oid 471 current snap is 0 2024-09-21T18:08:01.916 INFO:tasks.rados.rados.0.smithi160.stdout:940: got (2) No such file or directory 2024-09-21T18:08:01.918 INFO:tasks.rados.rados.0.smithi160.stdout:938: finishing write tid 1 to smithi16030472-92 2024-09-21T18:08:01.918 INFO:tasks.rados.rados.0.smithi160.stdout:938: finishing write tid 2 to smithi16030472-92 2024-09-21T18:08:01.918 INFO:tasks.rados.rados.0.smithi160.stdout:938: finishing write tid 3 to smithi16030472-92 2024-09-21T18:08:01.923 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:01.918+0000 7f6a3cca9640 -1 osd.3 66 handle_osd_map injecting map crc failure 2024-09-21T18:08:01.973 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:01.973 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:01.981 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:01.975+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:01.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:938: finishing write tid 4 to smithi16030472-92 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:938: oid 92 updating version 0 to 473 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:938: oid 92 version 473 is already newer than 472 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:938: oid 92 version 473 is already newer than 471 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 92 v 473 (ObjNum 636 snap 0 seq_num 636) dirty exists 2024-09-21T18:08:02.019 INFO:tasks.rados.rados.0.smithi160.stdout:938: left oid 92 (ObjNum 636 snap 0 seq_num 636) 2024-09-21T18:08:02.082 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:02.077+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:02.105 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:02.105 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:02.120 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:938: done (3 left) 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:940: done (2 left) 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:942: done (1 left) 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:943: cache_flush oid 178 snap -1 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:944: cache_flush oid 310 snap -1 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:945: write oid 142 current snap is 0 2024-09-21T18:08:02.124 INFO:tasks.rados.rados.0.smithi160.stdout:945: seq_num 639 ranges {739673=657763,2076229=663444} 2024-09-21T18:08:02.135 INFO:tasks.rados.rados.0.smithi160.stdout:945: writing smithi16030472-142 from 739673 to 1397436 tid 1 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:945: writing smithi16030472-142 from 2076229 to 2739673 tid 2 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:946: cache_try_flush oid 473 snap -1 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:943: got (2) No such file or directory 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:943: done (4 left) 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:947: copy_from oid 432 from oid 31 current snap is 0 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:948: write oid 292 current snap is 0 2024-09-21T18:08:02.146 INFO:tasks.rados.rados.0.smithi160.stdout:948: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-21T18:08:02.157 INFO:tasks.rados.rados.0.smithi160.stdout:948: writing smithi16030472-292 from 756480 to 1489492 tid 1 2024-09-21T18:08:02.166 INFO:tasks.rados.rados.0.smithi160.stdout:948: writing smithi16030472-292 from 2134711 to 2567260 tid 2 2024-09-21T18:08:02.166 INFO:tasks.rados.rados.0.smithi160.stdout:949: read oid 469 snap -1 2024-09-21T18:08:02.166 INFO:tasks.rados.rados.0.smithi160.stdout:949: expect (ObjNum 468 snap 0 seq_num 468) 2024-09-21T18:08:02.166 INFO:tasks.rados.rados.0.smithi160.stdout:950: write oid 132 current snap is 0 2024-09-21T18:08:02.166 INFO:tasks.rados.rados.0.smithi160.stdout:950: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-21T18:08:02.174 INFO:tasks.rados.rados.0.smithi160.stdout:950: writing smithi16030472-132 from 773287 to 1181548 tid 1 2024-09-21T18:08:02.183 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:02.181+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:02.184 INFO:tasks.rados.rados.0.smithi160.stdout:950: writing smithi16030472-132 from 1793193 to 2369400 tid 2 2024-09-21T18:08:02.184 INFO:tasks.rados.rados.0.smithi160.stdout:951: cache_try_flush oid 277 snap -1 2024-09-21T18:08:02.184 INFO:tasks.rados.rados.0.smithi160.stdout:952: read oid 28 snap -1 2024-09-21T18:08:02.184 INFO:tasks.rados.rados.0.smithi160.stdout:952: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:08:02.184 INFO:tasks.rados.rados.0.smithi160.stdout:953: copy_from oid 472 from oid 120 current snap is 0 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:954: cache_evict oid 73 snap -1 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:955: read oid 154 snap -1 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:955: expect (ObjNum 153 snap 0 seq_num 153) 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:956: read oid 458 snap -1 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:956: expect deleted 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:957: write oid 82 current snap is 0 2024-09-21T18:08:02.185 INFO:tasks.rados.rados.0.smithi160.stdout:957: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-21T18:08:02.193 INFO:tasks.rados.rados.0.smithi160.stdout:957: writing smithi16030472-82 from 790094 to 1273604 tid 1 2024-09-21T18:08:02.199 INFO:tasks.rados.rados.0.smithi160.stdout:957: writing smithi16030472-82 from 1851675 to 2287893 tid 2 2024-09-21T18:08:02.199 INFO:tasks.rados.rados.0.smithi160.stdout:958: cache_flush oid 275 snap -1 2024-09-21T18:08:02.199 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:02.248 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:02.249 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:02.263 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:02.306 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:02.302+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:02.385 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:02.381+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:02.428 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:02.428 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:02.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:02.468 INFO:tasks.rados.rados.0.smithi160.stdout:948: finishing write tid 1 to smithi16030472-292 2024-09-21T18:08:02.469 INFO:tasks.rados.rados.0.smithi160.stdout:944: got (0) Success 2024-09-21T18:08:02.469 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 0 (ObjNum 129 snap 0 seq_num 3145242376) clean dne 2024-09-21T18:08:02.469 INFO:tasks.rados.rados.0.smithi160.stdout:944: done (15 left) 2024-09-21T18:08:02.469 INFO:tasks.rados.rados.0.smithi160.stdout:959: write (excl) oid 470 current snap is 0 2024-09-21T18:08:02.469 INFO:tasks.rados.rados.0.smithi160.stdout:959: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-21T18:08:02.474 INFO:tasks.rados.rados.0.smithi160.stdout:959: writing smithi16030472-470 from 406901 to 965660 tid 1 2024-09-21T18:08:02.480 INFO:tasks.rados.rados.0.smithi160.stdout:959: writing smithi16030472-470 from 1393804 to 1973680 tid 2 2024-09-21T18:08:02.486 INFO:tasks.rados.rados.0.smithi160.stdout:959: writing smithi16030472-470 from 2608396 to 2806901 tid 3 2024-09-21T18:08:02.486 INFO:tasks.rados.rados.0.smithi160.stdout:949: expect (ObjNum 468 snap 0 seq_num 468) 2024-09-21T18:08:02.491 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:02.487+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:02.530 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:02.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:08:02.581 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:02.581 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:02.587 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:02.583+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:02.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:02.613 INFO:tasks.rados.rados.0.smithi160.stdout:948: finishing write tid 2 to smithi16030472-292 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: finishing write tid 3 to smithi16030472-292 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: finishing write tid 4 to smithi16030472-292 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: oid 292 updating version 0 to 475 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: oid 292 updating version 475 to 477 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: oid 292 version 477 is already newer than 476 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 292 v 477 (ObjNum 640 snap 0 seq_num 640) dirty exists 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: left oid 292 (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:954: got (16) Device or resource busy 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:948: done (15 left) 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:949: done (14 left) 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:954: done (13 left) 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:960: cache_flush oid 59 snap -1 2024-09-21T18:08:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:941: got (0) Success 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 119 v 0 (ObjNum 534 snap 0 seq_num 534) dirty exists 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:946: got (0) Success 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 473 v 0 (ObjNum 472 snap 0 seq_num 472) dirty exists 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:951: got (2) No such file or directory 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: finishing write tid 1 to smithi16030472-132 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: finishing write tid 2 to smithi16030472-132 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: finishing write tid 3 to smithi16030472-132 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: finishing write tid 4 to smithi16030472-132 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: oid 132 updating version 0 to 461 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: oid 132 version 461 is already newer than 459 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: oid 132 version 461 is already newer than 460 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 461 (ObjNum 641 snap 0 seq_num 641) dirty exists 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:950: left oid 132 (ObjNum 641 snap 0 seq_num 641) 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:952: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:08:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:02.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:941: done (13 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:946: done (12 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:950: done (11 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:951: done (10 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:952: done (9 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:956: done (8 left) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:961: cache_flush oid 423 snap -1 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:947: finishing copy_from to smithi16030472-432 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:947: got expected ENOENT (src dne) 2024-09-21T18:08:02.680 INFO:tasks.rados.rados.0.smithi160.stdout:947: finishing copy_from racing read to smithi16030472-432 2024-09-21T18:08:02.681 INFO:tasks.rados.rados.0.smithi160.stdout:958: got (2) No such file or directory 2024-09-21T18:08:02.681 INFO:tasks.rados.rados.0.smithi160.stdout:947: done (8 left) 2024-09-21T18:08:02.681 INFO:tasks.rados.rados.0.smithi160.stdout:958: done (7 left) 2024-09-21T18:08:02.681 INFO:tasks.rados.rados.0.smithi160.stdout:962: write (excl) oid 64 current snap is 0 2024-09-21T18:08:02.681 INFO:tasks.rados.rados.0.smithi160.stdout:962: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-21T18:08:02.685 INFO:tasks.rados.rados.0.smithi160.stdout:962: writing smithi16030472-64 from 423708 to 1057716 tid 1 2024-09-21T18:08:02.687 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:02.685+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:02.692 INFO:tasks.rados.rados.0.smithi160.stdout:962: writing smithi16030472-64 from 1852286 to 2292173 tid 2 2024-09-21T18:08:02.693 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:02.686+0000 7fb25ec0a640 -1 osd.11 67 handle_osd_map injecting map crc failure 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:962: writing smithi16030472-64 from 2752172 to 2823708 tid 3 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:963: read oid 451 snap -1 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:963: expect (ObjNum 603 snap 0 seq_num 603) 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:964: delete oid 378 current snap is 0 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:961: got (0) Success 2024-09-21T18:08:02.696 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 423 v 0 (ObjNum 422 snap 0 seq_num 422) dirty exists 2024-09-21T18:08:02.769 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:02.769 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:02.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:02.787 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:02.785+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:02.798 INFO:tasks.rados.rados.0.smithi160.stdout:957: finishing write tid 1 to smithi16030472-82 2024-09-21T18:08:02.798 INFO:tasks.rados.rados.0.smithi160.stdout:953: finishing copy_from to smithi16030472-472 2024-09-21T18:08:02.798 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 472 v 79 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-21T18:08:02.805 INFO:tasks.rados.rados.0.smithi160.stdout:961: done (9 left) 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:964: done (8 left) 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:965: copy_from oid 337 from oid 370 current snap is 0 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:966: cache_flush oid 214 snap -1 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:967: cache_flush oid 351 snap -1 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:968: copy_from oid 385 from oid 481 current snap is 0 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:969: write oid 307 current snap is 0 2024-09-21T18:08:02.806 INFO:tasks.rados.rados.0.smithi160.stdout:969: seq_num 646 ranges {457322=784506,1969250=443556} 2024-09-21T18:08:02.820 INFO:tasks.rados.rados.0.smithi160.stdout:969: writing smithi16030472-307 from 457322 to 1241828 tid 1 2024-09-21T18:08:02.832 INFO:tasks.rados.rados.0.smithi160.stdout:969: writing smithi16030472-307 from 1969250 to 2412806 tid 2 2024-09-21T18:08:02.832 INFO:tasks.rados.rados.0.smithi160.stdout:970: delete oid 300 current snap is 0 2024-09-21T18:08:02.832 INFO:tasks.rados.rados.0.smithi160.stdout:945: finishing write tid 1 to smithi16030472-142 2024-09-21T18:08:02.832 INFO:tasks.rados.rados.0.smithi160.stdout:945: finishing write tid 2 to smithi16030472-142 2024-09-21T18:08:02.833 INFO:tasks.rados.rados.0.smithi160.stdout:955: expect (ObjNum 153 snap 0 seq_num 153) 2024-09-21T18:08:02.839 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:02.839 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:02.859 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 79113 2024-09-21T18:08:02.860 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:02.860 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:02.880 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:02.881 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:02.888 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:02.883+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:02.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:02.911 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:02.911 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:02.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:02.911 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:02.911 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: finishing write tid 2 to smithi16030472-82 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:959: finishing write tid 1 to smithi16030472-470 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: finishing write tid 3 to smithi16030472-82 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: finishing write tid 4 to smithi16030472-82 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: oid 82 updating version 0 to 463 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: oid 82 version 463 is already newer than 462 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: oid 82 updating version 463 to 464 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 464 (ObjNum 642 snap 0 seq_num 642) dirty exists 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:957: left oid 82 (ObjNum 642 snap 0 seq_num 642) 2024-09-21T18:08:02.912 INFO:tasks.rados.rados.0.smithi160.stdout:953: finishing copy_from racing read to smithi16030472-472 2024-09-21T18:08:02.914 INFO:tasks.rados.rados.0.smithi160.stdout:969: finishing write tid 1 to smithi16030472-307 2024-09-21T18:08:02.920 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:02.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:02.990 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:02.989+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:03.006 INFO:tasks.rados.rados.0.smithi160.stdout:963: expect (ObjNum 603 snap 0 seq_num 603) 2024-09-21T18:08:03.010 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 745812 2024-09-21T18:08:03.036 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 672707 2024-09-21T18:08:03.042 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:03.042 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:03.065 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:03.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:03.091 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:03.087+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:03.095 INFO:tasks.rados.rados.0.smithi160.stdout:969: finishing write tid 2 to smithi16030472-307 2024-09-21T18:08:03.095 INFO:tasks.rados.rados.0.smithi160.stdout:969: finishing write tid 3 to smithi16030472-307 2024-09-21T18:08:03.095 INFO:tasks.rados.rados.0.smithi160.stdout:969: finishing write tid 4 to smithi16030472-307 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:969: oid 307 updating version 0 to 480 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:969: oid 307 version 480 is already newer than 479 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:969: oid 307 version 480 is already newer than 478 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 480 (ObjNum 646 snap 0 seq_num 646) dirty exists 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:969: left oid 307 (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:965: finishing copy_from racing read to smithi16030472-337 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: finishing write tid 3 to smithi16030472-142 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: finishing write tid 4 to smithi16030472-142 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: oid 142 updating version 0 to 392 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: oid 142 updating version 392 to 393 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: oid 142 updating version 393 to 398 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 398 (ObjNum 639 snap 0 seq_num 639) dirty exists 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:945: left oid 142 (ObjNum 639 snap 0 seq_num 639) 2024-09-21T18:08:03.096 INFO:tasks.rados.rados.0.smithi160.stdout:967: got (2) No such file or directory 2024-09-21T18:08:03.108 INFO:tasks.rados.rados.0.smithi160.stdout:959: finishing write tid 2 to smithi16030472-470 2024-09-21T18:08:03.111 INFO:tasks.rados.rados.0.smithi160.stdout:960: got (0) Success 2024-09-21T18:08:03.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 0 (ObjNum 58 snap 0 seq_num 58) dirty exists 2024-09-21T18:08:03.113 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:03.113 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:965: finishing copy_from to smithi16030472-337 2024-09-21T18:08:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 337 v 259 (ObjNum 369 snap 0 seq_num 369) dirty exists 2024-09-21T18:08:03.133 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:03.133 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:03.133 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:03.133 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:03.135 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:03.135 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:03.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:03.162 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:03.162 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:03.182 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:03.182 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:03.193 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:03.189+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:03.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:03.199 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:03.209 INFO:tasks.rados.rados.0.smithi160.stdout:962: finishing write tid 1 to smithi16030472-64 2024-09-21T18:08:03.209 INFO:tasks.rados.rados.0.smithi160.stdout:966: got (0) Success 2024-09-21T18:08:03.209 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 214 v 0 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-21T18:08:03.210 INFO:tasks.rados.rados.0.smithi160.stdout:962: finishing write tid 2 to smithi16030472-64 2024-09-21T18:08:03.210 INFO:tasks.rados.rados.0.smithi160.stdout:962: finishing write tid 3 to smithi16030472-64 2024-09-21T18:08:03.210 INFO:tasks.rados.rados.0.smithi160.stdout:962: finishing write tid 4 to smithi16030472-64 2024-09-21T18:08:03.210 INFO:tasks.rados.rados.0.smithi160.stdout:959: finishing write tid 3 to smithi16030472-470 2024-09-21T18:08:03.211 INFO:tasks.rados.rados.0.smithi160.stdout:959: finishing write tid 4 to smithi16030472-470 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: finishing write tid 5 to smithi16030472-470 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: oid 470 updating version 0 to 465 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: oid 470 updating version 465 to 467 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: oid 470 version 467 is already newer than 453 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: oid 470 version 467 is already newer than 466 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 470 v 467 (ObjNum 643 snap 0 seq_num 643) dirty exists 2024-09-21T18:08:03.212 INFO:tasks.rados.rados.0.smithi160.stdout:959: left oid 470 (ObjNum 643 snap 0 seq_num 643) 2024-09-21T18:08:03.221 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:03.221 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:03.222 INFO:tasks.rados.rados.0.smithi160.stdout:968: finishing copy_from racing read to smithi16030472-385 2024-09-21T18:08:03.224 INFO:tasks.rados.rados.0.smithi160.stdout:945: done (13 left) 2024-09-21T18:08:03.224 INFO:tasks.rados.rados.0.smithi160.stdout:953: done (12 left) 2024-09-21T18:08:03.224 INFO:tasks.rados.rados.0.smithi160.stdout:955: done (11 left) 2024-09-21T18:08:03.224 INFO:tasks.rados.rados.0.smithi160.stdout:957: done (10 left) 2024-09-21T18:08:03.224 INFO:tasks.rados.rados.0.smithi160.stdout:959: done (9 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:960: done (8 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:963: done (7 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:965: done (6 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:966: done (5 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:967: done (4 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:969: done (3 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:970: done (2 left) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:971: cache_flush oid 418 snap -1 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:972: cache_flush oid 56 snap -1 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:973: copy_from oid 354 from oid 288 current snap is 0 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:974: copy_from oid 59 from oid 172 current snap is 0 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:975: read oid 186 snap -1 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:975: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:08:03.225 INFO:tasks.rados.rados.0.smithi160.stdout:976: read oid 114 snap -1 2024-09-21T18:08:03.226 INFO:tasks.rados.rados.0.smithi160.stdout:976: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:08:03.226 INFO:tasks.rados.rados.0.smithi160.stdout:977: cache_flush oid 381 snap -1 2024-09-21T18:08:03.226 INFO:tasks.rados.rados.0.smithi160.stdout:978: delete oid 78 current snap is 0 2024-09-21T18:08:03.226 INFO:tasks.rados.rados.0.smithi160.stdout:971: got (2) No such file or directory 2024-09-21T18:08:03.230 INFO:tasks.rados.rados.0.smithi160.stdout:975: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:08:03.230 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:03.243 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:03.243 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:03.263 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:03.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:03.294 INFO:teuthology.orchestra.run.smithi160.stdout:get_pg_num_history pg_num_history pg_num_history(e61 pg_nums {1={23=1},2={25=8},3={31=4},4={33=4}} deleted_pools ) 2024-09-21T18:08:03.294 INFO:teuthology.orchestra.run.smithi160.stdout:Importing pgid 2.7 2024-09-21T18:08:03.295 INFO:teuthology.orchestra.run.smithi160.stdout:write_pg epoch 29 info 2.7( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/26/0 sis=25) 2024-09-21T18:08:03.296 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:03.296 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:03.296 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:03.296 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:03.296 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:03.296 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:03.291+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:03.300 INFO:teuthology.orchestra.run.smithi160.stdout:Import successful 2024-09-21T18:08:03.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:03.339 INFO:tasks.rados.rados.0.smithi160.stdout:977: got (0) Success 2024-09-21T18:08:03.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:08:03.356 INFO:tasks.rados.rados.0.smithi160.stdout:968: finishing copy_from to smithi16030472-385 2024-09-21T18:08:03.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 404 (ObjNum 480 snap 0 seq_num 480) dirty exists 2024-09-21T18:08:03.356 INFO:tasks.rados.rados.0.smithi160.stdout:962: finishing write tid 5 to smithi16030472-64 2024-09-21T18:08:03.356 INFO:tasks.rados.rados.0.smithi160.stdout:962: oid 64 updating version 0 to 473 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:962: oid 64 version 473 is already newer than 471 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:962: oid 64 version 473 is already newer than 472 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:962: oid 64 version 473 is already newer than 470 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 64 v 473 (ObjNum 644 snap 0 seq_num 644) dirty exists 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:962: left oid 64 (ObjNum 644 snap 0 seq_num 644) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:972: got (2) No such file or directory 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:962: done (9 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:968: done (8 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:971: done (7 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:972: done (6 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:975: done (5 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:977: done (4 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:978: done (3 left) 2024-09-21T18:08:03.357 INFO:tasks.rados.rados.0.smithi160.stdout:979: read oid 178 snap -1 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:979: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:980: copy_from oid 380 from oid 177 current snap is 0 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:981: cache_evict oid 119 snap -1 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:982: copy_from oid 431 from oid 65 current snap is 0 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:983: cache_evict oid 76 snap -1 2024-09-21T18:08:03.358 INFO:tasks.rados.rados.0.smithi160.stdout:984: delete oid 343 current snap is 0 2024-09-21T18:08:03.395 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:03.390+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:03.441 INFO:tasks.rados.rados.0.smithi160.stdout:974: finishing copy_from racing read to smithi16030472-59 2024-09-21T18:08:03.446 INFO:tasks.rados.rados.0.smithi160.stdout:984: done (8 left) 2024-09-21T18:08:03.446 INFO:tasks.rados.rados.0.smithi160.stdout:985: read oid 144 snap -1 2024-09-21T18:08:03.446 INFO:tasks.rados.rados.0.smithi160.stdout:985: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-21T18:08:03.446 INFO:tasks.rados.rados.0.smithi160.stdout:986: write (excl) oid 134 current snap is 0 2024-09-21T18:08:03.446 INFO:tasks.rados.rados.0.smithi160.stdout:986: seq_num 650 ranges {524550=401855,1403178=450894,2516710=407840} 2024-09-21T18:08:03.458 INFO:tasks.rados.rados.0.smithi160.stdout:986: writing smithi16030472-134 from 524550 to 926405 tid 1 2024-09-21T18:08:03.473 INFO:tasks.rados.rados.0.smithi160.stdout:986: writing smithi16030472-134 from 1403178 to 1854072 tid 2 2024-09-21T18:08:03.490 INFO:tasks.rados.rados.0.smithi160.stdout:986: writing smithi16030472-134 from 2516710 to 2924550 tid 3 2024-09-21T18:08:03.491 INFO:tasks.rados.rados.0.smithi160.stdout:987: read oid 5 snap -1 2024-09-21T18:08:03.491 INFO:tasks.rados.rados.0.smithi160.stdout:987: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:08:03.491 INFO:tasks.rados.rados.0.smithi160.stdout:988: write (excl) oid 129 current snap is 0 2024-09-21T18:08:03.491 INFO:tasks.rados.rados.0.smithi160.stdout:988: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.492 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 50.276471491999999, 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.498 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.732533943, 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:08:03.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:08:03.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.504 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.571610479, 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:08:03.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.570935884000001, 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.570909346000001, 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.561075604000003, 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.517 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:08:03.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.548663816000001, 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.521 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:08:03.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:08:03.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.527 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.539773746000002, 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.533 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.538961729, 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:08:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:08:03.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:08:03.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:08:03.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.538853517, 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.540 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.545 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.438582371000003, 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:03.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:08:03.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.437686112000002, 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:08:03.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:08:03.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:08:03.553 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.437204499000003, 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.554 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:08:03.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:08:03.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.436752073999997, 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:08:03.562 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.436701747999997, 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.565 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.566 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.310968967999997, 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.567 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.306690910999997, 2024-09-21T18:08:03.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.306320671000002, 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.580 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:08:03.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:08:03.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:08:03.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:08:03.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:08:03.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 32.306250941000002, 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.586 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:08:03.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 31.714673951000002, 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:03.591 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:03.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:08:03.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:03.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:03.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.597 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:03.597 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:03.597 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:03.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:988: writing smithi16030472-129 from 541357 to 1018461 tid 1 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:988: writing smithi16030472-129 from 1745307 to 2339859 tid 2 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:989: copy_from oid 117 from oid 269 current snap is 0 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:974: finishing copy_from to smithi16030472-59 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 133 (ObjNum 171 snap 0 seq_num 171) dirty exists 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:974: done (12 left) 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:990: read oid 165 snap -1 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:990: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:991: delete oid 16 current snap is 0 2024-09-21T18:08:03.600 INFO:tasks.rados.rados.0.smithi160.stdout:976: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:08:03.601 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:03.601 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:08:03.602 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:03.593+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:03.699 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:03.697+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:03.719 INFO:tasks.rados.rados.0.smithi160.stdout:981: got (0) Success 2024-09-21T18:08:03.719 INFO:tasks.rados.rados.0.smithi160.stdout:983: got (16) Device or resource busy 2024-09-21T18:08:03.719 INFO:tasks.rados.rados.0.smithi160.stdout:987: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:08:03.774 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:03.775 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:03.775 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:03.775 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:03.786 INFO:tasks.rados.rados.0.smithi160.stdout:990: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-21T18:08:03.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:03.793 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1297300 2024-09-21T18:08:03.800 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:03.794+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:03.837 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f /var/log/ceph/exp.2.7.7 2024-09-21T18:08:03.852 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:03.846+0000 7fb260c0e640 -1 osd.11 68 handle_osd_map injecting map crc failure 2024-09-21T18:08:03.854 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:03.846+0000 7fb25ec0a640 -1 osd.11 68 handle_osd_map injecting map crc failure 2024-09-21T18:08:03.864 INFO:tasks.thrashosds.thrasher:Waiting for clean again 2024-09-21T18:08:03.864 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T18:08:03.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:08:03.865 INFO:tasks.rados.rados.0.smithi160.stdout:988: finishing write tid 1 to smithi16030472-129 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:986: finishing write tid 1 to smithi16030472-134 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:976: done (13 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:981: done (12 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:983: done (11 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:987: done (10 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:990: done (9 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:991: done (8 left) 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:992: write (excl) oid 98 current snap is 0 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:986: finishing write tid 2 to smithi16030472-134 2024-09-21T18:08:03.866 INFO:tasks.rados.rados.0.smithi160.stdout:992: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-21T18:08:03.875 INFO:tasks.rados.rados.0.smithi160.stdout:992: writing smithi16030472-98 from 574971 to 1202573 tid 1 2024-09-21T18:08:03.886 INFO:tasks.rados.rados.0.smithi160.stdout:992: writing smithi16030472-98 from 1862271 to 2460492 tid 2 2024-09-21T18:08:03.886 INFO:tasks.rados.rados.0.smithi160.stdout:993: read oid 111 snap -1 2024-09-21T18:08:03.886 INFO:tasks.rados.rados.0.smithi160.stdout:993: expect deleted 2024-09-21T18:08:03.886 INFO:tasks.rados.rados.0.smithi160.stdout:994: write (excl) oid 408 current snap is 0 2024-09-21T18:08:03.886 INFO:tasks.rados.rados.0.smithi160.stdout:994: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-21T18:08:03.893 INFO:tasks.rados.rados.0.smithi160.stdout:994: writing smithi16030472-408 from 591778 to 1010982 tid 1 2024-09-21T18:08:03.901 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:03.897+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:994: writing smithi16030472-408 from 1637106 to 2378985 tid 2 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:994: writing smithi16030472-408 from 2910049 to 2991778 tid 3 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:995: cache_flush oid 186 snap -1 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:996: read oid 2 snap -1 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:996: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:997: delete oid 31 current snap is 0 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:986: finishing write tid 3 to smithi16030472-134 2024-09-21T18:08:03.909 INFO:tasks.rados.rados.0.smithi160.stdout:979: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:03.925 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:03.925 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:03.925 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:03.925 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:03.925 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:03.939 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:03.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:03.950 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:03.945+0000 7f440780b640 -1 osd.13 68 handle_osd_map injecting map crc failure 2024-09-21T18:08:04.002 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:04.001+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: finishing write tid 4 to smithi16030472-134 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: finishing write tid 5 to smithi16030472-134 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: oid 134 updating version 0 to 485 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: oid 134 version 485 is already newer than 484 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: oid 134 version 485 is already newer than 483 2024-09-21T18:08:04.017 INFO:tasks.rados.rados.0.smithi160.stdout:986: oid 134 updating version 485 to 486 2024-09-21T18:08:04.018 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 134 v 486 (ObjNum 650 snap 0 seq_num 650) dirty exists 2024-09-21T18:08:04.018 INFO:tasks.rados.rados.0.smithi160.stdout:986: left oid 134 (ObjNum 650 snap 0 seq_num 650) 2024-09-21T18:08:04.018 INFO:tasks.rados.rados.0.smithi160.stdout:985: expect (ObjNum 143 snap 0 seq_num 143) 2024-09-21T18:08:04.021 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 723216 2024-09-21T18:08:04.039 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.071 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:04.072 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:04.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:04.093 INFO:tasks.rados.rados.0.smithi160.stdout:992: finishing write tid 1 to smithi16030472-98 2024-09-21T18:08:04.093 INFO:tasks.rados.rados.0.smithi160.stdout:979: done (13 left) 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:985: done (12 left) 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:986: done (11 left) 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:997: done (10 left) 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:998: read oid 4 snap -1 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:998: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:999: cache_try_flush oid 25 snap -1 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:995: got (0) Success 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 186 v 0 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:973: finishing copy_from to smithi16030472-354 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 354 v 206 (ObjNum 287 snap 0 seq_num 287) dirty exists 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:989: finishing copy_from racing read to smithi16030472-117 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:982: finishing copy_from racing read to smithi16030472-431 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:982: finishing copy_from to smithi16030472-431 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 431 v 42 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:08:04.094 INFO:tasks.rados.rados.0.smithi160.stdout:973: finishing copy_from racing read to smithi16030472-354 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:994: finishing write tid 1 to smithi16030472-408 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:994: finishing write tid 2 to smithi16030472-408 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:973: done (11 left) 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:982: done (10 left) 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:995: done (9 left) 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:1000: write oid 356 current snap is 0 2024-09-21T18:08:04.095 INFO:tasks.rados.rados.0.smithi160.stdout:1000: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-21T18:08:04.100 INFO:tasks.rados.rados.0.smithi160.stdout:1000: writing smithi16030472-356 from 625392 to 1195094 tid 1 2024-09-21T18:08:04.103 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:04.101+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:04.106 INFO:tasks.rados.rados.0.smithi160.stdout:1000: writing smithi16030472-356 from 1637717 to 2383265 tid 2 2024-09-21T18:08:04.110 INFO:tasks.rados.rados.0.smithi160.stdout:1000: writing smithi16030472-356 from 2848542 to 3025392 tid 3 2024-09-21T18:08:04.110 INFO:tasks.rados.rados.0.smithi160.stdout:1001: write (excl) oid 73 current snap is 0 2024-09-21T18:08:04.110 INFO:tasks.rados.rados.0.smithi160.stdout:1001: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-21T18:08:04.117 INFO:tasks.rados.rados.0.smithi160.stdout:1001: writing smithi16030472-73 from 642199 to 1287150 tid 1 2024-09-21T18:08:04.123 INFO:tasks.rados.rados.0.smithi160.stdout:1001: writing smithi16030472-73 from 1696199 to 2301758 tid 2 2024-09-21T18:08:04.127 INFO:tasks.rados.rados.0.smithi160.stdout:1001: writing smithi16030472-73 from 2992318 to 3042199 tid 3 2024-09-21T18:08:04.127 INFO:tasks.rados.rados.0.smithi160.stdout:1002: delete oid 234 current snap is 0 2024-09-21T18:08:04.127 INFO:tasks.rados.rados.0.smithi160.stdout:992: finishing write tid 2 to smithi16030472-98 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:992: finishing write tid 3 to smithi16030472-98 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:980: finishing copy_from racing read to smithi16030472-380 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:980: finishing copy_from to smithi16030472-380 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 380 v 90 (ObjNum 176 snap 0 seq_num 176) dirty exists 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:996: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:08:04.128 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.173 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.184 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:08:04.184 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:08:04.195 INFO:tasks.rados.rados.0.smithi160.stdout:992: finishing write tid 4 to smithi16030472-98 2024-09-21T18:08:04.195 INFO:tasks.rados.rados.0.smithi160.stdout:992: oid 98 updating version 0 to 401 2024-09-21T18:08:04.195 INFO:tasks.rados.rados.0.smithi160.stdout:992: oid 98 updating version 401 to 402 2024-09-21T18:08:04.195 INFO:tasks.rados.rados.0.smithi160.stdout:992: oid 98 updating version 402 to 403 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 98 v 403 (ObjNum 653 snap 0 seq_num 653) dirty exists 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:992: left oid 98 (ObjNum 653 snap 0 seq_num 653) 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: finishing write tid 2 to smithi16030472-129 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: finishing write tid 3 to smithi16030472-129 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: finishing write tid 4 to smithi16030472-129 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: oid 129 updating version 0 to 487 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: oid 129 version 487 is already newer than 435 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: oid 129 updating version 487 to 488 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 488 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:988: left oid 129 (ObjNum 651 snap 0 seq_num 651) 2024-09-21T18:08:04.196 INFO:tasks.rados.rados.0.smithi160.stdout:1001: finishing write tid 1 to smithi16030472-73 2024-09-21T18:08:04.198 INFO:tasks.rados.rados.0.smithi160.stdout:998: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-21T18:08:04.198 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.203 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-21T18:08:03.544274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1954481593,"num_objects":739,"num_object_clones":0,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":27,"num_read":2277,"num_read_kb":709477,"num_write":2909,"num_write_kb":995058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":36,"num_flush_kb":94559,"num_evict":0,"num_evict_kb":0,"num_promote":271,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2603,"ondisk_log_size":2603,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":14,"kb":1415577600,"kb_used":2307248,"kb_used_data":1852272,"kb_used_omap":23,"kb_used_meta":446696,"kb_avail":1413270352,"statfs":{"total":1449551462400,"available":1447188840448,"internally_reserved":0,"allocated":1896726528,"data_stored":1938680922,"data_compressed":45785569,"data_compressed_allocated":48893952,"data_compressed_original":104235454,"omap_allocated":23930,"internal_metadata":457417350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4589088,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":7,"num_read":234,"num_read_kb":212831,"num_write":75,"num_write_kb":64695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":13,"num_flush_kb":32391,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037099"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":230,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963391+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:08:01.963391+0000","last_peered":"2024-09-21T18:08:01.963391+0000","last_clean":"2024-09-21T18:08:01.963391+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:08:01.963391+0000","last_undegraded":"2024-09-21T18:08:01.963391+0000","last_fullsized":"2024-09-21T18:08:01.963391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963734+0000","last_change":"2024-09-21T18:07:46.555026+0000","last_active":"2024-09-21T18:08:01.963734+0000","last_peered":"2024-09-21T18:08:01.963734+0000","last_clean":"2024-09-21T18:08:01.963734+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:08:01.963734+0000","last_undegraded":"2024-09-21T18:08:01.963734+0000","last_fullsized":"2024-09-21T18:08:01.963734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:07:46.554845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:07:46.554845+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:10.266490+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"66'367","reported_seq":700,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963497+0000","last_change":"2024-09-21T18:07:01.287376+0000","last_active":"2024-09-21T18:08:01.963497+0000","last_peered":"2024-09-21T18:08:01.963497+0000","last_clean":"2024-09-21T18:08:01.963497+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:08:01.963497+0000","last_undegraded":"2024-09-21T18:08:01.963497+0000","last_fullsized":"2024-09-21T18:08:01.963497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:21.792563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315181030,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":338,"num_read_kb":30796,"num_write":444,"num_write_kb":206438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.026772+0000","last_change":"2024-09-21T18:07:57.279542+0000","last_active":"2024-09-21T18:08:02.026772+0000","last_peered":"2024-09-21T18:08:02.026772+0000","last_clean":"2024-09-21T18:08:02.026772+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:08:02.026772+0000","last_undegraded":"2024-09-21T18:08:02.026772+0000","last_fullsized":"2024-09-21T18:08:02.026772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:57.279346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:07:57.279346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:20.133385+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"64'448","reported_seq":762,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.010634+0000","last_change":"2024-09-21T18:07:01.490023+0000","last_active":"2024-09-21T18:08:00.010634+0000","last_peered":"2024-09-21T18:08:00.010634+0000","last_clean":"2024-09-21T18:08:00.010634+0000","last_became_active":"2024-09-21T18:07:01.389573+0000","last_became_peered":"2024-09-21T18:07:01.389573+0000","last_unstale":"2024-09-21T18:08:00.010634+0000","last_undegraded":"2024-09-21T18:08:00.010634+0000","last_fullsized":"2024-09-21T18:08:00.010634+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:03.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347212733,"num_objects":121,"num_object_clones":0,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":385,"num_read_kb":114328,"num_write":558,"num_write_kb":189120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.717655+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:08:02.717655+0000","last_peered":"2024-09-21T18:08:02.717655+0000","last_clean":"2024-09-21T18:08:02.717655+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:08:02.717655+0000","last_undegraded":"2024-09-21T18:08:02.717655+0000","last_fullsized":"2024-09-21T18:08:02.717655+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"68'362","reported_seq":724,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.917006+0000","last_change":"2024-09-21T18:07:01.405525+0000","last_active":"2024-09-21T18:08:02.917006+0000","last_peered":"2024-09-21T18:08:02.917006+0000","last_clean":"2024-09-21T18:08:02.917006+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:08:02.917006+0000","last_undegraded":"2024-09-21T18:08:02.917006+0000","last_fullsized":"2024-09-21T18:08:02.917006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.560432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280975380,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":387,"num_read_kb":79011,"num_write":438,"num_write_kb":180638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":99,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.926458+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:08:01.926458+0000","last_peered":"2024-09-21T18:08:01.926458+0000","last_clean":"2024-09-21T18:08:01.926458+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:08:01.926458+0000","last_undegraded":"2024-09-21T18:08:01.926458+0000","last_fullsized":"2024-09-21T18:08:01.926458+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"68'478","reported_seq":831,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.911781+0000","last_change":"2024-09-21T18:07:01.288102+0000","last_active":"2024-09-21T18:08:02.911781+0000","last_peered":"2024-09-21T18:08:02.911781+0000","last_clean":"2024-09-21T18:08:02.911781+0000","last_became_active":"2024-09-21T18:07:01.287689+0000","last_became_peered":"2024-09-21T18:07:01.287689+0000","last_unstale":"2024-09-21T18:08:02.911781+0000","last_undegraded":"2024-09-21T18:08:02.911781+0000","last_fullsized":"2024-09-21T18:08:02.911781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.216721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307813077,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":439,"num_read_kb":97233,"num_write":594,"num_write_kb":224612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"68'270","reported_seq":508,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:03.021676+0000","last_change":"2024-09-21T18:07:03.553363+0000","last_active":"2024-09-21T18:08:03.021676+0000","last_peered":"2024-09-21T18:08:03.021676+0000","last_clean":"2024-09-21T18:08:03.021676+0000","last_became_active":"2024-09-21T18:07:03.552723+0000","last_became_peered":"2024-09-21T18:07:03.552723+0000","last_unstale":"2024-09-21T18:08:03.021676+0000","last_undegraded":"2024-09-21T18:08:03.021676+0000","last_fullsized":"2024-09-21T18:08:03.021676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:07.591572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227485220,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":7,"num_read":167,"num_read_kb":87948,"num_write":243,"num_write_kb":52575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":14,"num_flush_kb":42337,"num_evict":0,"num_evict_kb":0,"num_promote":83,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:07:59.533634+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:07:59.533634+0000","last_peered":"2024-09-21T18:07:59.533634+0000","last_clean":"2024-09-21T18:07:59.533634+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:07:59.533634+0000","last_undegraded":"2024-09-21T18:07:59.533634+0000","last_fullsized":"2024-09-21T18:07:59.533634+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'150","reported_seq":330,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963503+0000","last_change":"2024-09-21T18:07:04.454454+0000","last_active":"2024-09-21T18:08:01.963503+0000","last_peered":"2024-09-21T18:08:01.963503+0000","last_clean":"2024-09-21T18:08:01.963503+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:08:01.963503+0000","last_undegraded":"2024-09-21T18:08:01.963503+0000","last_fullsized":"2024-09-21T18:08:01.963503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.533440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114456693,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":81,"num_read_kb":44224,"num_write":127,"num_write_kb":39697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":9,"num_flush_kb":22934,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.528930+0000","last_change":"2024-09-21T18:07:51.660167+0000","last_active":"2024-09-21T18:08:01.528930+0000","last_peered":"2024-09-21T18:08:01.528930+0000","last_clean":"2024-09-21T18:08:01.528930+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:08:01.528930+0000","last_undegraded":"2024-09-21T18:08:01.528930+0000","last_fullsized":"2024-09-21T18:08:01.528930+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.473319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"65'281","reported_seq":551,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.110380+0000","last_change":"2024-09-21T18:07:03.453942+0000","last_active":"2024-09-21T18:08:00.110380+0000","last_peered":"2024-09-21T18:08:00.110380+0000","last_clean":"2024-09-21T18:08:00.110380+0000","last_became_active":"2024-09-21T18:07:03.453360+0000","last_became_peered":"2024-09-21T18:07:03.453360+0000","last_unstale":"2024-09-21T18:08:00.110380+0000","last_undegraded":"2024-09-21T18:08:00.110380+0000","last_fullsized":"2024-09-21T18:08:00.110380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:14.477508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237233709,"num_objects":90,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":10,"num_read":287,"num_read_kb":197481,"num_write":245,"num_write_kb":66694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":9,"num_flush_kb":21210,"num_evict":0,"num_evict_kb":0,"num_promote":94,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.401336+0000","last_change":"2024-09-21T18:08:01.296359+0000","last_active":"2024-09-21T18:08:01.401336+0000","last_peered":"2024-09-21T18:08:01.401336+0000","last_clean":"2024-09-21T18:08:01.401336+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:08:01.401336+0000","last_undegraded":"2024-09-21T18:08:01.401336+0000","last_fullsized":"2024-09-21T18:08:01.401336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:08:01.296093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:08:01.296093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:24.736120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"67'132","reported_seq":307,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.185668+0000","last_change":"2024-09-21T18:07:04.220636+0000","last_active":"2024-09-21T18:08:02.185668+0000","last_peered":"2024-09-21T18:08:02.185668+0000","last_clean":"2024-09-21T18:08:02.185668+0000","last_became_active":"2024-09-21T18:07:04.220350+0000","last_became_peered":"2024-09-21T18:07:04.220350+0000","last_unstale":"2024-09-21T18:08:02.185668+0000","last_undegraded":"2024-09-21T18:08:02.185668+0000","last_fullsized":"2024-09-21T18:08:02.185668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.012491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123533364,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":4,"num_read":107,"num_read_kb":58383,"num_write":113,"num_write_kb":32610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":4,"num_flush_kb":8078,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.318675+0000","last_change":"2024-09-21T18:07:58.908355+0000","last_active":"2024-09-21T18:08:00.318675+0000","last_peered":"2024-09-21T18:08:00.318675+0000","last_clean":"2024-09-21T18:08:00.318675+0000","last_became_active":"2024-09-21T18:07:58.907761+0000","last_became_peered":"2024-09-21T18:07:58.907761+0000","last_unstale":"2024-09-21T18:08:00.318675+0000","last_undegraded":"2024-09-21T18:08:00.318675+0000","last_fullsized":"2024-09-21T18:08:00.318675+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":702708986,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":27,"num_read":642,"num_read_kb":388036,"num_write":728,"num_write_kb":191576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":36,"num_flush_kb":94559,"num_evict":0,"num_evict_kb":0,"num_promote":271,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":687771648,"data_stored":725121357,"data_compressed":34673255,"data_compressed_allocated":36937728,"data_compressed_original":76431806,"omap_allocated":0,"internal_metadata":0},"log_size":833,"ondisk_log_size":833,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1251182220,"num_objects":475,"num_object_clones":0,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":1549,"num_read_kb":321368,"num_write":2034,"num_write_kb":800808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1200644096,"data_stored":1208141401,"data_compressed":11316120,"data_compressed_allocated":12161024,"data_compressed_original":27435008,"omap_allocated":0,"internal_metadata":0},"log_size":1655,"ondisk_log_size":1655,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286408,"kb_used_data":255216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94085432,"statfs":{"total":96636764160,"available":96343482368,"internally_reserved":0,"allocated":261341184,"data_stored":268004840,"data_compressed":6061803,"data_compressed_allocated":6504448,"data_compressed_original":14765821,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":622592,"data_stored":376827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161216,"kb_used_data":130024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94210624,"statfs":{"total":96636764160,"available":96471678976,"internally_reserved":0,"allocated":133144576,"data_stored":133909299,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313235,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307456,"kb_used_data":272028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94064384,"statfs":{"total":96636764160,"available":96321929216,"internally_reserved":0,"allocated":278556672,"data_stored":283695852,"data_compressed":6516153,"data_compressed_allocated":6938624,"data_compressed_original":13816545,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3,5,6,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313236,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":552960,"data_stored":354982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":610304,"data_stored":374308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246552,"kb_used_data":211200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94125288,"statfs":{"total":96636764160,"available":96384294912,"internally_reserved":0,"allocated":216268800,"data_stored":220951350,"data_compressed":4959701,"data_compressed_allocated":5312512,"data_compressed_original":11364647,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,5,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27580,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":565248,"data_stored":366585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232272,"kb_used_data":203656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94139568,"statfs":{"total":96636764160,"available":96398917632,"internally_reserved":0,"allocated":208543744,"data_stored":213558675,"data_compressed":4968709,"data_compressed_allocated":5300224,"data_compressed_original":11634138,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":581632,"data_stored":358775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159496,"kb_used_data":132468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94212344,"statfs":{"total":96636764160,"available":96473440256,"internally_reserved":0,"allocated":135647232,"data_stored":136434782,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257440,"kb_used_data":218120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94114400,"statfs":{"total":96636764160,"available":96373145600,"internally_reserved":0,"allocated":223354880,"data_stored":235067795,"data_compressed":10938690,"data_compressed_allocated":11669504,"data_compressed_original":24351198,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299108,"kb_used_data":266012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94072732,"statfs":{"total":96636764160,"available":96330477568,"internally_reserved":0,"allocated":272396288,"data_stored":281708610,"data_compressed":8774558,"data_compressed_allocated":9359360,"data_compressed_original":20143873,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":593920,"data_stored":370663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187136,"kb_used_data":160104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94184704,"statfs":{"total":96636764160,"available":96445136896,"internally_reserved":0,"allocated":163946496,"data_stored":163147579,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132505600,"data_stored":133521903,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151875584,"data_stored":152916976,"data_compressed":1321392,"data_compressed_allocated":1437696,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163336192,"data_stored":162772897,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":151601152,"data_stored":153833833,"data_compressed":2279103,"data_compressed_allocated":2445312,"data_compressed_original":5570560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":135069696,"data_stored":136066011,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":152190976,"data_stored":154425565,"data_compressed":2279103,"data_compressed_allocated":2445312,"data_compressed_original":5570560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163336192,"data_stored":162772897,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":150728704,"data_stored":151831319,"data_compressed":1402002,"data_compressed_allocated":1523712,"data_compressed_original":3575808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63942656,"data_stored":67802669,"data_compressed":3601366,"data_compressed_allocated":3833856,"data_compressed_original":7854375,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":113283072,"data_stored":119198900,"data_compressed":6047588,"data_compressed_allocated":6438912,"data_compressed_original":12702433,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":56340480,"data_stored":59311565,"data_compressed":2824359,"data_compressed_allocated":2998272,"data_compressed_original":6141402,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":222490624,"data_stored":234184554,"data_compressed":10802465,"data_compressed_allocated":11522048,"data_compressed_original":23957982,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":120283136,"data_stored":127114234,"data_compressed":6425725,"data_compressed_allocated":6832128,"data_compressed_original":13995777,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":111431680,"data_stored":117509435,"data_compressed":4971752,"data_compressed_allocated":5312512,"data_compressed_original":11779837,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:08:04.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:08:04.205 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:04.198+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:04.209 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:04.209 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:04.209 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:04.209 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:04.219 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:04.243 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1559368 2024-09-21T18:08:04.254 INFO:tasks.rados.rados.0.smithi160.stdout:1001: finishing write tid 2 to smithi16030472-73 2024-09-21T18:08:04.315 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:04.311+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:980: done (11 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:988: done (10 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:992: done (9 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:993: done (8 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:996: done (7 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:998: done (6 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:1002: done (5 left) 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:1003: write (excl) oid 164 current snap is 0 2024-09-21T18:08:04.332 INFO:tasks.rados.rados.0.smithi160.stdout:1003: seq_num 659 ranges {675813=795449,2096810=609228} 2024-09-21T18:08:04.344 INFO:tasks.rados.rados.0.smithi160.stdout:1003: writing smithi16030472-164 from 675813 to 1471262 tid 1 2024-09-21T18:08:04.354 INFO:tasks.rados.rados.0.smithi160.stdout:1003: writing smithi16030472-164 from 2096810 to 2706038 tid 2 2024-09-21T18:08:04.354 INFO:tasks.rados.rados.0.smithi160.stdout:1004: read oid 143 snap -1 2024-09-21T18:08:04.354 INFO:tasks.rados.rados.0.smithi160.stdout:1004: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:08:04.354 INFO:tasks.rados.rados.0.smithi160.stdout:1005: delete oid 345 current snap is 0 2024-09-21T18:08:04.389 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:04.389 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:04.389 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:04.389 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:04.390 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:04.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:04.405 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:04.402+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:04.429 INFO:tasks.rados.rados.0.smithi160.stdout:1001: finishing write tid 3 to smithi16030472-73 2024-09-21T18:08:04.497 INFO:tasks.rados.rados.0.smithi160.stdout:999: got (2) No such file or directory 2024-09-21T18:08:04.497 INFO:tasks.rados.rados.0.smithi160.stdout:1000: finishing write tid 1 to smithi16030472-356 2024-09-21T18:08:04.506 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:04.502+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:04.518 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:08:04.518 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:08:04.529 INFO:tasks.rados.rados.0.smithi160.stdout:1001: finishing write tid 4 to smithi16030472-73 2024-09-21T18:08:04.538 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-21T18:08:03.544274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1954481593,"num_objects":739,"num_object_clones":0,"num_object_copies":1478,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":630,"num_whiteouts":27,"num_read":2277,"num_read_kb":709477,"num_write":2909,"num_write_kb":995058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":36,"num_flush_kb":94559,"num_evict":0,"num_evict_kb":0,"num_promote":271,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2603,"ondisk_log_size":2603,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":33,"num_osds":15,"num_per_pool_osds":15,"num_per_pool_omap_osds":14,"kb":1415577600,"kb_used":2307248,"kb_used_data":1852272,"kb_used_omap":23,"kb_used_meta":446696,"kb_avail":1413270352,"statfs":{"total":1449551462400,"available":1447188840448,"internally_reserved":0,"allocated":1896726528,"data_stored":1938680922,"data_compressed":45785569,"data_compressed_allocated":48893952,"data_compressed_original":104235454,"omap_allocated":23930,"internal_metadata":457417350},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4589088,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":7,"num_read":234,"num_read_kb":212831,"num_write":75,"num_write_kb":64695,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":13,"num_flush_kb":32391,"num_evict":0,"num_evict_kb":0,"num_promote":15,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.037099"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":230,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963391+0000","last_change":"2024-09-21T18:06:42.957061+0000","last_active":"2024-09-21T18:08:01.963391+0000","last_peered":"2024-09-21T18:08:01.963391+0000","last_clean":"2024-09-21T18:08:01.963391+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:08:01.963391+0000","last_undegraded":"2024-09-21T18:08:01.963391+0000","last_fullsized":"2024-09-21T18:08:01.963391+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:06:41.831877+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:10.243064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":103,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963734+0000","last_change":"2024-09-21T18:07:46.555026+0000","last_active":"2024-09-21T18:08:01.963734+0000","last_peered":"2024-09-21T18:08:01.963734+0000","last_clean":"2024-09-21T18:08:01.963734+0000","last_became_active":"2024-09-21T18:06:44.874190+0000","last_became_peered":"2024-09-21T18:06:44.874190+0000","last_unstale":"2024-09-21T18:08:01.963734+0000","last_undegraded":"2024-09-21T18:08:01.963734+0000","last_fullsized":"2024-09-21T18:08:01.963734+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:07:46.554845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:07:46.554845+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:10.266490+0000","scrub_duration":303,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"66'367","reported_seq":700,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963497+0000","last_change":"2024-09-21T18:07:01.287376+0000","last_active":"2024-09-21T18:08:01.963497+0000","last_peered":"2024-09-21T18:08:01.963497+0000","last_clean":"2024-09-21T18:08:01.963497+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:08:01.963497+0000","last_undegraded":"2024-09-21T18:08:01.963497+0000","last_fullsized":"2024-09-21T18:08:01.963497+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:21.792563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315181030,"num_objects":123,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":123,"num_whiteouts":0,"num_read":338,"num_read_kb":30796,"num_write":444,"num_write_kb":206438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.026772+0000","last_change":"2024-09-21T18:07:57.279542+0000","last_active":"2024-09-21T18:08:02.026772+0000","last_peered":"2024-09-21T18:08:02.026772+0000","last_clean":"2024-09-21T18:08:02.026772+0000","last_became_active":"2024-09-21T18:06:45.016813+0000","last_became_peered":"2024-09-21T18:06:45.016813+0000","last_unstale":"2024-09-21T18:08:02.026772+0000","last_undegraded":"2024-09-21T18:08:02.026772+0000","last_fullsized":"2024-09-21T18:08:02.026772+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:57.279346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:07:57.279346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:20.133385+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00089119199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"64'448","reported_seq":762,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.010634+0000","last_change":"2024-09-21T18:07:01.490023+0000","last_active":"2024-09-21T18:08:00.010634+0000","last_peered":"2024-09-21T18:08:00.010634+0000","last_clean":"2024-09-21T18:08:00.010634+0000","last_became_active":"2024-09-21T18:07:01.389573+0000","last_became_peered":"2024-09-21T18:07:01.389573+0000","last_unstale":"2024-09-21T18:08:00.010634+0000","last_undegraded":"2024-09-21T18:08:00.010634+0000","last_fullsized":"2024-09-21T18:08:00.010634+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:03.870831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":347212733,"num_objects":121,"num_object_clones":0,"num_object_copies":242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":121,"num_whiteouts":0,"num_read":385,"num_read_kb":114328,"num_write":558,"num_write_kb":189120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.717655+0000","last_change":"2024-09-21T18:06:47.988658+0000","last_active":"2024-09-21T18:08:02.717655+0000","last_peered":"2024-09-21T18:08:02.717655+0000","last_clean":"2024-09-21T18:08:02.717655+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:08:02.717655+0000","last_undegraded":"2024-09-21T18:08:02.717655+0000","last_fullsized":"2024-09-21T18:08:02.717655+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:08.890525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"68'362","reported_seq":724,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.917006+0000","last_change":"2024-09-21T18:07:01.405525+0000","last_active":"2024-09-21T18:08:02.917006+0000","last_peered":"2024-09-21T18:08:02.917006+0000","last_clean":"2024-09-21T18:08:02.917006+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:08:02.917006+0000","last_undegraded":"2024-09-21T18:08:02.917006+0000","last_fullsized":"2024-09-21T18:08:02.917006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.560432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280975380,"num_objects":118,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":118,"num_whiteouts":0,"num_read":387,"num_read_kb":79011,"num_write":438,"num_write_kb":180638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":99,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.926458+0000","last_change":"2024-09-21T18:06:46.982554+0000","last_active":"2024-09-21T18:08:01.926458+0000","last_peered":"2024-09-21T18:08:01.926458+0000","last_clean":"2024-09-21T18:08:01.926458+0000","last_became_active":"2024-09-21T18:06:44.966708+0000","last_became_peered":"2024-09-21T18:06:44.966708+0000","last_unstale":"2024-09-21T18:08:01.926458+0000","last_undegraded":"2024-09-21T18:08:01.926458+0000","last_fullsized":"2024-09-21T18:08:01.926458+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:12.459432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019511330000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"68'478","reported_seq":831,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.911781+0000","last_change":"2024-09-21T18:07:01.288102+0000","last_active":"2024-09-21T18:08:02.911781+0000","last_peered":"2024-09-21T18:08:02.911781+0000","last_clean":"2024-09-21T18:08:02.911781+0000","last_became_active":"2024-09-21T18:07:01.287689+0000","last_became_peered":"2024-09-21T18:07:01.287689+0000","last_unstale":"2024-09-21T18:08:02.911781+0000","last_undegraded":"2024-09-21T18:08:02.911781+0000","last_fullsized":"2024-09-21T18:08:02.911781+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:07:00.079908+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.216721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307813077,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113,"num_whiteouts":0,"num_read":439,"num_read_kb":97233,"num_write":594,"num_write_kb":224612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"68'270","reported_seq":508,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-21T18:08:03.021676+0000","last_change":"2024-09-21T18:07:03.553363+0000","last_active":"2024-09-21T18:08:03.021676+0000","last_peered":"2024-09-21T18:08:03.021676+0000","last_clean":"2024-09-21T18:08:03.021676+0000","last_became_active":"2024-09-21T18:07:03.552723+0000","last_became_peered":"2024-09-21T18:07:03.552723+0000","last_unstale":"2024-09-21T18:08:03.021676+0000","last_undegraded":"2024-09-21T18:08:03.021676+0000","last_fullsized":"2024-09-21T18:08:03.021676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:07.591572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227485220,"num_objects":81,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":7,"num_read":167,"num_read_kb":87948,"num_write":243,"num_write_kb":52575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":14,"num_flush_kb":42337,"num_evict":0,"num_evict_kb":0,"num_promote":83,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:07:59.533634+0000","last_change":"2024-09-21T18:06:46.978855+0000","last_active":"2024-09-21T18:07:59.533634+0000","last_peered":"2024-09-21T18:07:59.533634+0000","last_clean":"2024-09-21T18:07:59.533634+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:07:59.533634+0000","last_undegraded":"2024-09-21T18:07:59.533634+0000","last_fullsized":"2024-09-21T18:07:59.533634+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:04.750992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'150","reported_seq":330,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.963503+0000","last_change":"2024-09-21T18:07:04.454454+0000","last_active":"2024-09-21T18:08:01.963503+0000","last_peered":"2024-09-21T18:08:01.963503+0000","last_clean":"2024-09-21T18:08:01.963503+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:08:01.963503+0000","last_undegraded":"2024-09-21T18:08:01.963503+0000","last_fullsized":"2024-09-21T18:08:01.963503+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.533440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114456693,"num_objects":44,"num_object_clones":0,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":6,"num_read":81,"num_read_kb":44224,"num_write":127,"num_write_kb":39697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":9,"num_flush_kb":22934,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":99,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.528930+0000","last_change":"2024-09-21T18:07:51.660167+0000","last_active":"2024-09-21T18:08:01.528930+0000","last_peered":"2024-09-21T18:08:01.528930+0000","last_clean":"2024-09-21T18:08:01.528930+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:08:01.528930+0000","last_undegraded":"2024-09-21T18:08:01.528930+0000","last_fullsized":"2024-09-21T18:08:01.528930+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:18.473319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"65'281","reported_seq":551,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.110380+0000","last_change":"2024-09-21T18:07:03.453942+0000","last_active":"2024-09-21T18:08:00.110380+0000","last_peered":"2024-09-21T18:08:00.110380+0000","last_clean":"2024-09-21T18:08:00.110380+0000","last_became_active":"2024-09-21T18:07:03.453360+0000","last_became_peered":"2024-09-21T18:07:03.453360+0000","last_unstale":"2024-09-21T18:08:00.110380+0000","last_undegraded":"2024-09-21T18:08:00.110380+0000","last_fullsized":"2024-09-21T18:08:00.110380+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:14.477508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237233709,"num_objects":90,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":10,"num_read":287,"num_read_kb":197481,"num_write":245,"num_write_kb":66694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":9,"num_flush_kb":21210,"num_evict":0,"num_evict_kb":0,"num_promote":94,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":102,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-09-21T18:08:01.401336+0000","last_change":"2024-09-21T18:08:01.296359+0000","last_active":"2024-09-21T18:08:01.401336+0000","last_peered":"2024-09-21T18:08:01.401336+0000","last_clean":"2024-09-21T18:08:01.401336+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:08:01.401336+0000","last_undegraded":"2024-09-21T18:08:01.401336+0000","last_fullsized":"2024-09-21T18:08:01.401336+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:08:01.296093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:08:01.296093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:09:24.736120+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"67'132","reported_seq":307,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-21T18:08:02.185668+0000","last_change":"2024-09-21T18:07:04.220636+0000","last_active":"2024-09-21T18:08:02.185668+0000","last_peered":"2024-09-21T18:08:02.185668+0000","last_clean":"2024-09-21T18:08:02.185668+0000","last_became_active":"2024-09-21T18:07:04.220350+0000","last_became_peered":"2024-09-21T18:07:04.220350+0000","last_unstale":"2024-09-21T18:08:02.185668+0000","last_undegraded":"2024-09-21T18:08:02.185668+0000","last_fullsized":"2024-09-21T18:08:02.185668+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:07:02.095335+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:08:28.012491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123533364,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":4,"num_read":107,"num_read_kb":58383,"num_write":113,"num_write_kb":32610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":4,"num_flush_kb":8078,"num_evict":0,"num_evict_kb":0,"num_promote":47,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-09-21T18:08:00.318675+0000","last_change":"2024-09-21T18:07:58.908355+0000","last_active":"2024-09-21T18:08:00.318675+0000","last_peered":"2024-09-21T18:08:00.318675+0000","last_clean":"2024-09-21T18:08:00.318675+0000","last_became_active":"2024-09-21T18:07:58.907761+0000","last_became_peered":"2024-09-21T18:07:58.907761+0000","last_unstale":"2024-09-21T18:08:00.318675+0000","last_undegraded":"2024-09-21T18:08:00.318675+0000","last_fullsized":"2024-09-21T18:08:00.318675+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:06:43.843471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8],"acting":[6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":702708986,"num_objects":260,"num_object_clones":0,"num_object_copies":520,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":151,"num_whiteouts":27,"num_read":642,"num_read_kb":388036,"num_write":728,"num_write_kb":191576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":36,"num_flush_kb":94559,"num_evict":0,"num_evict_kb":0,"num_promote":271,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":687771648,"data_stored":725121357,"data_compressed":34673255,"data_compressed_allocated":36937728,"data_compressed_original":76431806,"omap_allocated":0,"internal_metadata":0},"log_size":833,"ondisk_log_size":833,"up":8,"acting":8,"num_store_stats":6},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1251182220,"num_objects":475,"num_object_clones":0,"num_object_copies":950,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":475,"num_whiteouts":0,"num_read":1549,"num_read_kb":321368,"num_write":2034,"num_write_kb":800808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1200644096,"data_stored":1208141401,"data_compressed":11316120,"data_compressed_allocated":12161024,"data_compressed_original":27435008,"omap_allocated":0,"internal_metadata":0},"log_size":1655,"ondisk_log_size":1655,"up":8,"acting":8,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286408,"kb_used_data":255216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94085432,"statfs":{"total":96636764160,"available":96343482368,"internally_reserved":0,"allocated":261341184,"data_stored":268004840,"data_compressed":6061803,"data_compressed_allocated":6504448,"data_compressed_original":14765821,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247826,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27744,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344096,"statfs":{"total":96636764160,"available":96608354304,"internally_reserved":0,"allocated":622592,"data_stored":376827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247827,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":161216,"kb_used_data":130024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94210624,"statfs":{"total":96636764160,"available":96471678976,"internally_reserved":0,"allocated":133144576,"data_stored":133909299,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,8,9,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313235,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307456,"kb_used_data":272028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94064384,"statfs":{"total":96636764160,"available":96321929216,"internally_reserved":0,"allocated":278556672,"data_stored":283695852,"data_compressed":6516153,"data_compressed_allocated":6938624,"data_compressed_original":13816545,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,3,5,6,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,1],"upper_bound":512},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313236,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":552960,"data_stored":354982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,4,5,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":610304,"data_stored":374308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280530,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":246552,"kb_used_data":211200,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94125288,"statfs":{"total":96636764160,"available":96384294912,"internally_reserved":0,"allocated":216268800,"data_stored":220951350,"data_compressed":4959701,"data_compressed_allocated":5312512,"data_compressed_original":11364647,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,3,5,6,8,9,11,12,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280530,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27580,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344260,"statfs":{"total":96636764160,"available":96608522240,"internally_reserved":0,"allocated":565248,"data_stored":366585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280531,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":232272,"kb_used_data":203656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94139568,"statfs":{"total":96636764160,"available":96398917632,"internally_reserved":0,"allocated":208543744,"data_stored":213558675,"data_compressed":4968709,"data_compressed_allocated":5300224,"data_compressed_original":11634138,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,6,8,9,10,11,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":7,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27700,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":581632,"data_stored":358775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,5,6,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159496,"kb_used_data":132468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94212344,"statfs":{"total":96636764160,"available":96473440256,"internally_reserved":0,"allocated":135647232,"data_stored":136434782,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[2,3,4,6,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280530,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":257440,"kb_used_data":218120,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94114400,"statfs":{"total":96636764160,"available":96373145600,"internally_reserved":0,"allocated":223354880,"data_stored":235067795,"data_compressed":10938690,"data_compressed_allocated":11669504,"data_compressed_original":24351198,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[1,3,5,8,9,11,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280531,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299108,"kb_used_data":266012,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94072732,"statfs":{"total":96636764160,"available":96330477568,"internally_reserved":0,"allocated":272396288,"data_stored":281708610,"data_compressed":8774558,"data_compressed_allocated":9359360,"data_compressed_original":20143873,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,2,3,4,5,8,9,10,12,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27608,"kb_used_data":580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":593920,"data_stored":370663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,3,4,5,6,8,11,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280531,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":187136,"kb_used_data":160104,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94184704,"statfs":{"total":96636764160,"available":96445136896,"internally_reserved":0,"allocated":163946496,"data_stored":163147579,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,4,5,6,8,9,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":132505600,"data_stored":133521903,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151875584,"data_stored":152916976,"data_compressed":1321392,"data_compressed_allocated":1437696,"data_compressed_original":3444736,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163336192,"data_stored":162772897,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":151601152,"data_stored":153833833,"data_compressed":2279103,"data_compressed_allocated":2445312,"data_compressed_original":5570560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":135069696,"data_stored":136066011,"data_compressed":1548695,"data_compressed_allocated":1654784,"data_compressed_original":3522560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":152190976,"data_stored":154425565,"data_compressed":2279103,"data_compressed_allocated":2445312,"data_compressed_original":5570560,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163336192,"data_stored":162772897,"data_compressed":468565,"data_compressed_allocated":499712,"data_compressed_original":1114112,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":150728704,"data_stored":151831319,"data_compressed":1402002,"data_compressed_allocated":1523712,"data_compressed_original":3575808,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63942656,"data_stored":67802669,"data_compressed":3601366,"data_compressed_allocated":3833856,"data_compressed_original":7854375,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":113283072,"data_stored":119198900,"data_compressed":6047588,"data_compressed_allocated":6438912,"data_compressed_original":12702433,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":56340480,"data_stored":59311565,"data_compressed":2824359,"data_compressed_allocated":2998272,"data_compressed_original":6141402,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":222490624,"data_stored":234184554,"data_compressed":10802465,"data_compressed_allocated":11522048,"data_compressed_original":23957982,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":120283136,"data_stored":127114234,"data_compressed":6425725,"data_compressed_allocated":6832128,"data_compressed_original":13995777,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":111431680,"data_stored":117509435,"data_compressed":4971752,"data_compressed_allocated":5312512,"data_compressed_original":11779837,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:08:04.539 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T18:08:04.539 INFO:tasks.thrashosds.thrasher:Waiting for trim 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 78.676782727000003, 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:04.574 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:04.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:08:04.578 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 77.591390337999997, 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:04.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:08:04.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:08:04.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:04.582 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:04.583 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:04.588 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:04.602 INFO:tasks.rados.rados.0.smithi160.stdout:1000: finishing write tid 2 to smithi16030472-356 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: finishing write tid 3 to smithi16030472-356 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:989: finishing copy_from to smithi16030472-117 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 117 v 175 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: finishing write tid 4 to smithi16030472-356 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: finishing write tid 5 to smithi16030472-356 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: oid 356 updating version 0 to 483 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: oid 356 version 483 is already newer than 482 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: oid 356 updating version 483 to 484 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: oid 356 version 484 is already newer than 481 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 356 v 484 (ObjNum 656 snap 0 seq_num 656) dirty exists 2024-09-21T18:08:04.603 INFO:tasks.rados.rados.0.smithi160.stdout:1000: left oid 356 (ObjNum 656 snap 0 seq_num 656) 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: finishing write tid 3 to smithi16030472-408 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: finishing write tid 4 to smithi16030472-408 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: finishing write tid 5 to smithi16030472-408 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: oid 408 updating version 0 to 474 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: oid 408 updating version 474 to 475 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: oid 408 updating version 475 to 476 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: oid 408 updating version 476 to 477 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 477 (ObjNum 654 snap 0 seq_num 654) dirty exists 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:994: left oid 408 (ObjNum 654 snap 0 seq_num 654) 2024-09-21T18:08:04.604 INFO:tasks.rados.rados.0.smithi160.stdout:1003: finishing write tid 1 to smithi16030472-164 2024-09-21T18:08:04.630 INFO:tasks.rados.rados.0.smithi160.stdout:1001: finishing write tid 5 to smithi16030472-73 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: oid 73 updating version 0 to 490 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: oid 73 version 490 is already newer than 489 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: oid 73 updating version 490 to 491 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: oid 73 updating version 491 to 492 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 73 v 492 (ObjNum 657 snap 0 seq_num 657) dirty exists 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: left oid 73 (ObjNum 657 snap 0 seq_num 657) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:989: done (7 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:994: done (6 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:999: done (5 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1000: done (4 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1001: done (3 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1005: done (2 left) 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1006: write (excl) oid 330 current snap is 0 2024-09-21T18:08:04.631 INFO:tasks.rados.rados.0.smithi160.stdout:1006: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-21T18:08:04.638 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:04.631+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:04.642 INFO:tasks.rados.rados.0.smithi160.stdout:1006: writing smithi16030472-330 from 709427 to 1255374 tid 1 2024-09-21T18:08:04.653 INFO:tasks.rados.rados.0.smithi160.stdout:1006: writing smithi16030472-330 from 1813774 to 2426671 tid 2 2024-09-21T18:08:04.660 INFO:tasks.rados.rados.0.smithi160.stdout:1006: writing smithi16030472-330 from 2985657 to 3109427 tid 3 2024-09-21T18:08:04.660 INFO:tasks.rados.rados.0.smithi160.stdout:1007: write (excl) oid 31 current snap is 0 2024-09-21T18:08:04.660 INFO:tasks.rados.rados.0.smithi160.stdout:1007: seq_num 662 ranges {726234=737549,1988609=756555} 2024-09-21T18:08:04.674 INFO:tasks.rados.rados.0.smithi160.stdout:1007: writing smithi16030472-31 from 726234 to 1463783 tid 1 2024-09-21T18:08:04.684 INFO:tasks.rados.rados.0.smithi160.stdout:1007: writing smithi16030472-31 from 1988609 to 2745164 tid 2 2024-09-21T18:08:04.685 INFO:tasks.rados.rados.0.smithi160.stdout:1008: delete oid 144 current snap is 0 2024-09-21T18:08:04.704 INFO:tasks.rados.rados.0.smithi160.stdout:1003: finishing write tid 2 to smithi16030472-164 2024-09-21T18:08:04.705 INFO:tasks.rados.rados.0.smithi160.stdout:1003: finishing write tid 3 to smithi16030472-164 2024-09-21T18:08:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:1003: finishing write tid 4 to smithi16030472-164 2024-09-21T18:08:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:1003: oid 164 updating version 0 to 479 2024-09-21T18:08:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:1003: oid 164 updating version 479 to 480 2024-09-21T18:08:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:1003: oid 164 version 480 is already newer than 478 2024-09-21T18:08:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 480 (ObjNum 659 snap 0 seq_num 659) dirty exists 2024-09-21T18:08:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:1003: left oid 164 (ObjNum 659 snap 0 seq_num 659) 2024-09-21T18:08:04.708 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:04.703+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:04.724 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:04.725 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:04.725 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:04.741 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:04.761 INFO:tasks.rados.rados.0.smithi160.stdout:1006: finishing write tid 1 to smithi16030472-330 2024-09-21T18:08:04.809 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:04.805+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:04.857 INFO:tasks.rados.rados.0.smithi160.stdout:1004: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:08:04.857 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 455005 2024-09-21T18:08:04.865 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:04.866 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:04.866 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:04.880 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:04.923 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:04.921+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:04.952 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:04.982 INFO:tasks.rados.rados.0.smithi160.stdout:1006: finishing write tid 2 to smithi16030472-330 2024-09-21T18:08:04.984 INFO:tasks.rados.rados.0.smithi160.stdout:1007: finishing write tid 1 to smithi16030472-31 2024-09-21T18:08:04.984 INFO:tasks.rados.rados.0.smithi160.stdout:1007: finishing write tid 2 to smithi16030472-31 2024-09-21T18:08:05.010 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:05.003+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:05.051 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009441287000001, 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:08:05.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009201662999999, 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009166256, 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:08:05.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:08:05.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009133083000002, 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:08:05.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:08:05.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009103523, 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:08:05.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.009073866999998, 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:08:05.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:08:05.079 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.008587611000003, 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:08:05.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:08:05.083 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 34.008554017999998, 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:08:05.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.908336325999997, 2024-09-21T18:08:05.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.093 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.907737421999997, 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:05.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:08:05.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:08:05.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:08:05.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.098 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.907022804999997, 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:08:05.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:08:05.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.897770072999997, 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:08:05.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:08:05.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:08:05.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.897505371999998, 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.109 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.113 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.897410682999997, 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.897369392999998, 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.118 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:08:05.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.122 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.886099489999999, 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:05.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:08:05.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.127 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.876983256999999, 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:08:05.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:08:05.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:08:05.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.132 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.875919889999999, 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:08:05.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:08:05.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:08:05.136 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.875879580000003, 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:08:05.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:08:05.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 33.113241737999999, 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:08:05.141 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.145 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.146 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.146 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:05.146 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:05.146 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:05.147 INFO:tasks.rados.rados.0.smithi160.stdout:1006: finishing write tid 3 to smithi16030472-330 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: finishing write tid 3 to smithi16030472-31 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: finishing write tid 4 to smithi16030472-330 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: finishing write tid 4 to smithi16030472-31 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: oid 31 updating version 0 to 496 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: oid 31 version 496 is already newer than 494 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: oid 31 version 496 is already newer than 495 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 496 (ObjNum 662 snap 0 seq_num 662) dirty exists 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1007: left oid 31 (ObjNum 662 snap 0 seq_num 662) 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: finishing write tid 5 to smithi16030472-330 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: oid 330 updating version 0 to 394 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: oid 330 updating version 394 to 395 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: oid 330 updating version 395 to 406 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: oid 330 version 406 is already newer than 405 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 330 v 406 (ObjNum 661 snap 0 seq_num 661) dirty exists 2024-09-21T18:08:05.148 INFO:tasks.rados.rados.0.smithi160.stdout:1006: left oid 330 (ObjNum 661 snap 0 seq_num 661) 2024-09-21T18:08:05.149 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:05.149 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:05.145+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:05.186 INFO:tasks.rados.rados.0.smithi160.stdout:1003: done (4 left) 2024-09-21T18:08:05.186 INFO:tasks.rados.rados.0.smithi160.stdout:1004: done (3 left) 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1006: done (2 left) 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1007: done (1 left) 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1008: done (0 left) 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1009: read oid 454 snap -1 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1009: expect deleted 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1010: cache_try_flush oid 107 snap -1 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1011: read oid 482 snap -1 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1011: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1012: copy_from oid 354 from oid 477 current snap is 0 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1013: cache_try_flush oid 439 snap -1 2024-09-21T18:08:05.187 INFO:tasks.rados.rados.0.smithi160.stdout:1014: delete oid 425 current snap is 0 2024-09-21T18:08:05.248 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:05.243+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:05.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:05.281 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:05.281 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:05.281 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:05.287 INFO:tasks.rados.rados.0.smithi160.stdout:1010: got (0) Success 2024-09-21T18:08:05.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 107 v 0 (ObjNum 106 snap 0 seq_num 106) dirty exists 2024-09-21T18:08:05.289 INFO:tasks.rados.rados.0.smithi160.stdout:1013: got (0) Success 2024-09-21T18:08:05.289 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 439 v 0 (ObjNum 438 snap 0 seq_num 438) dirty exists 2024-09-21T18:08:05.290 INFO:tasks.rados.rados.0.smithi160.stdout:1011: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-21T18:08:05.297 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:05.349 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:05.345+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:05.425 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:05.425 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:05.425 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:05.426 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:05.440 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:05.444 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:05.449 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:05.443+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:05.465 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:05.466 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:05.486 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:05.486 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1009: done (5 left) 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1010: done (4 left) 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1011: done (3 left) 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1013: done (2 left) 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1014: done (1 left) 2024-09-21T18:08:05.494 INFO:tasks.rados.rados.0.smithi160.stdout:1015: delete oid 79 current snap is 0 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1015: done (1 left) 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1016: read oid 306 snap -1 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1016: expect deleted 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1017: read oid 431 snap -1 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1017: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1018: read oid 485 snap -1 2024-09-21T18:08:05.497 INFO:tasks.rados.rados.0.smithi160.stdout:1018: expect deleted 2024-09-21T18:08:05.498 INFO:tasks.rados.rados.0.smithi160.stdout:1019: copy_from oid 294 from oid 315 current snap is 0 2024-09-21T18:08:05.498 INFO:tasks.rados.rados.0.smithi160.stdout:1020: delete oid 201 current snap is 0 2024-09-21T18:08:05.503 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:05.504 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:05.505 INFO:tasks.rados.rados.0.smithi160.stdout:1012: finishing copy_from racing read to smithi16030472-354 2024-09-21T18:08:05.513 INFO:tasks.rados.rados.0.smithi160.stdout:1017: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-21T18:08:05.522 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:05.522 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:05.524 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1414785 2024-09-21T18:08:05.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:05.543 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:05.551 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:05.545+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:05.562 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:05.562 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:05.567 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:05.567 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:05.582 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:05.583 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:05.583 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:05.589 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1795578 2024-09-21T18:08:05.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:08:05.605 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:05.605 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:05.616 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:05.625 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:05.625 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:05.642 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:1012: finishing copy_from to smithi16030472-354 2024-09-21T18:08:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 354 v 361 (ObjNum 476 snap 0 seq_num 476) dirty exists 2024-09-21T18:08:05.652 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:05.646+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:05.660 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:05.660 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:05.681 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:05.712 INFO:tasks.rados.rados.0.smithi160.stdout:1019: finishing copy_from racing read to smithi16030472-294 2024-09-21T18:08:05.712 INFO:tasks.rados.rados.0.smithi160.stdout:1019: finishing copy_from to smithi16030472-294 2024-09-21T18:08:05.712 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 269 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-21T18:08:05.734 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:05.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:05.748 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:05.749 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:05.753 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:05.749+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:05.763 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1012: done (5 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1016: done (4 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1017: done (3 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1018: done (2 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1019: done (1 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1020: done (0 left) 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1021: cache_flush oid 410 snap -1 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1022: write (excl) oid 364 current snap is 0 2024-09-21T18:08:05.813 INFO:tasks.rados.rados.0.smithi160.stdout:1022: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-21T18:08:05.831 INFO:tasks.rados.rados.0.smithi160.stdout:1022: writing smithi16030472-364 from 410269 to 1124063 tid 1 2024-09-21T18:08:05.847 INFO:tasks.rados.rados.0.smithi160.stdout:1022: writing smithi16030472-364 from 1648313 to 2155864 tid 2 2024-09-21T18:08:05.853 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:05.846+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:05.864 INFO:tasks.rados.rados.0.smithi160.stdout:1022: writing smithi16030472-364 from 2917489 to 3210269 tid 3 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1023: cache_try_flush oid 355 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1021: got (0) Success 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 0 (ObjNum 409 snap 0 seq_num 409) dirty exists 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1021: done (2 left) 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1024: cache_try_flush oid 29 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1025: cache_flush oid 441 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1026: cache_try_flush oid 455 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1027: cache_flush oid 437 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1028: cache_flush oid 381 snap -1 2024-09-21T18:08:05.865 INFO:tasks.rados.rados.0.smithi160.stdout:1029: delete oid 473 current snap is 0 2024-09-21T18:08:05.866 INFO:tasks.rados.rados.0.smithi160.stdout:1024: got (2) No such file or directory 2024-09-21T18:08:05.866 INFO:tasks.rados.rados.0.smithi160.stdout:1023: got (0) Success 2024-09-21T18:08:05.866 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 0 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-09-21T18:08:05.866 INFO:tasks.rados.rados.0.smithi160.stdout:1027: got (0) Success 2024-09-21T18:08:05.866 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 437 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:08:05.892 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:05.892 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:05.907 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:05.955 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:05.950+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:05.966 INFO:tasks.rados.rados.0.smithi160.stdout:1028: got (0) Success 2024-09-21T18:08:05.966 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:08:05.966 INFO:tasks.rados.rados.0.smithi160.stdout:1022: finishing write tid 1 to smithi16030472-364 2024-09-21T18:08:05.980 INFO:tasks.rados.rados.0.smithi160.stdout:1026: got (0) Success 2024-09-21T18:08:05.980 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 0 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-21T18:08:06.004 INFO:tasks.rados.rados.0.smithi160.stdout:1025: got (0) Success 2024-09-21T18:08:06.004 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 0 (ObjNum 558 snap 0 seq_num 558) dirty exists 2024-09-21T18:08:06.029 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.029 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.044 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:06.055 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:06.051+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:06.156 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:06.150+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:06.163 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.164 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1023: done (7 left) 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1024: done (6 left) 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1025: done (5 left) 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1026: done (4 left) 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1027: done (3 left) 2024-09-21T18:08:06.170 INFO:tasks.rados.rados.0.smithi160.stdout:1028: done (2 left) 2024-09-21T18:08:06.171 INFO:tasks.rados.rados.0.smithi160.stdout:1029: done (1 left) 2024-09-21T18:08:06.171 INFO:tasks.rados.rados.0.smithi160.stdout:1030: delete oid 405 current snap is 0 2024-09-21T18:08:06.175 INFO:tasks.rados.rados.0.smithi160.stdout:1030: done (1 left) 2024-09-21T18:08:06.175 INFO:tasks.rados.rados.0.smithi160.stdout:1031: read oid 322 snap -1 2024-09-21T18:08:06.175 INFO:tasks.rados.rados.0.smithi160.stdout:1031: expect deleted 2024-09-21T18:08:06.175 INFO:tasks.rados.rados.0.smithi160.stdout:1032: cache_try_flush oid 459 snap -1 2024-09-21T18:08:06.176 INFO:tasks.rados.rados.0.smithi160.stdout:1033: read oid 327 snap -1 2024-09-21T18:08:06.176 INFO:tasks.rados.rados.0.smithi160.stdout:1033: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:06.176 INFO:tasks.rados.rados.0.smithi160.stdout:1034: copy_from oid 422 from oid 309 current snap is 0 2024-09-21T18:08:06.176 INFO:tasks.rados.rados.0.smithi160.stdout:1035: write oid 167 current snap is 0 2024-09-21T18:08:06.176 INFO:tasks.rados.rados.0.smithi160.stdout:1035: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-21T18:08:06.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:06.193 INFO:tasks.rados.rados.0.smithi160.stdout:1035: writing smithi16030472-167 from 460690 to 1116584 tid 1 2024-09-21T18:08:06.213 INFO:tasks.rados.rados.0.smithi160.stdout:1035: writing smithi16030472-167 from 1540112 to 2311343 tid 2 2024-09-21T18:08:06.227 INFO:tasks.rados.rados.0.smithi160.stdout:1035: writing smithi16030472-167 from 2716111 to 3255889 tid 3 2024-09-21T18:08:06.227 INFO:tasks.rados.rados.0.smithi160.stdout:1036: write (excl) oid 344 current snap is 0 2024-09-21T18:08:06.227 INFO:tasks.rados.rados.0.smithi160.stdout:1036: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-21T18:08:06.241 INFO:tasks.rados.rados.0.smithi160.stdout:1036: writing smithi16030472-344 from 477497 to 1208640 tid 1 2024-09-21T18:08:06.255 INFO:tasks.rados.rados.0.smithi160.stdout:1036: writing smithi16030472-344 from 1998594 to 2513483 tid 2 2024-09-21T18:08:06.257 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:06.251+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:06.260 INFO:tasks.rados.rados.0.smithi160.stdout:1036: writing smithi16030472-344 from 3143534 to 3277497 tid 3 2024-09-21T18:08:06.260 INFO:tasks.rados.rados.0.smithi160.stdout:1037: write (excl) oid 191 current snap is 0 2024-09-21T18:08:06.260 INFO:tasks.rados.rados.0.smithi160.stdout:1037: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-21T18:08:06.267 INFO:tasks.rados.rados.0.smithi160.stdout:1037: writing smithi16030472-191 from 494304 to 900696 tid 1 2024-09-21T18:08:06.276 INFO:tasks.rados.rados.0.smithi160.stdout:1037: writing smithi16030472-191 from 1540723 to 2315623 tid 2 2024-09-21T18:08:06.283 INFO:tasks.rados.rados.0.smithi160.stdout:1037: writing smithi16030472-191 from 2770957 to 3294304 tid 3 2024-09-21T18:08:06.283 INFO:tasks.rados.rados.0.smithi160.stdout:1038: cache_evict oid 170 snap -1 2024-09-21T18:08:06.283 INFO:tasks.rados.rados.0.smithi160.stdout:1022: finishing write tid 2 to smithi16030472-364 2024-09-21T18:08:06.283 INFO:tasks.rados.rados.0.smithi160.stdout:1039: cache_try_flush oid 80 snap -1 2024-09-21T18:08:06.283 INFO:tasks.rados.rados.0.smithi160.stdout:1040: delete oid 331 current snap is 0 2024-09-21T18:08:06.284 INFO:tasks.rados.rados.0.smithi160.stdout:1022: finishing write tid 3 to smithi16030472-364 2024-09-21T18:08:06.307 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.307 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.322 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:06.358 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:06.357+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:06.380 INFO:tasks.rados.rados.0.smithi160.stdout:1022: finishing write tid 4 to smithi16030472-364 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: finishing write tid 5 to smithi16030472-364 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: oid 364 updating version 0 to 409 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: oid 364 version 409 is already newer than 407 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: oid 364 version 409 is already newer than 396 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: oid 364 version 409 is already newer than 408 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 409 (ObjNum 667 snap 0 seq_num 667) dirty exists 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: left oid 364 (ObjNum 667 snap 0 seq_num 667) 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1038: got (2) No such file or directory 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1022: done (10 left) 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1031: done (9 left) 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1038: done (8 left) 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1040: done (7 left) 2024-09-21T18:08:06.382 INFO:tasks.rados.rados.0.smithi160.stdout:1041: write oid 467 current snap is 0 2024-09-21T18:08:06.383 INFO:tasks.rados.rados.0.smithi160.stdout:1041: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2024-09-21T18:08:06.392 INFO:tasks.rados.rados.0.smithi160.stdout:1041: writing smithi16030472-467 from 527918 to 1084808 tid 1 2024-09-21T18:08:06.402 INFO:tasks.rados.rados.0.smithi160.stdout:1041: writing smithi16030472-467 from 1657687 to 2436256 tid 2 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1041: writing smithi16030472-467 from 3225803 to 3327918 tid 3 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1042: read oid 430 snap -1 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1042: expect (ObjNum 429 snap 0 seq_num 429) 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1043: cache_try_flush oid 197 snap -1 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1044: write oid 464 current snap is 0 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:1033: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:06.409 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 78116 2024-09-21T18:08:06.451 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.451 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.458 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:06.453+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:06.466 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:06.491 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:06.533 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:06.557 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:08:06.558 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:06.553+0000 7f0b6c216640 -1 osd.9 70 handle_osd_map injecting map crc failure 2024-09-21T18:08:06.559 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:06.555+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1035: finishing write tid 1 to smithi16030472-167 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1036: finishing write tid 1 to smithi16030472-344 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1036: finishing write tid 2 to smithi16030472-344 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1036: finishing write tid 3 to smithi16030472-344 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1036: finishing write tid 4 to smithi16030472-344 2024-09-21T18:08:06.570 INFO:tasks.rados.rados.0.smithi160.stdout:1044: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-09-21T18:08:06.581 INFO:tasks.rados.rados.0.smithi160.stdout:1044: writing smithi16030472-464 from 544725 to 1176864 tid 1 2024-09-21T18:08:06.590 INFO:tasks.rados.rados.0.smithi160.stdout:1044: writing smithi16030472-464 from 1716169 to 2238396 tid 2 2024-09-21T18:08:06.599 INFO:tasks.rados.rados.0.smithi160.stdout:1044: writing smithi16030472-464 from 2736873 to 3249364 tid 3 2024-09-21T18:08:06.599 INFO:tasks.rados.rados.0.smithi160.stdout:1033: done (10 left) 2024-09-21T18:08:06.599 INFO:tasks.rados.rados.0.smithi160.stdout:1045: write oid 47 current snap is 0 2024-09-21T18:08:06.599 INFO:tasks.rados.rados.0.smithi160.stdout:1045: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-21T18:08:06.607 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.607 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.609 INFO:tasks.rados.rados.0.smithi160.stdout:1045: writing smithi16030472-47 from 561532 to 1268920 tid 1 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1045: writing smithi16030472-47 from 2058298 to 2840536 tid 2 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1046: delete oid 159 current snap is 0 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1043: got (0) Success 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 0 (ObjNum 468 snap 0 seq_num 468) dirty exists 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1036: finishing write tid 5 to smithi16030472-344 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1036: oid 344 updating version 0 to 500 2024-09-21T18:08:06.623 INFO:tasks.rados.rados.0.smithi160.stdout:1036: oid 344 version 500 is already newer than 498 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:1036: oid 344 version 500 is already newer than 499 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:1036: oid 344 updating version 500 to 501 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 344 v 501 (ObjNum 671 snap 0 seq_num 671) dirty exists 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:1036: left oid 344 (ObjNum 671 snap 0 seq_num 671) 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:1035: finishing write tid 2 to smithi16030472-167 2024-09-21T18:08:06.624 INFO:tasks.rados.rados.0.smithi160.stdout:1045: finishing write tid 1 to smithi16030472-47 2024-09-21T18:08:06.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:06.660 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:06.657+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:06.710 INFO:tasks.rados.rados.0.smithi160.stdout:1032: got (0) Success 2024-09-21T18:08:06.710 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 0 (ObjNum 458 snap 0 seq_num 458) dirty exists 2024-09-21T18:08:06.731 INFO:tasks.rados.rados.0.smithi160.stdout:1045: finishing write tid 2 to smithi16030472-47 2024-09-21T18:08:06.750 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.751 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.760 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:06.757+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:06.764 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:06.800 INFO:tasks.rados.rados.0.smithi160.stdout:1037: finishing write tid 1 to smithi16030472-191 2024-09-21T18:08:06.800 INFO:tasks.rados.rados.0.smithi160.stdout:1037: finishing write tid 2 to smithi16030472-191 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: finishing write tid 3 to smithi16030472-191 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: finishing write tid 4 to smithi16030472-191 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: finishing write tid 5 to smithi16030472-191 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: oid 191 updating version 0 to 489 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: oid 191 version 489 is already newer than 488 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: oid 191 version 489 is already newer than 487 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: oid 191 version 489 is already newer than 486 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 191 v 489 (ObjNum 672 snap 0 seq_num 672) dirty exists 2024-09-21T18:08:06.801 INFO:tasks.rados.rados.0.smithi160.stdout:1037: left oid 191 (ObjNum 672 snap 0 seq_num 672) 2024-09-21T18:08:06.806 INFO:tasks.rados.rados.0.smithi160.stdout:1042: expect (ObjNum 429 snap 0 seq_num 429) 2024-09-21T18:08:06.808 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:06.803+0000 7fb924fb6640 -1 osd.8 70 handle_osd_map injecting map crc failure 2024-09-21T18:08:06.861 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:06.854+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:06.890 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:06.890 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:06.905 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:06.938 INFO:tasks.rados.rados.0.smithi160.stdout:1035: finishing write tid 3 to smithi16030472-167 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: finishing write tid 4 to smithi16030472-167 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: finishing write tid 5 to smithi16030472-167 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: oid 167 updating version 0 to 437 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: oid 167 version 437 is already newer than 436 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: oid 167 updating version 437 to 502 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: oid 167 updating version 502 to 503 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 167 v 503 (ObjNum 670 snap 0 seq_num 670) dirty exists 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1035: left oid 167 (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1041: finishing write tid 1 to smithi16030472-467 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1041: finishing write tid 2 to smithi16030472-467 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1041: finishing write tid 3 to smithi16030472-467 2024-09-21T18:08:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:1041: finishing write tid 4 to smithi16030472-467 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: finishing write tid 5 to smithi16030472-467 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: oid 467 updating version 0 to 491 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: oid 467 updating version 491 to 492 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: oid 467 updating version 492 to 493 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: oid 467 version 493 is already newer than 490 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 467 v 493 (ObjNum 674 snap 0 seq_num 674) dirty exists 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1041: left oid 467 (ObjNum 674 snap 0 seq_num 674) 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1039: got (0) Success 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 0 (ObjNum 79 snap 0 seq_num 79) dirty exists 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: finishing write tid 3 to smithi16030472-47 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: finishing write tid 4 to smithi16030472-47 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: oid 47 updating version 0 to 412 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: oid 47 version 412 is already newer than 411 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: oid 47 updating version 412 to 413 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 413 (ObjNum 676 snap 0 seq_num 676) dirty exists 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1045: left oid 47 (ObjNum 676 snap 0 seq_num 676) 2024-09-21T18:08:06.940 INFO:tasks.rados.rados.0.smithi160.stdout:1034: finishing copy_from racing read to smithi16030472-422 2024-09-21T18:08:06.941 INFO:tasks.rados.rados.0.smithi160.stdout:1044: finishing write tid 1 to smithi16030472-464 2024-09-21T18:08:06.961 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:06.957+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:07.025 INFO:tasks.rados.rados.0.smithi160.stdout:1032: done (11 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1035: done (10 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1036: done (9 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1037: done (8 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1039: done (7 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1041: done (6 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1042: done (5 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1043: done (4 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1045: done (3 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1046: done (2 left) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1047: read oid 121 snap -1 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1047: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1048: read oid 307 snap -1 2024-09-21T18:08:07.026 INFO:tasks.rados.rados.0.smithi160.stdout:1048: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:07.027 INFO:tasks.rados.rados.0.smithi160.stdout:1049: write (excl) oid 177 current snap is 0 2024-09-21T18:08:07.027 INFO:tasks.rados.rados.0.smithi160.stdout:1049: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-21T18:08:07.028 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.028 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.035 INFO:tasks.rados.rados.0.smithi160.stdout:1049: writing smithi16030472-177 from 595146 to 1169385 tid 1 2024-09-21T18:08:07.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:07.047 INFO:tasks.rados.rados.0.smithi160.stdout:1049: writing smithi16030472-177 from 1891615 to 2561169 tid 2 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1049: writing smithi16030472-177 from 3219142 to 3395146 tid 3 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1050: cache_evict oid 466 snap -1 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1034: finishing copy_from to smithi16030472-422 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 422 v 211 (ObjNum 308 snap 0 seq_num 308) dirty exists 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1044: finishing write tid 2 to smithi16030472-464 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1044: finishing write tid 3 to smithi16030472-464 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1049: finishing write tid 1 to smithi16030472-177 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1034: done (5 left) 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1051: read oid 147 snap -1 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1051: expect (ObjNum 146 snap 0 seq_num 146) 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1052: cache_try_flush oid 406 snap -1 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1053: copy_from oid 33 from oid 255 current snap is 0 2024-09-21T18:08:07.056 INFO:tasks.rados.rados.0.smithi160.stdout:1054: cache_try_flush oid 325 snap -1 2024-09-21T18:08:07.057 INFO:tasks.rados.rados.0.smithi160.stdout:1055: read oid 347 snap -1 2024-09-21T18:08:07.057 INFO:tasks.rados.rados.0.smithi160.stdout:1055: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-21T18:08:07.057 INFO:tasks.rados.rados.0.smithi160.stdout:1056: cache_flush oid 310 snap -1 2024-09-21T18:08:07.057 INFO:tasks.rados.rados.0.smithi160.stdout:1057: delete oid 305 current snap is 0 2024-09-21T18:08:07.057 INFO:tasks.rados.rados.0.smithi160.stdout:1054: got (2) No such file or directory 2024-09-21T18:08:07.062 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:07.055+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:07.131 INFO:tasks.rados.rados.0.smithi160.stdout:1047: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:07.131 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.162 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.162 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.163 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:07.157+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:07.177 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:07.220 INFO:tasks.rados.rados.0.smithi160.stdout:1048: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:07.263 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:07.259+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:07.269 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.299 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.299 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.314 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:07.318 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: finishing write tid 4 to smithi16030472-464 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: finishing write tid 5 to smithi16030472-464 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: oid 464 updating version 0 to 496 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: oid 464 updating version 496 to 497 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: oid 464 version 497 is already newer than 494 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: oid 464 updating version 497 to 498 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 498 (ObjNum 675 snap 0 seq_num 675) dirty exists 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1044: left oid 464 (ObjNum 675 snap 0 seq_num 675) 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1056: got (0) Success 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 0 (ObjNum 129 snap 0 seq_num 3145242376) clean dne 2024-09-21T18:08:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:1049: finishing write tid 2 to smithi16030472-177 2024-09-21T18:08:07.365 INFO:tasks.rados.rados.0.smithi160.stdout:1049: finishing write tid 3 to smithi16030472-177 2024-09-21T18:08:07.365 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:07.361+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: finishing write tid 4 to smithi16030472-177 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: finishing write tid 5 to smithi16030472-177 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: oid 177 updating version 0 to 415 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: oid 177 version 415 is already newer than 414 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: oid 177 updating version 415 to 416 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: oid 177 updating version 416 to 417 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 177 v 417 (ObjNum 678 snap 0 seq_num 678) dirty exists 2024-09-21T18:08:07.367 INFO:tasks.rados.rados.0.smithi160.stdout:1049: left oid 177 (ObjNum 678 snap 0 seq_num 678) 2024-09-21T18:08:07.368 INFO:tasks.rados.rados.0.smithi160.stdout:1050: got (0) Success 2024-09-21T18:08:07.436 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.436 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.442 INFO:tasks.rados.rados.0.smithi160.stdout:1044: done (11 left) 2024-09-21T18:08:07.442 INFO:tasks.rados.rados.0.smithi160.stdout:1047: done (10 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1048: done (9 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1049: done (8 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1050: done (7 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1054: done (6 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1056: done (5 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1057: done (4 left) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1058: copy_from oid 375 from oid 161 current snap is 0 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1059: cache_flush oid 132 snap -1 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1060: read oid 106 snap -1 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1060: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1061: write (excl) oid 422 current snap is 0 2024-09-21T18:08:07.443 INFO:tasks.rados.rados.0.smithi160.stdout:1061: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-21T18:08:07.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:07.457 INFO:tasks.rados.rados.0.smithi160.stdout:1061: writing smithi16030472-422 from 628760 to 1353497 tid 1 2024-09-21T18:08:07.466 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:07.462+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:07.469 INFO:tasks.rados.rados.0.smithi160.stdout:1061: writing smithi16030472-422 from 1892226 to 2565449 tid 2 2024-09-21T18:08:07.478 INFO:tasks.rados.rados.0.smithi160.stdout:1061: writing smithi16030472-422 from 3157635 to 3428760 tid 3 2024-09-21T18:08:07.478 INFO:tasks.rados.rados.0.smithi160.stdout:1062: cache_flush oid 403 snap -1 2024-09-21T18:08:07.478 INFO:tasks.rados.rados.0.smithi160.stdout:1063: cache_try_flush oid 464 snap -1 2024-09-21T18:08:07.478 INFO:tasks.rados.rados.0.smithi160.stdout:1064: delete oid 223 current snap is 0 2024-09-21T18:08:07.554 INFO:tasks.rados.rados.0.smithi160.stdout:1051: expect (ObjNum 146 snap 0 seq_num 146) 2024-09-21T18:08:07.566 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:07.559+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:07.585 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.620 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.621 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.634 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:07.640 INFO:tasks.rados.rados.0.smithi160.stdout:1053: finishing copy_from to smithi16030472-33 2024-09-21T18:08:07.640 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 33 v 459 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:08:07.640 INFO:tasks.rados.rados.0.smithi160.stdout:1062: got (2) No such file or directory 2024-09-21T18:08:07.641 INFO:tasks.rados.rados.0.smithi160.stdout:1055: expect (ObjNum 346 snap 0 seq_num 346) 2024-09-21T18:08:07.641 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.667 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:07.665+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:07.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:07.749 INFO:tasks.rados.rados.0.smithi160.stdout:1058: finishing copy_from racing read to smithi16030472-375 2024-09-21T18:08:07.749 INFO:tasks.rados.rados.0.smithi160.stdout:1053: finishing copy_from racing read to smithi16030472-33 2024-09-21T18:08:07.749 INFO:tasks.rados.rados.0.smithi160.stdout:1058: finishing copy_from to smithi16030472-375 2024-09-21T18:08:07.749 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 375 v 90 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-09-21T18:08:07.749 INFO:tasks.rados.rados.0.smithi160.stdout:1059: got (0) Success 2024-09-21T18:08:07.750 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 0 (ObjNum 641 snap 0 seq_num 641) dirty exists 2024-09-21T18:08:07.750 INFO:tasks.rados.rados.0.smithi160.stdout:1060: expect (ObjNum 105 snap 0 seq_num 105) 2024-09-21T18:08:07.753 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 740315 2024-09-21T18:08:07.760 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.760 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:07.783 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:07.779+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:07.784 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:07.859 INFO:tasks.rados.rados.0.smithi160.stdout:1061: finishing write tid 1 to smithi16030472-422 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: finishing write tid 2 to smithi16030472-422 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: finishing write tid 3 to smithi16030472-422 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: finishing write tid 4 to smithi16030472-422 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: finishing write tid 5 to smithi16030472-422 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: oid 422 updating version 0 to 502 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: oid 422 version 502 is already newer than 500 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: oid 422 updating version 502 to 503 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: oid 422 version 503 is already newer than 501 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 422 v 503 (ObjNum 680 snap 0 seq_num 680) dirty exists 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1061: left oid 422 (ObjNum 680 snap 0 seq_num 680) 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1052: got (0) Success 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 406 v 0 (ObjNum 547 snap 0 seq_num 547) dirty exists 2024-09-21T18:08:07.860 INFO:tasks.rados.rados.0.smithi160.stdout:1051: done (10 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1052: done (9 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1053: done (8 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1055: done (7 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1058: done (6 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1059: done (5 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1060: done (4 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1061: done (3 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1062: done (2 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1064: done (1 left) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1065: read oid 441 snap -1 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1065: expect (ObjNum 558 snap 0 seq_num 558) 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1066: copy_from oid 409 from oid 184 current snap is 0 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1067: write (excl) oid 270 current snap is 0 2024-09-21T18:08:07.861 INFO:tasks.rados.rados.0.smithi160.stdout:1067: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-21T18:08:07.869 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:07.865+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:07.871 INFO:tasks.rados.rados.0.smithi160.stdout:1067: writing smithi16030472-270 from 662374 to 1137609 tid 1 2024-09-21T18:08:07.883 INFO:tasks.rados.rados.0.smithi160.stdout:1067: writing smithi16030472-270 from 1609190 to 2286082 tid 2 2024-09-21T18:08:07.895 INFO:tasks.rados.rados.0.smithi160.stdout:1067: writing smithi16030472-270 from 2812481 to 3436582 tid 3 2024-09-21T18:08:07.895 INFO:tasks.rados.rados.0.smithi160.stdout:1068: copy_from oid 347 from oid 186 current snap is 0 2024-09-21T18:08:07.896 INFO:tasks.rados.rados.0.smithi160.stdout:1066: finishing copy_from to smithi16030472-409 2024-09-21T18:08:07.896 INFO:tasks.rados.rados.0.smithi160.stdout:1066: got expected ENOENT (src dne) 2024-09-21T18:08:07.896 INFO:tasks.rados.rados.0.smithi160.stdout:1066: finishing copy_from racing read to smithi16030472-409 2024-09-21T18:08:07.896 INFO:tasks.rados.rados.0.smithi160.stdout:1065: expect (ObjNum 558 snap 0 seq_num 558) 2024-09-21T18:08:07.897 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:07.903 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:07.903 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:07.917 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:07.918 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:07.937 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:07.937 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:07.958 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:07.958 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:07.969 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:07.963+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:07.977 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:07.999 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:07.999 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:08.015 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2574396 2024-09-21T18:08:08.019 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:08.037 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:08.058 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:08.059 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:08.070 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:08.067+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:08.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:08.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1063: got (0) Success 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 0 (ObjNum 675 snap 0 seq_num 675) dirty exists 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1063: done (4 left) 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1065: done (3 left) 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1066: done (2 left) 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1069: read oid 83 snap -1 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1069: expect (ObjNum 631 snap 0 seq_num 631) 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1070: cache_try_flush oid 249 snap -1 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1071: cache_try_flush oid 485 snap -1 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1072: cache_try_flush oid 491 snap -1 2024-09-21T18:08:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:1073: read oid 344 snap -1 2024-09-21T18:08:08.088 INFO:tasks.rados.rados.0.smithi160.stdout:1073: expect (ObjNum 671 snap 0 seq_num 671) 2024-09-21T18:08:08.088 INFO:tasks.rados.rados.0.smithi160.stdout:1074: write oid 291 current snap is 0 2024-09-21T18:08:08.088 INFO:tasks.rados.rados.0.smithi160.stdout:1074: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-21T18:08:08.088 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:08.088 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:08.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:08.103 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:08.103 INFO:tasks.rados.rados.0.smithi160.stdout:1074: writing smithi16030472-291 from 679181 to 1229665 tid 1 2024-09-21T18:08:08.103 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:08.115 INFO:tasks.rados.rados.0.smithi160.stdout:1074: writing smithi16030472-291 from 1667672 to 2204575 tid 2 2024-09-21T18:08:08.121 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:08.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1074: writing smithi16030472-291 from 2839904 to 3275277 tid 3 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1075: cache_evict oid 8 snap -1 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1072: got (0) Success 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1072: done (8 left) 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1076: cache_flush oid 417 snap -1 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1077: write oid 238 current snap is 0 2024-09-21T18:08:08.130 INFO:tasks.rados.rados.0.smithi160.stdout:1077: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-21T18:08:08.145 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:08.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:08.146 INFO:tasks.rados.rados.0.smithi160.stdout:1077: writing smithi16030472-238 from 695988 to 1321721 tid 1 2024-09-21T18:08:08.163 INFO:tasks.rados.rados.0.smithi160.stdout:1077: writing smithi16030472-238 from 2009801 to 2690362 tid 2 2024-09-21T18:08:08.164 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:08.171 INFO:tasks.rados.rados.0.smithi160.stdout:1077: writing smithi16030472-238 from 3150974 to 3495988 tid 3 2024-09-21T18:08:08.171 INFO:tasks.rados.rados.0.smithi160.stdout:1078: copy_from oid 430 from oid 395 current snap is 0 2024-09-21T18:08:08.171 INFO:tasks.rados.rados.0.smithi160.stdout:1075: got (16) Device or resource busy 2024-09-21T18:08:08.171 INFO:tasks.rados.rados.0.smithi160.stdout:1075: done (10 left) 2024-09-21T18:08:08.171 INFO:tasks.rados.rados.0.smithi160.stdout:1079: write (excl) oid 295 current snap is 0 2024-09-21T18:08:08.172 INFO:tasks.rados.rados.0.smithi160.stdout:1079: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-21T18:08:08.172 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:08.169+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:08.181 INFO:tasks.rados.rados.0.smithi160.stdout:1079: writing smithi16030472-295 from 712795 to 1130130 tid 1 2024-09-21T18:08:08.192 INFO:tasks.rados.rados.0.smithi160.stdout:1079: writing smithi16030472-295 from 1784636 to 2325208 tid 2 2024-09-21T18:08:08.201 INFO:tasks.rados.rados.0.smithi160.stdout:1079: writing smithi16030472-295 from 2894750 to 3352667 tid 3 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1080: copy_from oid 233 from oid 63 current snap is 0 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1081: cache_try_flush oid 203 snap -1 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1082: read oid 467 snap -1 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1082: expect (ObjNum 674 snap 0 seq_num 674) 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1083: cache_evict oid 68 snap -1 2024-09-21T18:08:08.202 INFO:tasks.rados.rados.0.smithi160.stdout:1084: delete oid 170 current snap is 0 2024-09-21T18:08:08.213 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:08.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:236 3.1 e45/31 3:96516620:::smithi16030472-58:head v 45'30, mlcod=45'24)", 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 53.541891059999998, 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.202914602, 2024-09-21T18:08:08.228 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684354+0000", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2959999999999999e-06 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684626+0000", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027191299999999998 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684653+0000", 2024-09-21T18:08:08.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7195999999999998e-05 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684667+0000", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3767e-05 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787798+0000", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103131111 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787831+0000", 2024-09-21T18:08:08.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3229000000000003e-05 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887202+0000", 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099371371 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887265+0000", 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2718999999999994e-05 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 49.387696949000002, 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:08:08.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:08:08.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.976054654999999, 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.973223783000002, 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.242 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:08:08.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:08:08.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.969250875, 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:08:08.247 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.251 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.966133923999998, 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:08:08.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.964202919999998, 2024-09-21T18:08:08.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:08:08.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.961096161, 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.261 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:08:08.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:08:08.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.957938476999999, 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.266 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:08:08.270 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.955705426000002, 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:08.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.953453752999998, 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.860876165000001, 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:08:08.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:08:08.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:08.283 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.084206789, 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:08.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1098 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'211, mlcod=49'204)", 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 41.977709746000002, 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.135100678, 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248532+0000", 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248535+0000", 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6830000000000001e-06 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248818+0000", 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00028246800000000002 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248848+0000", 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0128999999999999e-05 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.248863+0000", 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5302999999999999e-05 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.380608+0000", 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.131744374 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.380649+0000", 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1430000000000001e-05 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.383553+0000", 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0029036890000000001 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.383632+0000", 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9602000000000001e-05 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.293 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 36.074699205000002, 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:08:08.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:08:08.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:08:08.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.967869225999998, 2024-09-21T18:08:08.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:08:08.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:08:08.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:08:08.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:08:08.307 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 28.634497673999999, 2024-09-21T18:08:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:08:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.308 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:08:08.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:08:08.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 23.862506204999999, 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:08:08.317 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:08.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.98035775, 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.326 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2913 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'440, mlcod=60'435)", 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 13.977991725000001, 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20445745200000001, 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:08.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248253+0000", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0139999999999999e-06 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248265+0000", 2024-09-21T18:08:08.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2289999999999999e-05 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248286+0000", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1285e-05 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248300+0000", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.375e-05 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451311+0000", 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.203011145 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451347+0000", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5355999999999998e-05 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.452613+0000", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001266971 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.452707+0000", 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3640999999999993e-05 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.330 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:08.331 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:08.331 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:08.332 INFO:tasks.rados.rados.0.smithi160.stdout:1067: finishing write tid 1 to smithi16030472-270 2024-09-21T18:08:08.332 INFO:tasks.rados.rados.0.smithi160.stdout:1067: finishing write tid 2 to smithi16030472-270 2024-09-21T18:08:08.332 INFO:tasks.rados.rados.0.smithi160.stdout:1067: finishing write tid 3 to smithi16030472-270 2024-09-21T18:08:08.332 INFO:tasks.rados.rados.0.smithi160.stdout:1067: finishing write tid 4 to smithi16030472-270 2024-09-21T18:08:08.332 INFO:tasks.rados.rados.0.smithi160.stdout:1067: finishing write tid 5 to smithi16030472-270 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1067: oid 270 updating version 0 to 418 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1067: oid 270 updating version 418 to 421 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1067: oid 270 version 421 is already newer than 420 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1067: oid 270 version 421 is already newer than 419 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 270 v 421 (ObjNum 682 snap 0 seq_num 682) dirty exists 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1067: left oid 270 (ObjNum 682 snap 0 seq_num 682) 2024-09-21T18:08:08.333 INFO:tasks.rados.rados.0.smithi160.stdout:1069: expect (ObjNum 631 snap 0 seq_num 631) 2024-09-21T18:08:08.334 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:08.334 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:08.329+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:08.367 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1646334 2024-09-21T18:08:08.423 INFO:tasks.rados.rados.0.smithi160.stdout:1068: finishing copy_from racing read to smithi16030472-347 2024-09-21T18:08:08.423 INFO:tasks.rados.rados.0.smithi160.stdout:1081: got (0) Success 2024-09-21T18:08:08.423 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 0 (ObjNum 580 snap 0 seq_num 580) dirty exists 2024-09-21T18:08:08.423 INFO:tasks.rados.rados.0.smithi160.stdout:1073: expect (ObjNum 671 snap 0 seq_num 671) 2024-09-21T18:08:08.433 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:08.429+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:08.448 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2859510 2024-09-21T18:08:08.468 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:08.468 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:08.468 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:08:08.468 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:08.482 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:08.521 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2155725 2024-09-21T18:08:08.534 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:08.533+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1071: got (0) Success 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 485 v 0 (ObjNum 177 snap 0 seq_num 3138308416) clean dne 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1079: finishing write tid 1 to smithi16030472-295 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1067: done (15 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1069: done (14 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1071: done (13 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1073: done (12 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1081: done (11 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1084: done (10 left) 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1085: read oid 241 snap -1 2024-09-21T18:08:08.538 INFO:tasks.rados.rados.0.smithi160.stdout:1085: expect (ObjNum 240 snap 0 seq_num 240) 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1086: cache_try_flush oid 189 snap -1 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: finishing write tid 2 to smithi16030472-295 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: finishing write tid 3 to smithi16030472-295 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: finishing write tid 4 to smithi16030472-295 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: finishing write tid 5 to smithi16030472-295 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: oid 295 updating version 0 to 504 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: oid 295 updating version 504 to 507 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: oid 295 version 507 is already newer than 505 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: oid 295 version 507 is already newer than 506 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 507 (ObjNum 685 snap 0 seq_num 685) dirty exists 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1079: left oid 295 (ObjNum 685 snap 0 seq_num 685) 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1076: got (0) Success 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 0 (ObjNum 502 snap 0 seq_num 502) dirty exists 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1083: got (16) Device or resource busy 2024-09-21T18:08:08.539 INFO:tasks.rados.rados.0.smithi160.stdout:1074: finishing write tid 1 to smithi16030472-291 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1076: done (11 left) 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1079: done (10 left) 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1083: done (9 left) 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1087: copy_from oid 152 from oid 85 current snap is 0 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1078: finishing copy_from racing read to smithi16030472-430 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1080: finishing copy_from to smithi16030472-233 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1080: got expected ENOENT (src dne) 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1080: finishing copy_from racing read to smithi16030472-233 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1068: finishing copy_from to smithi16030472-347 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 125 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1078: finishing copy_from to smithi16030472-430 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 430 v 325 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1068: done (9 left) 2024-09-21T18:08:08.540 INFO:tasks.rados.rados.0.smithi160.stdout:1078: done (8 left) 2024-09-21T18:08:08.541 INFO:tasks.rados.rados.0.smithi160.stdout:1080: done (7 left) 2024-09-21T18:08:08.541 INFO:tasks.rados.rados.0.smithi160.stdout:1088: cache_try_flush oid 184 snap -1 2024-09-21T18:08:08.541 INFO:tasks.rados.rados.0.smithi160.stdout:1082: expect (ObjNum 674 snap 0 seq_num 674) 2024-09-21T18:08:08.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:08:08.593 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:08.635 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:08.633+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:08.648 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:08.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:08.648 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:08.649 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:08.649 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:08.657 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:08.663 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:08.720 INFO:tasks.rados.rados.0.smithi160.stdout:1077: finishing write tid 1 to smithi16030472-238 2024-09-21T18:08:08.721 INFO:tasks.rados.rados.0.smithi160.stdout:1088: got (2) No such file or directory 2024-09-21T18:08:08.721 INFO:tasks.rados.rados.0.smithi160.stdout:1082: done (7 left) 2024-09-21T18:08:08.721 INFO:tasks.rados.rados.0.smithi160.stdout:1088: done (6 left) 2024-09-21T18:08:08.721 INFO:tasks.rados.rados.0.smithi160.stdout:1089: write (excl) oid 435 current snap is 0 2024-09-21T18:08:08.721 INFO:tasks.rados.rados.0.smithi160.stdout:1089: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2024-09-21T18:08:08.730 INFO:tasks.rados.rados.0.smithi160.stdout:1089: writing smithi16030472-435 from 746409 to 1314242 tid 1 2024-09-21T18:08:08.735 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:08.730+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:08.739 INFO:tasks.rados.rados.0.smithi160.stdout:1089: writing smithi16030472-435 from 1901600 to 2445841 tid 2 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1089: writing smithi16030472-435 from 3065949 to 3546409 tid 3 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1090: cache_evict oid 194 snap -1 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1091: read oid 421 snap -1 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1091: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1092: read oid 126 snap -1 2024-09-21T18:08:08.748 INFO:tasks.rados.rados.0.smithi160.stdout:1092: expect deleted 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1093: copy_from oid 330 from oid 29 current snap is 0 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1094: read oid 369 snap -1 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1094: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1095: read oid 33 snap -1 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1095: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1096: cache_try_flush oid 403 snap -1 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1097: read oid 335 snap -1 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1097: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout:1098: cache_try_flush oid 214 snap -1 2024-09-21T18:08:08.749 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:08.815 INFO:tasks.rados.rados.0.smithi160.stdout:1077: finishing write tid 2 to smithi16030472-238 2024-09-21T18:08:08.827 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:08.827 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:08.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:08.827 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:08:08.827 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:08.837 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:08.833+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:08.842 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:08.849 INFO:tasks.rados.rados.0.smithi160.stdout:1090: got (2) No such file or directory 2024-09-21T18:08:08.849 INFO:tasks.rados.rados.0.smithi160.stdout:1090: done (15 left) 2024-09-21T18:08:08.849 INFO:tasks.rados.rados.0.smithi160.stdout:1099: read oid 167 snap -1 2024-09-21T18:08:08.849 INFO:tasks.rados.rados.0.smithi160.stdout:1099: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout:1089: finishing write tid 1 to smithi16030472-435 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout:1089: finishing write tid 2 to smithi16030472-435 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout:1092: done (15 left) 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout:1100: cache_evict oid 135 snap -1 2024-09-21T18:08:08.850 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:1096: got (2) No such file or directory 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:1098: got (0) Success 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 214 v 0 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:1096: done (15 left) 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:1098: done (14 left) 2024-09-21T18:08:08.851 INFO:tasks.rados.rados.0.smithi160.stdout:1101: cache_flush oid 61 snap -1 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout:1102: cache_try_flush oid 405 snap -1 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout:1101: got (2) No such file or directory 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout:1101: done (15 left) 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout:1103: copy_from oid 196 from oid 460 current snap is 0 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:08.852 INFO:tasks.rados.rados.0.smithi160.stdout:1089: finishing write tid 3 to smithi16030472-435 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: finishing write tid 4 to smithi16030472-435 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: finishing write tid 5 to smithi16030472-435 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: oid 435 updating version 0 to 468 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: oid 435 updating version 468 to 471 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: oid 435 version 471 is already newer than 470 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: oid 435 version 471 is already newer than 469 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 435 v 471 (ObjNum 687 snap 0 seq_num 687) dirty exists 2024-09-21T18:08:08.853 INFO:tasks.rados.rados.0.smithi160.stdout:1089: left oid 435 (ObjNum 687 snap 0 seq_num 687) 2024-09-21T18:08:08.854 INFO:tasks.rados.rados.0.smithi160.stdout:1089: done (15 left) 2024-09-21T18:08:08.854 INFO:tasks.rados.rados.0.smithi160.stdout:1104: delete oid 30 current snap is 0 2024-09-21T18:08:08.915 INFO:tasks.rados.rados.0.smithi160.stdout:1077: finishing write tid 3 to smithi16030472-238 2024-09-21T18:08:08.919 INFO:tasks.rados.rados.0.smithi160.stdout:1077: finishing write tid 4 to smithi16030472-238 2024-09-21T18:08:08.938 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:08.933+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:08.953 INFO:tasks.rados.rados.0.smithi160.stdout:1099: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3595 4.3 4.25e394fb (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.851802+0000", 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.119397872, 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11941835000000001, 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3595 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851802+0000", 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851802+0000", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851804+0000", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.26e-06 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851807+0000", 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.743e-06 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851817+0000", 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8779999999999993e-06 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.851823+0000", 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5219999999999998e-06 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3557 4.3 4:de124e16:::smithi16030472-241:head [sparse-read 0~2033680] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.636001+0000", 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.33519929700000001, 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33526022599999999, 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3557 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636001+0000", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636001+0000", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636002+0000", 2024-09-21T18:08:08.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1969999999999999e-06 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636006+0000", 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8630000000000001e-06 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636054+0000", 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8001999999999999e-05 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636066+0000", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1586999999999999e-05 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636456+0000", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039064999999999998 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636491+0000", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5315000000000003e-05 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.746073+0000", 2024-09-21T18:08:08.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10958113899999999 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.746138+0000", 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5790000000000005e-05 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.11.0:818 4.3 e71/33)", 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.852023+0000", 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.119177556, 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11927635, 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.981 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.852023+0000", 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.852023+0000", 2024-09-21T18:08:08.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.852023+0000", 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.852026+0000", 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7640000000000001e-06 2024-09-21T18:08:08.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.852032+0000", 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7219999999999996e-06 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.952140+0000", 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10010851599999999 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3558 4.3 4:fd21b2cb:::smithi16030472-189:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.636082+0000", 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.33511815499999997, 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33524197500000003, 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3558 2024-09-21T18:08:08.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636082+0000", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636082+0000", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636083+0000", 2024-09-21T18:08:08.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.33e-07 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636087+0000", 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3939999999999998e-06 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636099+0000", 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106000000000001e-05 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636108+0000", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2300000000000008e-06 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636523+0000", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00041554599999999998 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636617+0000", 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3881000000000005e-05 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.988 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3559 4.3 4:eb032882:::smithi16030472-152:head [copy-from ver 62 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.636133+0000", 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.33506749899999999, 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.335229952, 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3559 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636133+0000", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636133+0000", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636134+0000", 2024-09-21T18:08:08.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173e-06 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636140+0000", 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3969999999999999e-06 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636152+0000", 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636159+0000", 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4460000000000004e-06 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.636724+0000", 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056477399999999998 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.993 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3570 4.3 4:e74060bb:::smithi16030472-421:head [read 0~3058940 out=3058940b,checksum in=4b out=8b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747291+0000", 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.22390870700000001, 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.22410316999999999, 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3570 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.994 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747291+0000", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747291+0000", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747292+0000", 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.51e-07 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747303+0000", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1358999999999999e-05 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747309+0000", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8629999999999996e-06 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747314+0000", 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7620000000000003e-06 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.947711+0000", 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.200396982 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:679 3.2 3:70bf26e5:::smithi16030472-249:head [copy-from ver 380 in=56b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.301467+0000", 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.66973317200000004, 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.66995886100000002, 2024-09-21T18:08:08.997 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301467+0000", 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301467+0000", 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301469+0000", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7239999999999999e-06 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301476+0000", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6930000000000007e-06 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301489+0000", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3039e-05 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:08.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.301501+0000", 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1887e-05 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.302432+0000", 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00093054399999999995 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.302728+0000", 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029588599999999998 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3581 4.3 4.20b503af (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.847547+0000", 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.123653058, 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12390596, 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3581 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847547+0000", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847547+0000", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847547+0000", 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7099999999999997e-07 2024-09-21T18:08:09.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847550+0000", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.492e-06 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847555+0000", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6659999999999999e-06 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.847558+0000", 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1219999999999999e-06 2024-09-21T18:08:09.003 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3590 4.3 4.4af8ff0b (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.849035+0000", 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.122164912, 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12244601099999999, 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.004 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3590 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849035+0000", 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849035+0000", 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849037+0000", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1830000000000001e-06 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849045+0000", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1439999999999997e-06 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849056+0000", 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1511e-05 2024-09-21T18:08:09.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849063+0000", 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4139999999999999e-06 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3593 4.3 4.35932d1b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.849936+0000", 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.12126408599999999, 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121570054, 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.007 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3593 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849936+0000", 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849936+0000", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849938+0000", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.807e-06 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849945+0000", 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359999999999999e-06 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849960+0000", 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5469e-05 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.849971+0000", 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0094e-05 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3594 4.3 4.35932d1b (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.010 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.850006+0000", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.121194284, 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121525575, 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3594 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850006+0000", 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850006+0000", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850007+0000", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.091e-06 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850010+0000", 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.489e-06 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850019+0000", 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5739999999999996e-06 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.850027+0000", 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8169999999999999e-06 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 11 2024-09-21T18:08:09.013 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:09.015 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:09.058 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:09.054+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:09.103 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2602811 2024-09-21T18:08:09.120 INFO:tasks.rados.rados.0.smithi160.stdout:1087: finishing copy_from racing read to smithi16030472-152 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: finishing write tid 5 to smithi16030472-238 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: oid 238 updating version 0 to 424 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: oid 238 version 424 is already newer than 422 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: oid 238 version 424 is already newer than 423 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: oid 238 version 424 is already newer than 399 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 238 v 424 (ObjNum 684 snap 0 seq_num 684) dirty exists 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1077: left oid 238 (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1074: finishing write tid 2 to smithi16030472-291 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1074: finishing write tid 3 to smithi16030472-291 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1074: finishing write tid 4 to smithi16030472-291 2024-09-21T18:08:09.121 INFO:tasks.rados.rados.0.smithi160.stdout:1095: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:08:09.139 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:09.135+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:09.146 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:09.174 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:09.179 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:09.179 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:09.180 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:09.180 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:09.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:09.200 INFO:tasks.rados.rados.0.smithi160.stdout:1091: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:09.224 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:09.240 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:09.237+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:09.260 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:09.294 INFO:tasks.rados.rados.0.smithi160.stdout:1087: finishing copy_from to smithi16030472-152 2024-09-21T18:08:09.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 152 v 62 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-09-21T18:08:09.294 INFO:tasks.rados.rados.0.smithi160.stdout:1086: got (0) Success 2024-09-21T18:08:09.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 189 v 0 (ObjNum 188 snap 0 seq_num 188) dirty exists 2024-09-21T18:08:09.294 INFO:tasks.rados.rados.0.smithi160.stdout:1085: expect (ObjNum 240 snap 0 seq_num 240) 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.324 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.175893008000003, 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:09.331 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.175846698999997, 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:09.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:08:09.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.173750785999999, 2024-09-21T18:08:09.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.340 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:08:09.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:08:09.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.173087670999998, 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.347 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:08:09.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:08:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:08:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:08:09.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:08:09.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:08:09.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.359 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.173051172000001, 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.360 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:09.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:08:09.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.039413000000003, 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.367 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:09.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:08:09.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:09.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.038701209000003, 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.374 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:09.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:08:09.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:08:09.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:08:09.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 56.038663646000003, 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.381 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:08:09.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:08:09.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.241277646999997, 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:09.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.241083101000001, 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.392 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:08:09.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.240796713000002, 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:08:09.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.240780633, 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.401 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.211853380999997, 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.405 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:09.409 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.205634306, 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:09.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:08:09.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.200532711000001, 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:08:09.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:08:09.415 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.416 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.417 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.200198763000003, 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.418 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.419 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:08:09.420 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:08:09.421 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.200178284000003, 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.422 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.423 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.424 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.425 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.113849756999997, 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.426 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:09.427 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.428 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.429 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:09.430 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.105389571000003, 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.431 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:08:09.432 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.433 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.434 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.099461865999999, 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.435 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:08:09.436 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.437 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.438 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.439 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.439 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.439 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.439 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:09.440 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:09.443 INFO:tasks.rados.rados.0.smithi160.stdout:1074: finishing write tid 5 to smithi16030472-291 2024-09-21T18:08:09.443 INFO:tasks.rados.rados.0.smithi160.stdout:1074: oid 291 updating version 0 to 398 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1074: oid 291 updating version 398 to 426 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1074: oid 291 version 426 is already newer than 425 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1074: oid 291 updating version 426 to 427 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 291 v 427 (ObjNum 683 snap 0 seq_num 683) dirty exists 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1074: left oid 291 (ObjNum 683 snap 0 seq_num 683) 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1102: got (0) Success 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 405 v 0 (ObjNum 129 snap 0 seq_num 3130056984) clean dne 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1103: finishing copy_from racing read to smithi16030472-196 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1100: got (0) Success 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1103: finishing copy_from to smithi16030472-196 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 196 v 312 (ObjNum 459 snap 0 seq_num 459) dirty exists 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1074: done (15 left) 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1077: done (14 left) 2024-09-21T18:08:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:1085: done (13 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1086: done (12 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1087: done (11 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1091: done (10 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1095: done (9 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1099: done (8 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1100: done (7 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1102: done (6 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1103: done (5 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1104: done (4 left) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1105: read oid 424 snap -1 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1105: expect (ObjNum 579 snap 0 seq_num 579) 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1106: cache_try_flush oid 400 snap -1 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1107: cache_try_flush oid 427 snap -1 2024-09-21T18:08:09.445 INFO:tasks.rados.rados.0.smithi160.stdout:1108: cache_evict oid 132 snap -1 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1109: copy_from oid 364 from oid 167 current snap is 0 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1110: copy_from oid 268 from oid 259 current snap is 0 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1111: read oid 88 snap -1 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1111: expect (ObjNum 87 snap 0 seq_num 87) 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1112: read oid 151 snap -1 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1112: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1113: write (excl) oid 166 current snap is 0 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1113: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1113: writing smithi16030472-166 from 780023 to 1498354 tid 1 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1113: writing smithi16030472-166 from 1902211 to 2450121 tid 2 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1113: writing smithi16030472-166 from 3004442 to 3580023 tid 3 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1114: delete oid 410 current snap is 0 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:1106: got (0) Success 2024-09-21T18:08:09.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 400 v 0 (ObjNum 533 snap 0 seq_num 533) dirty exists 2024-09-21T18:08:09.447 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:09.435+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:09.490 INFO:tasks.rados.rados.0.smithi160.stdout:1113: finishing write tid 1 to smithi16030472-166 2024-09-21T18:08:09.540 INFO:tasks.rados.rados.0.smithi160.stdout:1093: finishing copy_from to smithi16030472-330 2024-09-21T18:08:09.541 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 330 v 13 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-21T18:08:09.541 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:09.537+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:09.562 INFO:tasks.rados.rados.0.smithi160.stdout:1107: got (0) Success 2024-09-21T18:08:09.562 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 0 (ObjNum 426 snap 0 seq_num 426) dirty exists 2024-09-21T18:08:09.562 INFO:tasks.rados.rados.0.smithi160.stdout:1108: got (0) Success 2024-09-21T18:08:09.608 INFO:tasks.rados.rados.0.smithi160.stdout:1113: finishing write tid 2 to smithi16030472-166 2024-09-21T18:08:09.608 INFO:tasks.rados.rados.0.smithi160.stdout:1113: finishing write tid 3 to smithi16030472-166 2024-09-21T18:08:09.615 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:09.615 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3596 4.2 4.96aab486 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.359937+0000", 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.25288447400000003, 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25292669800000001, 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3596 2024-09-21T18:08:09.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.359937+0000", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.359937+0000", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.359941+0000", 2024-09-21T18:08:09.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7270000000000001e-06 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.359955+0000", 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4483e-05 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.359992+0000", 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6875999999999997e-05 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360015+0000", 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3023e-05 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3597 4.2 4.96aab486 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.360053+0000", 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.25276881200000001, 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25290084099999999, 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.619 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3597 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360053+0000", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360053+0000", 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360055+0000", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0229999999999999e-06 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360063+0000", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.32e-06 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360086+0000", 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2705999999999999e-05 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360102+0000", 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6133999999999998e-05 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3598 4.2 4.96aab486 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.360124+0000", 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.25269771299999999, 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.25290310799999999, 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.622 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3598 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360124+0000", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360124+0000", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360126+0000", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.553e-06 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360132+0000", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7329999999999997e-06 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.360149+0000", 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6447e-05 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460292+0000", 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100142853 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3602 4.2 4.d25eab1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.460333+0000", 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.15248899099999999, 2024-09-21T18:08:09.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.152766448, 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3602 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460333+0000", 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460333+0000", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460336+0000", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4970000000000001e-06 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460350+0000", 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3833e-05 2024-09-21T18:08:09.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460367+0000", 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7107000000000001e-05 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460378+0000", 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1209999999999999e-05 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.628 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3603 4.2 4.d25eab1e (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.460396+0000", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.15242557800000001, 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15277539700000001, 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3603 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460396+0000", 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460396+0000", 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460398+0000", 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460404+0000", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4829999999999996e-06 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460425+0000", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0930000000000001e-05 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460437+0000", 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2036e-05 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3604 3.3 3:f04b8987:::smithi16030472-268:head [copy-from ver 429 in=56b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.460853+0000", 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.151969251, 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.152404182, 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460853+0000", 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460853+0000", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460856+0000", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9700000000000001e-06 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460870+0000", 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3223e-05 2024-09-21T18:08:09.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460905+0000", 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4882999999999998e-05 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.460922+0000", 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7149000000000001e-05 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461087+0000", 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000164938 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461343+0000", 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025668300000000001 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:1605 3.3 3:f04b8987:::smithi16030472-268:head [stat] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:09.461074+0000", 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.15174792400000001, 2024-09-21T18:08:09.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.15226647099999999, 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461074+0000", 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461074+0000", 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461076+0000", 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0329999999999998e-06 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461083+0000", 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0230000000000004e-06 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.637 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461097+0000", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4287000000000001e-05 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.461113+0000", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6073e-05 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.561809+0000", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100696041 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.561895+0000", 2024-09-21T18:08:09.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5603e-05 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3575 4.2 4:5e0a3d4f:::smithi16030472-330:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747215+0000", 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86560723100000003, 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.866207493, 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.639 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3575 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747215+0000", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747215+0000", 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747216+0000", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9600000000000008e-07 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747220+0000", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1649999999999999e-06 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747228+0000", 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6349999999999995e-06 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747236+0000", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7840000000000004e-06 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.124915+0000", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37767931300000002 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.124946+0000", 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.027e-05 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.642 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3577 4.2 4:446029b0:::smithi16030472-369:head [sparse-read 1942536~242440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747293+0000", 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86552911300000002, 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86622628800000001, 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3577 2024-09-21T18:08:09.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747293+0000", 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747293+0000", 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747293+0000", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4399999999999999e-07 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747297+0000", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6540000000000001e-06 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747308+0000", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1328000000000001e-05 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747315+0000", 2024-09-21T18:08:09.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8619999999999997e-06 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125237+0000", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37792204699999998 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125317+0000", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0146000000000001e-05 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.439177+0000", 2024-09-21T18:08:09.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.31385915399999997 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.439327+0000", 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000150722 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3578 4.2 4:446029b0:::smithi16030472-369:head [sparse-read 639450~1545526] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747323+0000", 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86549835799999997, 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86630291400000003, 2024-09-21T18:08:09.647 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3578 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747323+0000", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747323+0000", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747324+0000", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8999999999999996e-07 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747326+0000", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1270000000000001e-06 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747330+0000", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0969999999999999e-06 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747335+0000", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8160000000000001e-06 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125404+0000", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37806926400000002 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125445+0000", 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1075999999999998e-05 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.539710+0000", 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41426445099999998 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.539795+0000", 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4795000000000004e-05 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.651 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3583 4.2 4:70fc244d:::smithi16030472-335:head [read 1206442~2407096 out=2407096b,checksum in=4b out=8b,omap-get-vals-by-keys in=4b out=4b,omap-get-keys in=12b out=5b,omap-get-vals in=16b out=5b,omap-get-header,getxattrs out=76b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747406+0000", 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86541553800000004, 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86635416899999995, 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3583 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.652 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747406+0000", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747406+0000", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747407+0000", 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8299999999999997e-07 2024-09-21T18:08:09.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747410+0000", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8710000000000001e-06 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747416+0000", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7730000000000002e-06 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747420+0000", 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0060000000000003e-06 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125524+0000", 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37810323000000001 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3584 4.2 4:70fc244d:::smithi16030472-335:head [sparse-read 0~3613538 out=1778799b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747429+0000", 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86539252499999997, 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86641519099999997, 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.655 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3584 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747429+0000", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747429+0000", 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747430+0000", 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.37e-07 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747433+0000", 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.354e-06 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747442+0000", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8179999999999993e-06 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747449+0000", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6680000000000004e-06 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.125927+0000", 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.37847856600000002 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.658 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:3585 4.2 4:70fc244d:::smithi16030472-335:head [sparse-read 3326899~286639 out=8b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:08.747457+0000", 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.86536519899999997, 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.86647071099999995, 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 3585 2024-09-21T18:08:09.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747457+0000", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747457+0000", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747457+0000", 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.99e-07 2024-09-21T18:08:09.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747461+0000", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2770000000000001e-06 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747469+0000", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9109999999999999e-06 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:08.747476+0000", 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6490000000000002e-06 2024-09-21T18:08:09.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:09.226202+0000", 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.478725717 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 13 2024-09-21T18:08:09.662 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:09.663 INFO:tasks.rados.rados.0.smithi160.stdout:1113: finishing write tid 4 to smithi16030472-166 2024-09-21T18:08:09.663 INFO:tasks.rados.rados.0.smithi160.stdout:1113: finishing write tid 5 to smithi16030472-166 2024-09-21T18:08:09.663 INFO:tasks.rados.rados.0.smithi160.stdout:1113: oid 166 updating version 0 to 439 2024-09-21T18:08:09.663 INFO:tasks.rados.rados.0.smithi160.stdout:1113: oid 166 updating version 439 to 509 2024-09-21T18:08:09.664 INFO:tasks.rados.rados.0.smithi160.stdout:1113: oid 166 version 509 is already newer than 508 2024-09-21T18:08:09.664 INFO:tasks.rados.rados.0.smithi160.stdout:1113: oid 166 updating version 509 to 510 2024-09-21T18:08:09.664 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 510 (ObjNum 689 snap 0 seq_num 689) dirty exists 2024-09-21T18:08:09.664 INFO:tasks.rados.rados.0.smithi160.stdout:1113: left oid 166 (ObjNum 689 snap 0 seq_num 689) 2024-09-21T18:08:09.665 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:09.665 INFO:tasks.rados.rados.0.smithi160.stdout:1094: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-21T18:08:09.665 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:09.661+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:09.700 INFO:tasks.rados.rados.0.smithi160.stdout:1093: finishing copy_from racing read to smithi16030472-330 2024-09-21T18:08:09.701 INFO:tasks.rados.rados.0.smithi160.stdout:1097: expect (ObjNum 334 snap 0 seq_num 334) 2024-09-21T18:08:09.702 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1206442 2024-09-21T18:08:09.764 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:09.761+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:09.795 INFO:tasks.rados.rados.0.smithi160.stdout:1070: got (0) Success 2024-09-21T18:08:09.795 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 249 v 0 (ObjNum 590 snap 0 seq_num 590) dirty exists 2024-09-21T18:08:09.795 INFO:tasks.rados.rados.0.smithi160.stdout:1111: expect (ObjNum 87 snap 0 seq_num 87) 2024-09-21T18:08:09.795 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:09.824 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:09.824 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:09.824 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:09.825 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:09.825 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:09.825 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:08:09.839 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:09.886 INFO:tasks.rados.rados.0.smithi160.stdout:1110: finishing copy_from racing read to smithi16030472-268 2024-09-21T18:08:09.886 INFO:tasks.rados.rados.0.smithi160.stdout:1110: finishing copy_from to smithi16030472-268 2024-09-21T18:08:09.886 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 268 v 429 (ObjNum 537 snap 0 seq_num 537) dirty exists 2024-09-21T18:08:09.886 INFO:tasks.rados.rados.0.smithi160.stdout:1105: expect (ObjNum 579 snap 0 seq_num 579) 2024-09-21T18:08:09.891 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:09.887+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:09.892 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 630073 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.964 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.287044266999999, 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:08:09.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:08:09.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.286973904999996, 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:08:09.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.972 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.275172470000001, 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.267306087000001, 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.977 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:08:09.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:08:09.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:08:09.980 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.266920442999997, 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.981 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:09.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:08:09.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:08:09.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 51.266874993999998, 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.985 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.990 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 50.218977076999998, 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.991 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:08:09.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:08:09.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 50.218869435000002, 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.995 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:09.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:09.998 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 44.643086820000001, 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:09.999 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:08:10.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:08:10.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:08:10.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 44.643034884999999, 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.005 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.007 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:10.010 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 43.876330496999998, 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:10.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.016 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 43.647227239000003, 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:08:10.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:08:10.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:08:10.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:08:10.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 43.646799248999997, 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.023 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:08:10.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:08:10.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:08:10.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 43.646752679999999, 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:10.035 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 39.146587570000001, 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:08:10.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:08:10.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.041 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 39.146083619999999, 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.138551823999997, 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.047 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:10.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.053 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 32.138477504000001, 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:08:10.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:08:10.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 31.736674259000001, 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.059 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:08:10.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:10.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:10.063 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 31.634324371000002, 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:10.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:10.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:10.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:10.069 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:10.070 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:10.066+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:10.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:10.076 INFO:tasks.rados.rados.0.smithi160.stdout:1109: finishing copy_from racing read to smithi16030472-364 2024-09-21T18:08:10.076 INFO:tasks.rados.rados.0.smithi160.stdout:1109: finishing copy_from to smithi16030472-364 2024-09-21T18:08:10.076 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 503 (ObjNum 670 snap 0 seq_num 670) dirty exists 2024-09-21T18:08:10.077 INFO:tasks.rados.rados.0.smithi160.stdout:1112: expect (ObjNum 150 snap 0 seq_num 150) 2024-09-21T18:08:10.143 INFO:tasks.rados.rados.0.smithi160.stdout:1070: done (13 left) 2024-09-21T18:08:10.143 INFO:tasks.rados.rados.0.smithi160.stdout:1093: done (12 left) 2024-09-21T18:08:10.143 INFO:tasks.rados.rados.0.smithi160.stdout:1094: done (11 left) 2024-09-21T18:08:10.143 INFO:tasks.rados.rados.0.smithi160.stdout:1097: done (10 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1105: done (9 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1106: done (8 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1107: done (7 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1108: done (6 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1109: done (5 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1110: done (4 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1111: done (3 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1112: done (2 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1113: done (1 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1114: done (0 left) 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1115: write oid 383 current snap is 0 2024-09-21T18:08:10.144 INFO:tasks.rados.rados.0.smithi160.stdout:1115: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-21T18:08:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:1115: writing smithi16030472-383 from 413637 to 882466 tid 1 2024-09-21T18:08:10.171 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:10.167+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:10.176 INFO:tasks.rados.rados.0.smithi160.stdout:1115: writing smithi16030472-383 from 1619175 to 2054401 tid 2 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1115: writing smithi16030472-383 from 2542935 to 3301190 tid 3 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1116: cache_evict oid 268 snap -1 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1117: read oid 453 snap -1 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1117: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1118: copy_from oid 363 from oid 342 current snap is 0 2024-09-21T18:08:10.196 INFO:tasks.rados.rados.0.smithi160.stdout:1115: finishing write tid 1 to smithi16030472-383 2024-09-21T18:08:10.197 INFO:tasks.rados.rados.0.smithi160.stdout:1119: delete oid 1 current snap is 0 2024-09-21T18:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:10.207 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:10.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:10.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:10.271 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:10.269+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:10.298 INFO:tasks.rados.rados.0.smithi160.stdout:1115: finishing write tid 2 to smithi16030472-383 2024-09-21T18:08:10.298 INFO:tasks.rados.rados.0.smithi160.stdout:1116: got (0) Success 2024-09-21T18:08:10.373 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:10.366+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:10.383 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:10.385 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:10.385+0000 7fca4e93e640 -1 osd.2 71 handle_osd_map injecting map crc failure 2024-09-21T18:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:10.400 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:10.401 INFO:tasks.rados.rados.0.smithi160.stdout:1116: done (4 left) 2024-09-21T18:08:10.401 INFO:tasks.rados.rados.0.smithi160.stdout:1119: done (3 left) 2024-09-21T18:08:10.401 INFO:tasks.rados.rados.0.smithi160.stdout:1120: read oid 259 snap -1 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1120: expect (ObjNum 537 snap 0 seq_num 537) 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1121: copy_from oid 112 from oid 406 current snap is 0 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1122: copy_from oid 114 from oid 186 current snap is 0 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1123: cache_try_flush oid 295 snap -1 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1124: cache_evict oid 60 snap -1 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1125: cache_evict oid 220 snap -1 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1126: cache_flush oid 79 snap -1 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1127: write oid 261 current snap is 0 2024-09-21T18:08:10.402 INFO:tasks.rados.rados.0.smithi160.stdout:1127: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-21T18:08:10.404 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:10.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:10.422 INFO:tasks.rados.rados.0.smithi160.stdout:1127: writing smithi16030472-261 from 447251 to 1182931 tid 1 2024-09-21T18:08:10.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:10.431 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:10.435 INFO:tasks.rados.rados.0.smithi160.stdout:1127: writing smithi16030472-261 from 1736139 to 2175034 tid 2 2024-09-21T18:08:10.451 INFO:tasks.rados.rados.0.smithi160.stdout:1127: writing smithi16030472-261 from 2597781 to 3378580 tid 3 2024-09-21T18:08:10.451 INFO:tasks.rados.rados.0.smithi160.stdout:1128: delete oid 123 current snap is 0 2024-09-21T18:08:10.451 INFO:tasks.rados.rados.0.smithi160.stdout:1124: got (16) Device or resource busy 2024-09-21T18:08:10.451 INFO:tasks.rados.rados.0.smithi160.stdout:1121: finishing copy_from racing read to smithi16030472-112 2024-09-21T18:08:10.451 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:10.468 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:10.468 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:10.473 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:10.473+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:10.488 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:10.488 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:10.505 INFO:tasks.rados.rados.0.smithi160.stdout:1117: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-21T18:08:10.505 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:10.509 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:10.509 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:10.528 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:10.528 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:10.538 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:10.549 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:10.549 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:10.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:10.570 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:10.574 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:10.569+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:10.590 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:10.590 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:10.595 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:10.608 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:10.602+0000 7fb260c0e640 -1 osd.11 71 handle_osd_map injecting map crc failure 2024-09-21T18:08:10.609 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:10.609 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:10.632 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:10.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:10.653 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:10.653 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:10.665 INFO:tasks.rados.rados.0.smithi160.stdout:1115: finishing write tid 3 to smithi16030472-383 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: finishing write tid 4 to smithi16030472-383 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: finishing write tid 5 to smithi16030472-383 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: oid 383 updating version 0 to 475 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: oid 383 version 475 is already newer than 473 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: oid 383 version 475 is already newer than 474 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: oid 383 version 475 is already newer than 472 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 475 (ObjNum 691 snap 0 seq_num 691) dirty exists 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1115: left oid 383 (ObjNum 691 snap 0 seq_num 691) 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1122: finishing copy_from racing read to smithi16030472-114 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:1120: expect (ObjNum 537 snap 0 seq_num 537) 2024-09-21T18:08:10.666 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:10.675 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:10.675 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:10.675 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:10.670+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:10.679 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:10.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:10.712 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:10.775 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:10.770+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:10.787 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3007308 2024-09-21T18:08:10.787 INFO:tasks.rados.rados.0.smithi160.stdout:1118: finishing copy_from racing read to smithi16030472-363 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:1125: got (0) Success 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:1121: finishing copy_from to smithi16030472-112 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 112 v 347 (ObjNum 547 snap 0 seq_num 547) dirty exists 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:1123: got (0) Success 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 685 snap 0 seq_num 685) dirty exists 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:1122: finishing copy_from to smithi16030472-114 2024-09-21T18:08:10.788 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 125 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:08:10.808 INFO:tasks.rados.rados.0.smithi160.stdout:1126: got (0) Success 2024-09-21T18:08:10.808 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:08:10.822 INFO:tasks.rados.rados.0.smithi160.stdout:1118: finishing copy_from to smithi16030472-363 2024-09-21T18:08:10.822 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 229 (ObjNum 341 snap 0 seq_num 341) dirty exists 2024-09-21T18:08:10.837 INFO:tasks.rados.rados.0.smithi160.stdout:1127: finishing write tid 1 to smithi16030472-261 2024-09-21T18:08:10.838 INFO:tasks.rados.rados.0.smithi160.stdout:1127: finishing write tid 2 to smithi16030472-261 2024-09-21T18:08:10.873 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:10.873 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:10.876 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:10.873+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:10.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:10.939 INFO:tasks.rados.rados.0.smithi160.stdout:1127: finishing write tid 3 to smithi16030472-261 2024-09-21T18:08:10.939 INFO:tasks.rados.rados.0.smithi160.stdout:1127: finishing write tid 4 to smithi16030472-261 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: finishing write tid 5 to smithi16030472-261 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: oid 261 updating version 0 to 402 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: oid 261 updating version 402 to 403 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: oid 261 version 403 is already newer than 400 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: oid 261 version 403 is already newer than 401 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 261 v 403 (ObjNum 693 snap 0 seq_num 693) dirty exists 2024-09-21T18:08:10.940 INFO:tasks.rados.rados.0.smithi160.stdout:1127: left oid 261 (ObjNum 693 snap 0 seq_num 693) 2024-09-21T18:08:10.977 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:10.971+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:11.016 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.016 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:11.040 INFO:tasks.rados.rados.0.smithi160.stdout:1115: done (11 left) 2024-09-21T18:08:11.040 INFO:tasks.rados.rados.0.smithi160.stdout:1117: done (10 left) 2024-09-21T18:08:11.040 INFO:tasks.rados.rados.0.smithi160.stdout:1118: done (9 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1120: done (8 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1121: done (7 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1122: done (6 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1123: done (5 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1124: done (4 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1125: done (3 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1126: done (2 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1127: done (1 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1128: done (0 left) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1129: cache_evict oid 95 snap -1 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1130: cache_try_flush oid 294 snap -1 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1131: read oid 92 snap -1 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1131: expect (ObjNum 636 snap 0 seq_num 636) 2024-09-21T18:08:11.041 INFO:tasks.rados.rados.0.smithi160.stdout:1132: read oid 486 snap -1 2024-09-21T18:08:11.042 INFO:tasks.rados.rados.0.smithi160.stdout:1132: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:08:11.042 INFO:tasks.rados.rados.0.smithi160.stdout:1133: write (excl) oid 12 current snap is 0 2024-09-21T18:08:11.042 INFO:tasks.rados.rados.0.smithi160.stdout:1133: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-21T18:08:11.052 INFO:tasks.rados.rados.0.smithi160.stdout:1133: writing smithi16030472-12 from 480865 to 967043 tid 1 2024-09-21T18:08:11.063 INFO:tasks.rados.rados.0.smithi160.stdout:1133: writing smithi16030472-12 from 1453103 to 1895667 tid 2 2024-09-21T18:08:11.078 INFO:tasks.rados.rados.0.smithi160.stdout:1133: writing smithi16030472-12 from 2652627 to 3172323 tid 3 2024-09-21T18:08:11.078 INFO:tasks.rados.rados.0.smithi160.stdout:1134: cache_flush oid 238 snap -1 2024-09-21T18:08:11.078 INFO:tasks.rados.rados.0.smithi160.stdout:1135: write oid 4 current snap is 0 2024-09-21T18:08:11.078 INFO:tasks.rados.rados.0.smithi160.stdout:1135: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-21T18:08:11.089 INFO:tasks.rados.rados.0.smithi160.stdout:1135: writing smithi16030472-4 from 497672 to 1059099 tid 1 2024-09-21T18:08:11.103 INFO:tasks.rados.rados.0.smithi160.stdout:1135: writing smithi16030472-4 from 1795232 to 2497807 tid 2 2024-09-21T18:08:11.114 INFO:tasks.rados.rados.0.smithi160.stdout:1135: writing smithi16030472-4 from 2963697 to 3694665 tid 3 2024-09-21T18:08:11.114 INFO:tasks.rados.rados.0.smithi160.stdout:1136: read oid 200 snap -1 2024-09-21T18:08:11.114 INFO:tasks.rados.rados.0.smithi160.stdout:1136: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-21T18:08:11.114 INFO:tasks.rados.rados.0.smithi160.stdout:1137: write oid 496 current snap is 0 2024-09-21T18:08:11.114 INFO:tasks.rados.rados.0.smithi160.stdout:1137: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-21T18:08:11.119 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:11.115+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:11.123 INFO:tasks.rados.rados.0.smithi160.stdout:1137: writing smithi16030472-496 from 514479 to 1151155 tid 1 2024-09-21T18:08:11.130 INFO:tasks.rados.rados.0.smithi160.stdout:1137: writing smithi16030472-496 from 1853714 to 2299947 tid 2 2024-09-21T18:08:11.138 INFO:tasks.rados.rados.0.smithi160.stdout:1137: writing smithi16030472-496 from 2991120 to 3533360 tid 3 2024-09-21T18:08:11.138 INFO:tasks.rados.rados.0.smithi160.stdout:1138: cache_evict oid 276 snap -1 2024-09-21T18:08:11.138 INFO:tasks.rados.rados.0.smithi160.stdout:1129: got (0) Success 2024-09-21T18:08:11.139 INFO:tasks.rados.rados.0.smithi160.stdout:1129: done (9 left) 2024-09-21T18:08:11.139 INFO:tasks.rados.rados.0.smithi160.stdout:1139: write (excl) oid 269 current snap is 0 2024-09-21T18:08:11.139 INFO:tasks.rados.rados.0.smithi160.stdout:1139: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-21T18:08:11.146 INFO:tasks.rados.rados.0.smithi160.stdout:1139: writing smithi16030472-269 from 531286 to 1243211 tid 1 2024-09-21T18:08:11.154 INFO:tasks.rados.rados.0.smithi160.stdout:1139: writing smithi16030472-269 from 1912196 to 2618440 tid 2 2024-09-21T18:08:11.161 INFO:tasks.rados.rados.0.smithi160.stdout:1139: writing smithi16030472-269 from 3018543 to 3731286 tid 3 2024-09-21T18:08:11.161 INFO:tasks.rados.rados.0.smithi160.stdout:1140: cache_evict oid 192 snap -1 2024-09-21T18:08:11.161 INFO:tasks.rados.rados.0.smithi160.stdout:1141: cache_try_flush oid 438 snap -1 2024-09-21T18:08:11.161 INFO:tasks.rados.rados.0.smithi160.stdout:1142: write oid 248 current snap is 0 2024-09-21T18:08:11.161 INFO:tasks.rados.rados.0.smithi160.stdout:1142: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-21T18:08:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:1142: writing smithi16030472-248 from 548093 to 1335267 tid 1 2024-09-21T18:08:11.175 INFO:tasks.rados.rados.0.smithi160.stdout:1142: writing smithi16030472-248 from 1970678 to 2420580 tid 2 2024-09-21T18:08:11.178 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:11.174+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:11.183 INFO:tasks.rados.rados.0.smithi160.stdout:1142: writing smithi16030472-248 from 3045966 to 3727103 tid 3 2024-09-21T18:08:11.183 INFO:tasks.rados.rados.0.smithi160.stdout:1143: delete oid 242 current snap is 0 2024-09-21T18:08:11.205 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.206 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:1135: finishing write tid 1 to smithi16030472-4 2024-09-21T18:08:11.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:11.247 INFO:tasks.rados.rados.0.smithi160.stdout:1131: expect (ObjNum 636 snap 0 seq_num 636) 2024-09-21T18:08:11.279 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:11.274+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:11.300 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:11.346 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 151963 2024-09-21T18:08:11.376 INFO:tasks.rados.rados.0.smithi160.stdout:1138: got (16) Device or resource busy 2024-09-21T18:08:11.376 INFO:tasks.rados.rados.0.smithi160.stdout:1135: finishing write tid 2 to smithi16030472-4 2024-09-21T18:08:11.376 INFO:tasks.rados.rados.0.smithi160.stdout:1139: finishing write tid 1 to smithi16030472-269 2024-09-21T18:08:11.379 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:11.375+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:11.399 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.400 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:11.448 INFO:tasks.rados.rados.0.smithi160.stdout:1140: got (2) No such file or directory 2024-09-21T18:08:11.449 INFO:tasks.rados.rados.0.smithi160.stdout:1132: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:08:11.449 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:11.481 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:11.477+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:11.562 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2066434 2024-09-21T18:08:11.563 INFO:tasks.rados.rados.0.smithi160.stdout:1136: expect (ObjNum 199 snap 0 seq_num 199) 2024-09-21T18:08:11.581 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.581 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.581 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3136123 2024-09-21T18:08:11.581 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:11.573+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:11.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:11.626 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2888031 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1137: finishing write tid 1 to smithi16030472-496 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: finishing write tid 3 to smithi16030472-4 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: finishing write tid 4 to smithi16030472-4 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: finishing write tid 5 to smithi16030472-4 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: oid 4 updating version 0 to 507 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: oid 4 version 507 is already newer than 506 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: oid 4 version 507 is already newer than 504 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: oid 4 version 507 is already newer than 505 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 4 v 507 (ObjNum 696 snap 0 seq_num 696) dirty exists 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1135: left oid 4 (ObjNum 696 snap 0 seq_num 696) 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1142: finishing write tid 1 to smithi16030472-248 2024-09-21T18:08:11.637 INFO:tasks.rados.rados.0.smithi160.stdout:1142: finishing write tid 2 to smithi16030472-248 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: finishing write tid 3 to smithi16030472-248 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: finishing write tid 4 to smithi16030472-248 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: finishing write tid 5 to smithi16030472-248 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: oid 248 updating version 0 to 508 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: oid 248 updating version 508 to 511 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: oid 248 version 511 is already newer than 509 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: oid 248 version 511 is already newer than 510 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 248 v 511 (ObjNum 699 snap 0 seq_num 699) dirty exists 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1142: left oid 248 (ObjNum 699 snap 0 seq_num 699) 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1130: got (0) Success 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 0 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-21T18:08:11.638 INFO:tasks.rados.rados.0.smithi160.stdout:1139: finishing write tid 2 to smithi16030472-269 2024-09-21T18:08:11.654 INFO:tasks.rados.rados.0.smithi160.stdout:1139: finishing write tid 3 to smithi16030472-269 2024-09-21T18:08:11.654 INFO:tasks.rados.rados.0.smithi160.stdout:1133: finishing write tid 1 to smithi16030472-12 2024-09-21T18:08:11.654 INFO:tasks.rados.rados.0.smithi160.stdout:1133: finishing write tid 2 to smithi16030472-12 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1137: finishing write tid 2 to smithi16030472-496 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1130: done (13 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1131: done (12 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1132: done (11 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1135: done (10 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1136: done (9 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1138: done (8 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1140: done (7 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1142: done (6 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1143: done (5 left) 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1144: cache_flush oid 282 snap -1 2024-09-21T18:08:11.655 INFO:tasks.rados.rados.0.smithi160.stdout:1133: finishing write tid 3 to smithi16030472-12 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1145: read oid 49 snap -1 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1145: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1146: cache_flush oid 386 snap -1 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: finishing write tid 4 to smithi16030472-12 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: finishing write tid 5 to smithi16030472-12 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: oid 12 updating version 0 to 500 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: oid 12 version 500 is already newer than 499 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: oid 12 updating version 500 to 501 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: oid 12 updating version 501 to 502 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 12 v 502 (ObjNum 695 snap 0 seq_num 695) dirty exists 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: left oid 12 (ObjNum 695 snap 0 seq_num 695) 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1133: done (7 left) 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1147: read oid 384 snap -1 2024-09-21T18:08:11.656 INFO:tasks.rados.rados.0.smithi160.stdout:1147: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-21T18:08:11.657 INFO:tasks.rados.rados.0.smithi160.stdout:1148: write (excl) oid 40 current snap is 0 2024-09-21T18:08:11.657 INFO:tasks.rados.rados.0.smithi160.stdout:1139: finishing write tid 4 to smithi16030472-269 2024-09-21T18:08:11.657 INFO:tasks.rados.rados.0.smithi160.stdout:1148: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-21T18:08:11.663 INFO:tasks.rados.rados.0.smithi160.stdout:1148: writing smithi16030472-40 from 581707 to 1235732 tid 1 2024-09-21T18:08:11.670 INFO:tasks.rados.rados.0.smithi160.stdout:1148: writing smithi16030472-40 from 1687642 to 2141213 tid 2 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1148: writing smithi16030472-40 from 2700812 to 3404493 tid 3 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1149: cache_try_flush oid 431 snap -1 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1150: read oid 107 snap -1 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1150: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1151: copy_from oid 457 from oid 272 current snap is 0 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1139: finishing write tid 5 to smithi16030472-269 2024-09-21T18:08:11.681 INFO:tasks.rados.rados.0.smithi160.stdout:1139: oid 269 updating version 0 to 513 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1139: oid 269 updating version 513 to 514 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1139: oid 269 version 514 is already newer than 512 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1139: oid 269 updating version 514 to 515 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 269 v 515 (ObjNum 698 snap 0 seq_num 698) dirty exists 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1139: left oid 269 (ObjNum 698 snap 0 seq_num 698) 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1144: got (2) No such file or directory 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1146: got (2) No such file or directory 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1148: finishing write tid 1 to smithi16030472-40 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1148: finishing write tid 2 to smithi16030472-40 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1139: done (11 left) 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1144: done (10 left) 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1146: done (9 left) 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1152: cache_flush oid 222 snap -1 2024-09-21T18:08:11.682 INFO:tasks.rados.rados.0.smithi160.stdout:1153: delete oid 320 current snap is 0 2024-09-21T18:08:11.683 INFO:tasks.rados.rados.0.smithi160.stdout:1149: got (0) Success 2024-09-21T18:08:11.683 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 431 v 0 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:08:11.683 INFO:tasks.rados.rados.0.smithi160.stdout:1152: got (2) No such file or directory 2024-09-21T18:08:11.684 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:11.679+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:11.718 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.718 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:11.785 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:11.777+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:11.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:08:11.859 INFO:tasks.rados.rados.0.smithi160.stdout:1145: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T18:08:11.863 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1459361 2024-09-21T18:08:11.885 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:11.881+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:11.895 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:11.896 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:11.896 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:11.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:11.937 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:11.968 INFO:tasks.rados.rados.0.smithi160.stdout:1137: finishing write tid 3 to smithi16030472-496 2024-09-21T18:08:11.968 INFO:tasks.rados.rados.0.smithi160.stdout:1137: finishing write tid 4 to smithi16030472-496 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: finishing write tid 5 to smithi16030472-496 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: oid 496 updating version 0 to 514 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: oid 496 version 514 is already newer than 512 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: oid 496 version 514 is already newer than 511 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: oid 496 version 514 is already newer than 513 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 514 (ObjNum 697 snap 0 seq_num 697) dirty exists 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1137: left oid 496 (ObjNum 697 snap 0 seq_num 697) 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1141: got (0) Success 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 438 v 0 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1148: finishing write tid 3 to smithi16030472-40 2024-09-21T18:08:11.969 INFO:tasks.rados.rados.0.smithi160.stdout:1147: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-21T18:08:11.986 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:11.985+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:11.989 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 872777 2024-09-21T18:08:12.030 INFO:tasks.rados.rados.0.smithi160.stdout:1134: got (0) Success 2024-09-21T18:08:12.031 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 238 v 0 (ObjNum 684 snap 0 seq_num 684) dirty exists 2024-09-21T18:08:12.031 INFO:tasks.rados.rados.0.smithi160.stdout:1150: expect (ObjNum 106 snap 0 seq_num 106) 2024-09-21T18:08:12.037 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.037 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.053 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:12.087 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:12.085+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:12.132 INFO:tasks.rados.rados.0.smithi160.stdout:1151: finishing copy_from racing read to smithi16030472-457 2024-09-21T18:08:12.132 INFO:tasks.rados.rados.0.smithi160.stdout:1148: finishing write tid 4 to smithi16030472-40 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: finishing write tid 5 to smithi16030472-40 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: oid 40 updating version 0 to 516 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: oid 40 updating version 516 to 517 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: oid 40 updating version 517 to 518 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: oid 40 updating version 518 to 519 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 519 (ObjNum 701 snap 0 seq_num 701) dirty exists 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: left oid 40 (ObjNum 701 snap 0 seq_num 701) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1134: done (10 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1137: done (9 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1141: done (8 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1145: done (7 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1147: done (6 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1148: done (5 left) 2024-09-21T18:08:12.133 INFO:tasks.rados.rados.0.smithi160.stdout:1149: done (4 left) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1150: done (3 left) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1152: done (2 left) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1153: done (1 left) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1154: cache_flush oid 449 snap -1 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1155: cache_flush oid 133 snap -1 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1156: read oid 340 snap -1 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1156: expect (ObjNum 339 snap 0 seq_num 339) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1157: copy_from oid 456 from oid 168 current snap is 0 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1158: read oid 307 snap -1 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1158: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1159: read oid 320 snap -1 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1159: expect deleted 2024-09-21T18:08:12.134 INFO:tasks.rados.rados.0.smithi160.stdout:1160: delete oid 392 current snap is 0 2024-09-21T18:08:12.187 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:12.185+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:12.188 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.188 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.203 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:12.233 INFO:tasks.rados.rados.0.smithi160.stdout:1154: got (0) Success 2024-09-21T18:08:12.233 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 0 (ObjNum 611 snap 0 seq_num 611) dirty exists 2024-09-21T18:08:12.233 INFO:tasks.rados.rados.0.smithi160.stdout:1155: got (2) No such file or directory 2024-09-21T18:08:12.236 INFO:tasks.rados.rados.0.smithi160.stdout:1154: done (7 left) 2024-09-21T18:08:12.236 INFO:tasks.rados.rados.0.smithi160.stdout:1155: done (6 left) 2024-09-21T18:08:12.236 INFO:tasks.rados.rados.0.smithi160.stdout:1160: done (5 left) 2024-09-21T18:08:12.236 INFO:tasks.rados.rados.0.smithi160.stdout:1161: write (excl) oid 342 current snap is 0 2024-09-21T18:08:12.236 INFO:tasks.rados.rados.0.smithi160.stdout:1161: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-21T18:08:12.254 INFO:tasks.rados.rados.0.smithi160.stdout:1161: writing smithi16030472-342 from 632128 to 1111900 tid 1 2024-09-21T18:08:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:1161: writing smithi16030472-342 from 1746735 to 2347633 tid 2 2024-09-21T18:08:12.288 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:12.285+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1161: writing smithi16030472-342 from 3066728 to 3720578 tid 3 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1162: cache_evict oid 296 snap -1 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1163: read oid 426 snap -1 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1163: expect (ObjNum 425 snap 0 seq_num 425) 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1164: write (excl) oid 341 current snap is 0 2024-09-21T18:08:12.295 INFO:tasks.rados.rados.0.smithi160.stdout:1164: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-09-21T18:08:12.308 INFO:tasks.rados.rados.0.smithi160.stdout:1164: writing smithi16030472-341 from 648935 to 1203956 tid 1 2024-09-21T18:08:12.321 INFO:tasks.rados.rados.0.smithi160.stdout:1164: writing smithi16030472-341 from 1805217 to 2266126 tid 2 2024-09-21T18:08:12.323 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.323 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.331 INFO:tasks.rados.rados.0.smithi160.stdout:1164: writing smithi16030472-341 from 2694151 to 3159273 tid 3 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1164: writing smithi16030472-341 from 3817442 to 3848935 tid 4 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1165: delete oid 305 current snap is 0 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1157: finishing copy_from to smithi16030472-456 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1157: got expected ENOENT (src dne) 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1162: got (16) Device or resource busy 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1157: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:08:12.339 INFO:tasks.rados.rados.0.smithi160.stdout:1156: expect (ObjNum 339 snap 0 seq_num 339) 2024-09-21T18:08:12.340 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:12.427 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:12.425+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:12.428 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3493149 2024-09-21T18:08:12.441 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2596870 2024-09-21T18:08:12.458 INFO:tasks.rados.rados.0.smithi160.stdout:1158: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:12.490 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:12.485+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:12.500 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:12.503 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.503 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.518 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:12.591 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:12.589+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:12.592 INFO:tasks.rados.rados.0.smithi160.stdout:1151: finishing copy_from to smithi16030472-457 2024-09-21T18:08:12.592 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 457 v 421 (ObjNum 520 snap 0 seq_num 520) dirty exists 2024-09-21T18:08:12.592 INFO:tasks.rados.rados.0.smithi160.stdout:1163: expect (ObjNum 425 snap 0 seq_num 425) 2024-09-21T18:08:12.592 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:12.659 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:12.679 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.679 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:12.692 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:12.693 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:12.689+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 1 to smithi16030472-341 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 2 to smithi16030472-341 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: finishing write tid 1 to smithi16030472-342 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: finishing write tid 2 to smithi16030472-342 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: finishing write tid 3 to smithi16030472-342 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: finishing write tid 4 to smithi16030472-342 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: finishing write tid 5 to smithi16030472-342 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: oid 342 updating version 0 to 512 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: oid 342 updating version 512 to 514 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: oid 342 version 514 is already newer than 513 2024-09-21T18:08:12.694 INFO:tasks.rados.rados.0.smithi160.stdout:1161: oid 342 updating version 514 to 515 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 515 (ObjNum 704 snap 0 seq_num 704) dirty exists 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1161: left oid 342 (ObjNum 704 snap 0 seq_num 704) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1151: done (9 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1156: done (8 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1157: done (7 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1158: done (6 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1159: done (5 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1161: done (4 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1162: done (3 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1163: done (2 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1165: done (1 left) 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1166: copy_from oid 186 from oid 310 current snap is 0 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1167: copy_from oid 114 from oid 390 current snap is 0 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1168: cache_flush oid 183 snap -1 2024-09-21T18:08:12.695 INFO:tasks.rados.rados.0.smithi160.stdout:1169: write oid 113 current snap is 0 2024-09-21T18:08:12.696 INFO:tasks.rados.rados.0.smithi160.stdout:1169: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-21T18:08:12.709 INFO:tasks.rados.rados.0.smithi160.stdout:1169: writing smithi16030472-113 from 682549 to 1388068 tid 1 2024-09-21T18:08:12.721 INFO:tasks.rados.rados.0.smithi160.stdout:1169: writing smithi16030472-113 from 1922181 to 2386759 tid 2 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1169: writing smithi16030472-113 from 3148997 to 3636663 tid 3 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1170: cache_flush oid 296 snap -1 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1171: cache_try_flush oid 304 snap -1 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1172: read oid 408 snap -1 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1172: expect (ObjNum 654 snap 0 seq_num 654) 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1173: read oid 329 snap -1 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1173: expect (ObjNum 510 snap 0 seq_num 510) 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1174: write (excl) oid 45 current snap is 0 2024-09-21T18:08:12.736 INFO:tasks.rados.rados.0.smithi160.stdout:1174: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-21T18:08:12.748 INFO:tasks.rados.rados.0.smithi160.stdout:1174: writing smithi16030472-45 from 699356 to 1196477 tid 1 2024-09-21T18:08:12.761 INFO:tasks.rados.rados.0.smithi160.stdout:1174: writing smithi16030472-45 from 1980663 to 2588899 tid 2 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1174: writing smithi16030472-45 from 3176420 to 3591711 tid 3 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1175: copy_from oid 326 from oid 396 current snap is 0 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1170: got (0) Success 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 296 v 0 (ObjNum 273 snap 0 seq_num 0) clean dne 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1170: done (10 left) 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1176: write (excl) oid 207 current snap is 0 2024-09-21T18:08:12.769 INFO:tasks.rados.rados.0.smithi160.stdout:1176: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-21T18:08:12.778 INFO:tasks.rados.rados.0.smithi160.stdout:1176: writing smithi16030472-207 from 716163 to 1288533 tid 1 2024-09-21T18:08:12.787 INFO:tasks.rados.rados.0.smithi160.stdout:1176: writing smithi16030472-207 from 2039145 to 2507392 tid 2 2024-09-21T18:08:12.793 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:12.789+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1176: writing smithi16030472-207 from 3203843 to 3830406 tid 3 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1177: cache_try_flush oid 267 snap -1 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1168: got (0) Success 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 0 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 3 to smithi16030472-341 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1168: done (11 left) 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1178: read oid 473 snap -1 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1178: expect deleted 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1179: cache_try_flush oid 392 snap -1 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1180: cache_evict oid 430 snap -1 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1181: write (excl) oid 139 current snap is 0 2024-09-21T18:08:12.798 INFO:tasks.rados.rados.0.smithi160.stdout:1181: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-21T18:08:12.808 INFO:tasks.rados.rados.0.smithi160.stdout:1181: writing smithi16030472-139 from 732970 to 1380589 tid 1 2024-09-21T18:08:12.811 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.811 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.817 INFO:tasks.rados.rados.0.smithi160.stdout:1181: writing smithi16030472-139 from 2097627 to 2709532 tid 2 2024-09-21T18:08:12.824 INFO:tasks.rados.rados.0.smithi160.stdout:1181: writing smithi16030472-139 from 3231266 to 3669101 tid 3 2024-09-21T18:08:12.824 INFO:tasks.rados.rados.0.smithi160.stdout:1182: write (excl) oid 243 current snap is 0 2024-09-21T18:08:12.824 INFO:tasks.rados.rados.0.smithi160.stdout:1182: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-21T18:08:12.826 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:1182: writing smithi16030472-243 from 749777 to 1472645 tid 1 2024-09-21T18:08:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:08:12.840 INFO:tasks.rados.rados.0.smithi160.stdout:1182: writing smithi16030472-243 from 2156109 to 2628025 tid 2 2024-09-21T18:08:12.840 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:12.841 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:12.847 INFO:tasks.rados.rados.0.smithi160.stdout:1182: writing smithi16030472-243 from 3258689 to 3907796 tid 3 2024-09-21T18:08:12.847 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:12.861 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:12.861 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:12.881 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:12.881 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:12.893 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:12.889+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:12.895 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 4 to smithi16030472-341 2024-09-21T18:08:12.896 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 5 to smithi16030472-341 2024-09-21T18:08:12.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:12.902 INFO:tasks.rados.rados.0.smithi160.stdout:1169: finishing write tid 1 to smithi16030472-113 2024-09-21T18:08:12.902 INFO:tasks.rados.rados.0.smithi160.stdout:1169: finishing write tid 2 to smithi16030472-113 2024-09-21T18:08:12.904 INFO:tasks.rados.rados.0.smithi160.stdout:1169: finishing write tid 3 to smithi16030472-113 2024-09-21T18:08:12.955 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:12.955 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:12.957 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:12.969 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:12.983 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:12.984 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:12.994 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:12.987+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: finishing write tid 6 to smithi16030472-341 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: oid 341 updating version 0 to 522 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: oid 341 updating version 522 to 523 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: oid 341 version 523 is already newer than 520 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: oid 341 updating version 523 to 524 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: oid 341 version 524 is already newer than 521 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 524 (ObjNum 705 snap 0 seq_num 705) dirty exists 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: left oid 341 (ObjNum 705 snap 0 seq_num 705) 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1164: done (15 left) 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1183: write oid 409 current snap is 0 2024-09-21T18:08:12.997 INFO:tasks.rados.rados.0.smithi160.stdout:1183: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-21T18:08:13.006 INFO:tasks.rados.rados.0.smithi160.stdout:1183: writing smithi16030472-409 from 766584 to 1564701 tid 1 2024-09-21T18:08:13.012 INFO:tasks.rados.rados.0.smithi160.stdout:1183: writing smithi16030472-409 from 2098238 to 2713812 tid 2 2024-09-21T18:08:13.014 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:13.014 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:13.015 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:13.017 INFO:tasks.rados.rados.0.smithi160.stdout:1183: writing smithi16030472-409 from 3169759 to 3630138 tid 3 2024-09-21T18:08:13.017 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.017 INFO:tasks.rados.rados.0.smithi160.stdout:1171: got (2) No such file or directory 2024-09-21T18:08:13.017 INFO:tasks.rados.rados.0.smithi160.stdout:1166: finishing copy_from to smithi16030472-186 2024-09-21T18:08:13.018 INFO:tasks.rados.rados.0.smithi160.stdout:1166: got expected ENOENT (src dne) 2024-09-21T18:08:13.018 INFO:tasks.rados.rados.0.smithi160.stdout:1166: finishing copy_from racing read to smithi16030472-186 2024-09-21T18:08:13.018 INFO:tasks.rados.rados.0.smithi160.stdout:1172: expect (ObjNum 654 snap 0 seq_num 654) 2024-09-21T18:08:13.031 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1731072 2024-09-21T18:08:13.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:13.033 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:13.052 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:13.052 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:13.076 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:13.076 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:13.088 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:13.094 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:13.087+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:13.124 INFO:tasks.rados.rados.0.smithi160.stdout:1180: got (16) Device or resource busy 2024-09-21T18:08:13.124 INFO:tasks.rados.rados.0.smithi160.stdout:1169: finishing write tid 4 to smithi16030472-113 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: finishing write tid 5 to smithi16030472-113 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: oid 113 updating version 0 to 516 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: oid 113 updating version 516 to 519 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: oid 113 version 519 is already newer than 518 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: oid 113 version 519 is already newer than 517 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 113 v 519 (ObjNum 707 snap 0 seq_num 707) dirty exists 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: left oid 113 (ObjNum 707 snap 0 seq_num 707) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1166: done (15 left) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1169: done (14 left) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1171: done (13 left) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1172: done (12 left) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1180: done (11 left) 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1184: copy_from oid 258 from oid 221 current snap is 0 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:1179: got (0) Success 2024-09-21T18:08:13.125 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 392 v 0 (ObjNum 129 snap 0 seq_num 3125802352) clean dne 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1177: got (0) Success 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 267 v 0 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1177: done (11 left) 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1179: done (10 left) 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1185: cache_evict oid 172 snap -1 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1186: copy_from oid 310 from oid 146 current snap is 0 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1187: cache_try_flush oid 294 snap -1 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1188: read oid 64 snap -1 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1188: expect (ObjNum 644 snap 0 seq_num 644) 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1189: write oid 144 current snap is 0 2024-09-21T18:08:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:1189: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-21T18:08:13.132 INFO:tasks.rados.rados.0.smithi160.stdout:1189: writing smithi16030472-144 from 783391 to 1256757 tid 1 2024-09-21T18:08:13.133 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.133 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:13.133 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:13.133 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:13.138 INFO:tasks.rados.rados.0.smithi160.stdout:1189: writing smithi16030472-144 from 1756720 to 2232305 tid 2 2024-09-21T18:08:13.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:13.147 INFO:tasks.rados.rados.0.smithi160.stdout:1189: writing smithi16030472-144 from 2797182 to 3585186 tid 3 2024-09-21T18:08:13.147 INFO:tasks.rados.rados.0.smithi160.stdout:1190: read oid 394 snap -1 2024-09-21T18:08:13.147 INFO:tasks.rados.rados.0.smithi160.stdout:1190: expect deleted 2024-09-21T18:08:13.147 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.196 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:13.191+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:13.213 INFO:tasks.rados.rados.0.smithi160.stdout:1167: finishing copy_from racing read to smithi16030472-114 2024-09-21T18:08:13.213 INFO:tasks.rados.rados.0.smithi160.stdout:1176: finishing write tid 1 to smithi16030472-207 2024-09-21T18:08:13.218 INFO:tasks.rados.rados.0.smithi160.stdout:1178: done (15 left) 2024-09-21T18:08:13.219 INFO:tasks.rados.rados.0.smithi160.stdout:1191: cache_flush oid 147 snap -1 2024-09-21T18:08:13.219 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.235 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:13.235 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:13.260 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:13.260 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:13.262 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.262 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:13.262 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:13.262 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:13.262 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:13.275 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:13.283 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:13.283 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:13.296 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:13.293+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:13.312 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:13.313 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:13.317 INFO:tasks.rados.rados.0.smithi160.stdout:1188: expect (ObjNum 644 snap 0 seq_num 644) 2024-09-21T18:08:13.341 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:13.346 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:13.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:13.394 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 619584 2024-09-21T18:08:13.397 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:13.393+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:13.406 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.406 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:13.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 42.372294807999999, 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 42.368684256999998, 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:13.415 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 42.365877718, 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:08:13.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:08:13.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.419 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 42.362129518000003, 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:08:13.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:08:13.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.741278714000003, 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.424 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:08:13.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:08:13.428 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.640424641999999, 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:08:13.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:08:13.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:08:13.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.639668362000002, 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:08:13.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.639276805000002, 2024-09-21T18:08:13.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.638975680999998, 2024-09-21T18:08:13.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:08:13.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.445 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.638251830999998, 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:08:13.449 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.537354991999997, 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:08:13.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:08:13.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:13.454 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.537260709000002, 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:13.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:08:13.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.536347513999999, 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:13.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:08:13.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.535777826, 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.535050341999998, 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.467 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.468 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.469 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.470 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:13.471 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 41.468868432999997, 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.472 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.473 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.474 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 40.212822627999998, 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:08:13.475 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.476 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:08:13.477 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.478 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 40.212539829999997, 2024-09-21T18:08:13.479 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.480 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.481 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:08:13.482 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.483 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 40.212460845000003, 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:08:13.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 40.209842514000002, 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:13.488 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:13.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:13.492 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:13.493 INFO:tasks.rados.rados.0.smithi160.stdout:1167: finishing copy_from to smithi16030472-114 2024-09-21T18:08:13.493 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 271 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-21T18:08:13.493 INFO:tasks.rados.rados.0.smithi160.stdout:1182: finishing write tid 1 to smithi16030472-243 2024-09-21T18:08:13.493 INFO:tasks.rados.rados.0.smithi160.stdout:1182: finishing write tid 2 to smithi16030472-243 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: finishing write tid 1 to smithi16030472-45 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: finishing write tid 2 to smithi16030472-45 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: finishing write tid 3 to smithi16030472-45 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: finishing write tid 4 to smithi16030472-45 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: finishing write tid 5 to smithi16030472-45 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: oid 45 updating version 0 to 526 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: oid 45 version 526 is already newer than 525 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: oid 45 updating version 526 to 528 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: oid 45 version 528 is already newer than 527 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 45 v 528 (ObjNum 708 snap 0 seq_num 708) dirty exists 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1174: left oid 45 (ObjNum 708 snap 0 seq_num 708) 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1183: finishing write tid 1 to smithi16030472-409 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1176: finishing write tid 2 to smithi16030472-207 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1176: finishing write tid 3 to smithi16030472-207 2024-09-21T18:08:13.494 INFO:tasks.rados.rados.0.smithi160.stdout:1183: finishing write tid 2 to smithi16030472-409 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1183: finishing write tid 3 to smithi16030472-409 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1183: finishing write tid 4 to smithi16030472-409 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1167: done (15 left) 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1174: done (14 left) 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1188: done (13 left) 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1192: write oid 356 current snap is 0 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1192: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-21T18:08:13.495 INFO:tasks.rados.rados.0.smithi160.stdout:1192: writing smithi16030472-356 from 400198 to 948813 tid 1 2024-09-21T18:08:13.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:13.498 INFO:tasks.rados.rados.0.smithi160.stdout:1192: writing smithi16030472-356 from 1415202 to 2000198 tid 2 2024-09-21T18:08:13.498 INFO:tasks.rados.rados.0.smithi160.stdout:1193: read oid 498 snap -1 2024-09-21T18:08:13.498 INFO:tasks.rados.rados.0.smithi160.stdout:1193: expect (ObjNum 497 snap 0 seq_num 497) 2024-09-21T18:08:13.498 INFO:tasks.rados.rados.0.smithi160.stdout:1194: write (excl) oid 27 current snap is 0 2024-09-21T18:08:13.498 INFO:tasks.rados.rados.0.smithi160.stdout:1194: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-21T18:08:13.507 INFO:tasks.rados.rados.0.smithi160.stdout:1194: writing smithi16030472-27 from 417005 to 1157222 tid 1 2024-09-21T18:08:13.517 INFO:tasks.rados.rados.0.smithi160.stdout:1194: writing smithi16030472-27 from 1590037 to 2017005 tid 2 2024-09-21T18:08:13.517 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.525 INFO:tasks.rados.rados.0.smithi160.stdout:1176: finishing write tid 4 to smithi16030472-207 2024-09-21T18:08:13.525 INFO:tasks.rados.rados.0.smithi160.stdout:1176: finishing write tid 5 to smithi16030472-207 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: oid 207 updating version 0 to 477 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: oid 207 version 477 is already newer than 466 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: oid 207 updating version 477 to 478 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: oid 207 version 478 is already newer than 476 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 207 v 478 (ObjNum 709 snap 0 seq_num 709) dirty exists 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: left oid 207 (ObjNum 709 snap 0 seq_num 709) 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1181: finishing write tid 1 to smithi16030472-139 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1176: done (15 left) 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout:1195: cache_try_flush oid 343 snap -1 2024-09-21T18:08:13.526 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1181: finishing write tid 2 to smithi16030472-139 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1181: finishing write tid 3 to smithi16030472-139 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: finishing write tid 5 to smithi16030472-409 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: oid 409 updating version 0 to 516 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: oid 409 updating version 516 to 517 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: oid 409 updating version 517 to 518 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: oid 409 version 518 is already newer than 515 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 518 (ObjNum 712 snap 0 seq_num 712) dirty exists 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: left oid 409 (ObjNum 712 snap 0 seq_num 712) 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1185: got (0) Success 2024-09-21T18:08:13.527 INFO:tasks.rados.rados.0.smithi160.stdout:1183: done (15 left) 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:1185: done (14 left) 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:1196: cache_evict oid 121 snap -1 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:1197: delete oid 170 current snap is 0 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:1187: got (0) Success 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 0 (ObjNum 314 snap 0 seq_num 314) dirty exists 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:1191: got (0) Success 2024-09-21T18:08:13.528 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 0 (ObjNum 146 snap 0 seq_num 146) dirty exists 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: finishing write tid 4 to smithi16030472-139 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: finishing write tid 5 to smithi16030472-139 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: oid 139 updating version 0 to 468 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: oid 139 updating version 468 to 469 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: oid 139 version 469 is already newer than 467 2024-09-21T18:08:13.529 INFO:tasks.rados.rados.0.smithi160.stdout:1181: oid 139 updating version 469 to 470 2024-09-21T18:08:13.530 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 470 (ObjNum 710 snap 0 seq_num 710) dirty exists 2024-09-21T18:08:13.530 INFO:tasks.rados.rados.0.smithi160.stdout:1181: left oid 139 (ObjNum 710 snap 0 seq_num 710) 2024-09-21T18:08:13.539 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:13.537+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:13.572 INFO:tasks.rados.rados.0.smithi160.stdout:1186: finishing copy_from racing read to smithi16030472-310 2024-09-21T18:08:13.572 INFO:tasks.rados.rados.0.smithi160.stdout:1189: finishing write tid 1 to smithi16030472-144 2024-09-21T18:08:13.572 INFO:tasks.rados.rados.0.smithi160.stdout:1189: finishing write tid 2 to smithi16030472-144 2024-09-21T18:08:13.572 INFO:tasks.rados.rados.0.smithi160.stdout:1189: finishing write tid 3 to smithi16030472-144 2024-09-21T18:08:13.573 INFO:tasks.rados.rados.0.smithi160.stdout:1189: finishing write tid 4 to smithi16030472-144 2024-09-21T18:08:13.599 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:13.597+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:13.627 INFO:tasks.rados.rados.0.smithi160.stdout:1192: finishing write tid 1 to smithi16030472-356 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1181: done (15 left) 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1187: done (14 left) 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1191: done (13 left) 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1197: done (12 left) 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1198: cache_flush oid 132 snap -1 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1199: copy_from oid 342 from oid 466 current snap is 0 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1200: cache_try_flush oid 169 snap -1 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout:1201: cache_try_flush oid 323 snap -1 2024-09-21T18:08:13.631 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.634 INFO:tasks.rados.rados.0.smithi160.stdout:1193: expect (ObjNum 497 snap 0 seq_num 497) 2024-09-21T18:08:13.634 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 518345 2024-09-21T18:08:13.669 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.669 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:13.669 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:13.669 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:13.683 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:13.725 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 740475 2024-09-21T18:08:13.727 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:13.725+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:13.750 INFO:tasks.rados.rados.0.smithi160.stdout:1175: finishing copy_from to smithi16030472-326 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 326 v 274 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1184: finishing copy_from racing read to smithi16030472-258 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1198: got (0) Success 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 0 (ObjNum 641 snap 0 seq_num 641) dirty exists 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1184: finishing copy_from to smithi16030472-258 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 183 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: finishing write tid 5 to smithi16030472-144 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: oid 144 updating version 0 to 530 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: oid 144 updating version 530 to 532 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: oid 144 version 532 is already newer than 531 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: oid 144 version 532 is already newer than 529 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 144 v 532 (ObjNum 713 snap 0 seq_num 713) dirty exists 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1189: left oid 144 (ObjNum 713 snap 0 seq_num 713) 2024-09-21T18:08:13.751 INFO:tasks.rados.rados.0.smithi160.stdout:1186: finishing copy_from to smithi16030472-310 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 80 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1200: got (2) No such file or directory 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1184: done (15 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1186: done (14 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1189: done (13 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1190: done (12 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1193: done (11 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1198: done (10 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1200: done (9 left) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1202: cache_flush oid 387 snap -1 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1203: read oid 438 snap -1 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1203: expect (ObjNum 437 snap 0 seq_num 437) 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1204: cache_evict oid 151 snap -1 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1205: copy_from oid 372 from oid 475 current snap is 0 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1206: write (excl) oid 78 current snap is 0 2024-09-21T18:08:13.752 INFO:tasks.rados.rados.0.smithi160.stdout:1206: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-21T18:08:13.763 INFO:tasks.rados.rados.0.smithi160.stdout:1206: writing smithi16030472-78 from 450619 to 941334 tid 1 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:1206: writing smithi16030472-78 from 1590648 to 2050619 tid 2 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:1207: cache_try_flush oid 46 snap -1 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:1201: got (0) Success 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 323 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:1202: got (0) Success 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 387 v 0 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-21T18:08:13.780 INFO:tasks.rados.rados.0.smithi160.stdout:1201: done (14 left) 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout:1202: done (13 left) 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout:1208: read oid 332 snap -1 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout:1208: expect deleted 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout:1209: cache_try_flush oid 109 snap -1 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout:1210: cache_try_flush oid 24 snap -1 2024-09-21T18:08:13.781 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.804 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:13.801+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:13.822 INFO:tasks.rados.rados.0.smithi160.stdout:1182: finishing write tid 3 to smithi16030472-243 2024-09-21T18:08:13.827 INFO:tasks.rados.rados.0.smithi160.stdout:1175: finishing copy_from racing read to smithi16030472-326 2024-09-21T18:08:13.827 INFO:tasks.rados.rados.0.smithi160.stdout:1175: done (15 left) 2024-09-21T18:08:13.827 INFO:tasks.rados.rados.0.smithi160.stdout:1211: write oid 92 current snap is 0 2024-09-21T18:08:13.827 INFO:tasks.rados.rados.0.smithi160.stdout:1211: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-21T18:08:13.844 INFO:tasks.rados.rados.0.smithi160.stdout:1211: writing smithi16030472-92 from 467426 to 1033390 tid 1 2024-09-21T18:08:13.854 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.854 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:13.854 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:13.854 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:13.855 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1211: writing smithi16030472-92 from 1649130 to 2067426 tid 2 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: finishing write tid 4 to smithi16030472-243 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: finishing write tid 5 to smithi16030472-243 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: oid 243 updating version 0 to 521 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: oid 243 version 521 is already newer than 520 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: oid 243 version 521 is already newer than 426 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: oid 243 version 521 is already newer than 427 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 521 (ObjNum 711 snap 0 seq_num 711) dirty exists 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1182: left oid 243 (ObjNum 711 snap 0 seq_num 711) 2024-09-21T18:08:13.862 INFO:tasks.rados.rados.0.smithi160.stdout:1173: expect (ObjNum 510 snap 0 seq_num 510) 2024-09-21T18:08:13.869 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:13.904 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:13.897+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:13.905 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:13.950 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1140704 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:1194: finishing write tid 1 to smithi16030472-27 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:1207: got (0) Success 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 0 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:1195: got (0) Success 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 0 (ObjNum 97 snap 0 seq_num 3126253072) clean dne 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:1194: finishing write tid 2 to smithi16030472-27 2024-09-21T18:08:13.975 INFO:tasks.rados.rados.0.smithi160.stdout:1173: done (15 left) 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1182: done (14 left) 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1195: done (13 left) 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1207: done (12 left) 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1212: cache_flush oid 422 snap -1 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: finishing write tid 3 to smithi16030472-27 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: finishing write tid 4 to smithi16030472-27 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: oid 27 updating version 0 to 534 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: oid 27 updating version 534 to 535 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: oid 27 version 535 is already newer than 533 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 535 (ObjNum 715 snap 0 seq_num 715) dirty exists 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1194: left oid 27 (ObjNum 715 snap 0 seq_num 715) 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1196: got (0) Success 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1192: finishing write tid 2 to smithi16030472-356 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1192: finishing write tid 3 to smithi16030472-356 2024-09-21T18:08:13.976 INFO:tasks.rados.rados.0.smithi160.stdout:1192: finishing write tid 4 to smithi16030472-356 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:1192: oid 356 updating version 0 to 517 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:1192: oid 356 updating version 517 to 519 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:1192: oid 356 version 519 is already newer than 518 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 356 v 519 (ObjNum 714 snap 0 seq_num 714) dirty exists 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:1192: left oid 356 (ObjNum 714 snap 0 seq_num 714) 2024-09-21T18:08:13.977 INFO:tasks.rados.rados.0.smithi160.stdout:1203: expect (ObjNum 437 snap 0 seq_num 437) 2024-09-21T18:08:13.992 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:13.992 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:13.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:13.992 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:08:13.992 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:14.005 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:14.001+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:14.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:1209: got (0) Success 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 0 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:1204: got (0) Success 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:1206: finishing write tid 1 to smithi16030472-78 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:1206: finishing write tid 2 to smithi16030472-78 2024-09-21T18:08:14.069 INFO:tasks.rados.rados.0.smithi160.stdout:1206: finishing write tid 3 to smithi16030472-78 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1206: finishing write tid 4 to smithi16030472-78 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1206: oid 78 updating version 0 to 536 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1206: oid 78 updating version 536 to 538 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1206: oid 78 version 538 is already newer than 537 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 538 (ObjNum 717 snap 0 seq_num 717) dirty exists 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1206: left oid 78 (ObjNum 717 snap 0 seq_num 717) 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1205: finishing copy_from to smithi16030472-372 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 358 (ObjNum 474 snap 0 seq_num 474) dirty exists 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1205: finishing copy_from racing read to smithi16030472-372 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1192: done (12 left) 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1194: done (11 left) 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1196: done (10 left) 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1203: done (9 left) 2024-09-21T18:08:14.070 INFO:tasks.rados.rados.0.smithi160.stdout:1204: done (8 left) 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1205: done (7 left) 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1206: done (6 left) 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1208: done (5 left) 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1209: done (4 left) 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1213: cache_flush oid 416 snap -1 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1214: write oid 175 current snap is 0 2024-09-21T18:08:14.071 INFO:tasks.rados.rados.0.smithi160.stdout:1214: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-21T18:08:14.087 INFO:tasks.rados.rados.0.smithi160.stdout:1214: writing smithi16030472-175 from 484233 to 1125446 tid 1 2024-09-21T18:08:14.103 INFO:tasks.rados.rados.0.smithi160.stdout:1214: writing smithi16030472-175 from 1707612 to 2084233 tid 2 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1215: delete oid 105 current snap is 0 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1212: got (0) Success 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 422 v 0 (ObjNum 680 snap 0 seq_num 680) dirty exists 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1210: got (0) Success 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 24 v 0 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1199: finishing copy_from racing read to smithi16030472-342 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1199: finishing copy_from to smithi16030472-342 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 310 (ObjNum 465 snap 0 seq_num 465) dirty exists 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:1213: got (0) Success 2024-09-21T18:08:14.104 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 0 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-09-21T18:08:14.107 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:14.105+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:14.139 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.139 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:14.140 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:14.140 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.153 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:14.189 INFO:tasks.rados.rados.0.smithi160.stdout:1211: finishing write tid 1 to smithi16030472-92 2024-09-21T18:08:14.207 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:14.203+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:14.273 INFO:tasks.rados.rados.0.smithi160.stdout:1211: finishing write tid 2 to smithi16030472-92 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1199: done (6 left) 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1210: done (5 left) 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1212: done (4 left) 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1213: done (3 left) 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1215: done (2 left) 2024-09-21T18:08:14.274 INFO:tasks.rados.rados.0.smithi160.stdout:1216: delete oid 269 current snap is 0 2024-09-21T18:08:14.277 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.278 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:14.278 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:14.278 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:14.278 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:14.308 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:14.301+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:14.372 INFO:tasks.rados.rados.0.smithi160.stdout:1211: finishing write tid 3 to smithi16030472-92 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1211: finishing write tid 4 to smithi16030472-92 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1211: oid 92 updating version 0 to 541 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1211: oid 92 version 541 is already newer than 540 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1211: oid 92 version 541 is already newer than 539 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 92 v 541 (ObjNum 718 snap 0 seq_num 718) dirty exists 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1211: left oid 92 (ObjNum 718 snap 0 seq_num 718) 2024-09-21T18:08:14.373 INFO:tasks.rados.rados.0.smithi160.stdout:1214: finishing write tid 1 to smithi16030472-175 2024-09-21T18:08:14.375 INFO:tasks.rados.rados.0.smithi160.stdout:1214: finishing write tid 2 to smithi16030472-175 2024-09-21T18:08:14.375 INFO:tasks.rados.rados.0.smithi160.stdout:1214: finishing write tid 3 to smithi16030472-175 2024-09-21T18:08:14.376 INFO:tasks.rados.rados.0.smithi160.stdout:1214: finishing write tid 4 to smithi16030472-175 2024-09-21T18:08:14.376 INFO:tasks.rados.rados.0.smithi160.stdout:1214: oid 175 updating version 0 to 542 2024-09-21T18:08:14.376 INFO:tasks.rados.rados.0.smithi160.stdout:1214: oid 175 updating version 542 to 543 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1214: oid 175 updating version 543 to 544 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 544 (ObjNum 719 snap 0 seq_num 719) dirty exists 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1214: left oid 175 (ObjNum 719 snap 0 seq_num 719) 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1211: done (2 left) 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1214: done (1 left) 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1216: done (0 left) 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1217: cache_flush oid 296 snap -1 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1218: copy_from oid 312 from oid 214 current snap is 0 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1219: write oid 163 current snap is 0 2024-09-21T18:08:14.377 INFO:tasks.rados.rados.0.smithi160.stdout:1219: seq_num 722 ranges {534654=466960,1766705=367949} 2024-09-21T18:08:14.393 INFO:tasks.rados.rados.0.smithi160.stdout:1219: writing smithi16030472-163 from 534654 to 1001614 tid 1 2024-09-21T18:08:14.410 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:14.405+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:14.411 INFO:tasks.rados.rados.0.smithi160.stdout:1219: writing smithi16030472-163 from 1766705 to 2134654 tid 2 2024-09-21T18:08:14.411 INFO:tasks.rados.rados.0.smithi160.stdout:1220: write (excl) oid 78 current snap is 0 2024-09-21T18:08:14.411 INFO:tasks.rados.rados.0.smithi160.stdout:1220: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 56.079717707999997, 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:08:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:08:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.239865711999997, 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:08:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.136145264, 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:08:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.135370520999999, 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.134722502000002, 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:08:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:08:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:08:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.120850806999997, 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:08:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.117909791000002, 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.114892943000001, 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:08:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:08:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:08:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.112681361, 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:08:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:08:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.007344269000001, 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:08:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:08:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 49.315921895000002, 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.461 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 49.312891895, 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:08:14.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 44.346327932999998, 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:08:14.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 42.679492498000002, 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.470 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:08:14.474 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.376001416999998, 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:14.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:08:14.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:08:14.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.373704384, 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:08:14.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:08:14.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:08:14.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.371653936000001, 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:08:14.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.369498575000001, 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.487 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.368735817000001, 2024-09-21T18:08:14.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:08:14.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:08:14.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:08:14.495 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.266830771999999, 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:14.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:08:14.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:14.499 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.501 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1220: writing smithi16030472-78 from 551461 to 1210023 tid 1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1220: writing smithi16030472-78 from 1941540 to 2151461 tid 2 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1221: read oid 14 snap -1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1221: expect (ObjNum 305 snap 0 seq_num 305) 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1222: read oid 158 snap -1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1222: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1223: cache_evict oid 202 snap -1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1224: cache_flush oid 116 snap -1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1225: read oid 84 snap -1 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1225: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-21T18:08:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:1226: cache_evict oid 323 snap -1 2024-09-21T18:08:14.503 INFO:tasks.rados.rados.0.smithi160.stdout:1227: read oid 121 snap -1 2024-09-21T18:08:14.503 INFO:tasks.rados.rados.0.smithi160.stdout:1227: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:14.503 INFO:tasks.rados.rados.0.smithi160.stdout:1228: delete oid 128 current snap is 0 2024-09-21T18:08:14.503 INFO:tasks.rados.rados.0.smithi160.stdout:1217: got (0) Success 2024-09-21T18:08:14.503 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 296 v 0 (ObjNum 273 snap 0 seq_num 0) clean dne 2024-09-21T18:08:14.510 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:14.505+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:14.538 INFO:tasks.rados.rados.0.smithi160.stdout:1220: finishing write tid 1 to smithi16030472-78 2024-09-21T18:08:14.612 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:14.609+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:14.628 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.628 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:14.628 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:14.628 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.639 INFO:tasks.rados.rados.0.smithi160.stdout:1220: finishing write tid 2 to smithi16030472-78 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:1220: finishing write tid 3 to smithi16030472-78 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:1220: finishing write tid 4 to smithi16030472-78 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:1220: oid 78 updating version 0 to 546 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:1220: oid 78 updating version 546 to 548 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:1220: oid 78 version 548 is already newer than 547 2024-09-21T18:08:14.640 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 548 (ObjNum 723 snap 0 seq_num 723) dirty exists 2024-09-21T18:08:14.641 INFO:tasks.rados.rados.0.smithi160.stdout:1220: left oid 78 (ObjNum 723 snap 0 seq_num 723) 2024-09-21T18:08:14.642 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:14.691 INFO:tasks.rados.rados.0.smithi160.stdout:1219: finishing write tid 1 to smithi16030472-163 2024-09-21T18:08:14.691 INFO:tasks.rados.rados.0.smithi160.stdout:1221: expect (ObjNum 305 snap 0 seq_num 305) 2024-09-21T18:08:14.691 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:14.712 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:14.709+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:14.762 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.763 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:14.763 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:14.763 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:14.763 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.772 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:14.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:14.814 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:14.809+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:14.823 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2864068 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: finishing write tid 2 to smithi16030472-163 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: finishing write tid 3 to smithi16030472-163 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: finishing write tid 4 to smithi16030472-163 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: oid 163 updating version 0 to 472 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: oid 163 updating version 472 to 473 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: oid 163 version 473 is already newer than 471 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 163 v 473 (ObjNum 722 snap 0 seq_num 722) dirty exists 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1219: left oid 163 (ObjNum 722 snap 0 seq_num 722) 2024-09-21T18:08:14.832 INFO:tasks.rados.rados.0.smithi160.stdout:1222: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-21T18:08:14.835 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1590187 2024-09-21T18:08:14.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:08:14.899 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:08:14.900 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:14.902 INFO:tasks.rados.rados.0.smithi160.stdout:1225: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-21T18:08:14.914 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:14.915 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:14.909+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:14.934 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2143644 2024-09-21T18:08:14.983 INFO:tasks.rados.rados.0.smithi160.stdout:1223: got (2) No such file or directory 2024-09-21T18:08:14.983 INFO:tasks.rados.rados.0.smithi160.stdout:1224: got (0) Success 2024-09-21T18:08:14.983 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 0 (ObjNum 597 snap 0 seq_num 597) dirty exists 2024-09-21T18:08:14.983 INFO:tasks.rados.rados.0.smithi160.stdout:1226: got (0) Success 2024-09-21T18:08:14.983 INFO:tasks.rados.rados.0.smithi160.stdout:1227: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:14.992 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1756643 2024-09-21T18:08:14.995 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.015 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:15.009+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1218: finishing copy_from to smithi16030472-312 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 312 v 171 (ObjNum 213 snap 0 seq_num 213) dirty exists 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1217: done (11 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1219: done (10 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1220: done (9 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1221: done (8 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1222: done (7 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1223: done (6 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1224: done (5 left) 2024-09-21T18:08:15.029 INFO:tasks.rados.rados.0.smithi160.stdout:1225: done (4 left) 2024-09-21T18:08:15.030 INFO:tasks.rados.rados.0.smithi160.stdout:1226: done (3 left) 2024-09-21T18:08:15.030 INFO:tasks.rados.rados.0.smithi160.stdout:1227: done (2 left) 2024-09-21T18:08:15.030 INFO:tasks.rados.rados.0.smithi160.stdout:1228: done (1 left) 2024-09-21T18:08:15.030 INFO:tasks.rados.rados.0.smithi160.stdout:1229: write oid 79 current snap is 0 2024-09-21T18:08:15.030 INFO:tasks.rados.rados.0.smithi160.stdout:1229: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-21T18:08:15.041 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:15.041 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:15.041 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:15.041 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:15.045 INFO:tasks.rados.rados.0.smithi160.stdout:1229: writing smithi16030472-79 from 585075 to 994135 tid 1 2024-09-21T18:08:15.056 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1229: writing smithi16030472-79 from 1542151 to 2185075 tid 2 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1230: cache_evict oid 132 snap -1 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1231: cache_evict oid 393 snap -1 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1232: cache_try_flush oid 444 snap -1 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1218: finishing copy_from racing read to smithi16030472-312 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1229: finishing write tid 1 to smithi16030472-79 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1218: done (4 left) 2024-09-21T18:08:15.065 INFO:tasks.rados.rados.0.smithi160.stdout:1233: write (excl) oid 472 current snap is 0 2024-09-21T18:08:15.066 INFO:tasks.rados.rados.0.smithi160.stdout:1233: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-21T18:08:15.083 INFO:tasks.rados.rados.0.smithi160.stdout:1233: writing smithi16030472-472 from 601882 to 1086191 tid 1 2024-09-21T18:08:15.099 INFO:tasks.rados.rados.0.smithi160.stdout:1233: writing smithi16030472-472 from 1600633 to 2125537 tid 2 2024-09-21T18:08:15.099 INFO:tasks.rados.rados.0.smithi160.stdout:1234: cache_try_flush oid 258 snap -1 2024-09-21T18:08:15.099 INFO:tasks.rados.rados.0.smithi160.stdout:1231: got (2) No such file or directory 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1231: done (5 left) 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1235: read oid 334 snap -1 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1235: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1236: cache_try_flush oid 53 snap -1 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1237: write oid 412 current snap is 0 2024-09-21T18:08:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:1237: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-21T18:08:15.114 INFO:tasks.rados.rados.0.smithi160.stdout:1237: writing smithi16030472-412 from 618689 to 1178247 tid 1 2024-09-21T18:08:15.116 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:15.113+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:15.127 INFO:tasks.rados.rados.0.smithi160.stdout:1237: writing smithi16030472-412 from 1659115 to 2218689 tid 2 2024-09-21T18:08:15.127 INFO:tasks.rados.rados.0.smithi160.stdout:1238: copy_from oid 210 from oid 47 current snap is 0 2024-09-21T18:08:15.127 INFO:tasks.rados.rados.0.smithi160.stdout:1239: read oid 276 snap -1 2024-09-21T18:08:15.127 INFO:tasks.rados.rados.0.smithi160.stdout:1239: expect (ObjNum 518 snap 0 seq_num 518) 2024-09-21T18:08:15.127 INFO:tasks.rados.rados.0.smithi160.stdout:1240: delete oid 352 current snap is 0 2024-09-21T18:08:15.136 INFO:tasks.rados.rados.0.smithi160.stdout:1235: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:08:15.178 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:15.178 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:15.178 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:15.178 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:15.178 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:15.192 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:15.217 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:15.211+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:15.248 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1391901 2024-09-21T18:08:15.269 INFO:tasks.rados.rados.0.smithi160.stdout:1239: expect (ObjNum 518 snap 0 seq_num 518) 2024-09-21T18:08:15.270 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.315 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 652557 2024-09-21T18:08:15.317 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:15.317+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:15.320 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:15.320 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:241 4.2 e48/33 4:6513c01e:::smithi16030472-210:head v 48'96, mlcod=48'94)", 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 51.196379514, 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.216069437, 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120843+0000", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.120847+0000", 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.1960000000000001e-06 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125270+0000", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0044228310000000003 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125303+0000", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2975999999999997e-05 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.125320+0000", 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.717e-05 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225582+0000", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100261425 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.225619+0000", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6962000000000003e-05 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336827+0000", 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111208272 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.336912+0000", 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5605000000000005e-05 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:15.325 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 43.721421341000003, 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:15.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:08:15.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.291451393999999, 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.333 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2892 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'238, mlcod=59'234)", 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.121662+0000", 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.195560337, 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.217054471, 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.121662+0000", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.121662+0000", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.121666+0000", 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9539999999999998e-06 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.122305+0000", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00063885700000000005 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.122338+0000", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2722000000000003e-05 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.122355+0000", 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7059e-05 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.231674+0000", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109319602 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.231708+0000", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3586e-05 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338623+0000", 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106915387 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338717+0000", 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3304e-05 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.193931019000001, 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.338 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:08:15.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.343 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.193246769000002, 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:08:15.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.192413414000001, 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.189694784, 2024-09-21T18:08:15.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:08:15.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:08:15.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.093886378000001, 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.091581837, 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:08:15.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:08:15.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:08:15.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2900 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'245, mlcod=59'234)", 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.991301549999999, 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22363195199999999, 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325923+0000", 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2929999999999999e-06 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325935+0000", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1763000000000001e-05 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325952+0000", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6722999999999999e-05 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325965+0000", 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3743e-05 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545223+0000", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21925772700000001 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545279+0000", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6085999999999998e-05 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549487+0000", 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004207868 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549553+0000", 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5748999999999997e-05 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:693 4.2 e60/33 4:7cb2799f:::smithi16030472-270:head v 60'246, mlcod=60'235)", 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.329609+0000", 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.987613377999999, 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22041367100000001, 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.369 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.329609+0000", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.329609+0000", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.329612+0000", 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1099999999999999e-06 2024-09-21T18:08:15.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.334084+0000", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004471671 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.334114+0000", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.976e-05 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.334129+0000", 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5522000000000001e-05 2024-09-21T18:08:15.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546097+0000", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21196764500000001 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546135+0000", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7953000000000001e-05 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549942+0000", 2024-09-21T18:08:15.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038073849999999999 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.550023+0000", 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0624999999999995e-05 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.1139532170000002, 2024-09-21T18:08:15.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:08:15.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:08:15.378 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.0127975349999998, 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.012001873, 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.383 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:08:15.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:08:15.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:08:15.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 2.010866053, 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.387 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.9901484599999999, 2024-09-21T18:08:15.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:08:15.395 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.9807957759999999, 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.399 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.975421128, 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:08:15.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 1.9728728530000001, 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:15.404 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:08:15.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:08:15.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:08:15.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:15.408 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:15.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 532144 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:1232: got (2) No such file or directory 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:1230: got (0) Success 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:1230: done (10 left) 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:1232: done (9 left) 2024-09-21T18:08:15.410 INFO:tasks.rados.rados.0.smithi160.stdout:1235: done (8 left) 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1239: done (7 left) 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1240: done (6 left) 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1241: read oid 393 snap -1 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1241: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1242: delete oid 499 current snap is 0 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:1236: got (0) Success 2024-09-21T18:08:15.411 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 53 v 0 (ObjNum 52 snap 0 seq_num 52) dirty exists 2024-09-21T18:08:15.411 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:15.418 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:15.413+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:15.424 INFO:tasks.rados.rados.0.smithi160.stdout:1237: finishing write tid 1 to smithi16030472-412 2024-09-21T18:08:15.425 INFO:tasks.rados.rados.0.smithi160.stdout:1237: finishing write tid 2 to smithi16030472-412 2024-09-21T18:08:15.426 INFO:tasks.rados.rados.0.smithi160.stdout:1237: finishing write tid 3 to smithi16030472-412 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:1237: finishing write tid 4 to smithi16030472-412 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:1237: oid 412 updating version 0 to 523 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:1237: oid 412 updating version 523 to 524 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:1237: oid 412 version 524 is already newer than 522 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 524 (ObjNum 727 snap 0 seq_num 727) dirty exists 2024-09-21T18:08:15.427 INFO:tasks.rados.rados.0.smithi160.stdout:1237: left oid 412 (ObjNum 727 snap 0 seq_num 727) 2024-09-21T18:08:15.448 INFO:tasks.rados.rados.0.smithi160.stdout:1236: done (7 left) 2024-09-21T18:08:15.448 INFO:tasks.rados.rados.0.smithi160.stdout:1237: done (6 left) 2024-09-21T18:08:15.448 INFO:tasks.rados.rados.0.smithi160.stdout:1242: done (5 left) 2024-09-21T18:08:15.449 INFO:tasks.rados.rados.0.smithi160.stdout:1243: write oid 166 current snap is 0 2024-09-21T18:08:15.449 INFO:tasks.rados.rados.0.smithi160.stdout:1243: seq_num 730 ranges {669110=501658,1834561=434549} 2024-09-21T18:08:15.467 INFO:tasks.rados.rados.0.smithi160.stdout:1243: writing smithi16030472-166 from 669110 to 1170768 tid 1 2024-09-21T18:08:15.486 INFO:tasks.rados.rados.0.smithi160.stdout:1243: writing smithi16030472-166 from 1834561 to 2269110 tid 2 2024-09-21T18:08:15.486 INFO:tasks.rados.rados.0.smithi160.stdout:1244: read oid 485 snap -1 2024-09-21T18:08:15.486 INFO:tasks.rados.rados.0.smithi160.stdout:1244: expect deleted 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1245: read oid 384 snap -1 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1245: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1246: cache_flush oid 464 snap -1 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1229: finishing write tid 2 to smithi16030472-79 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1247: write (excl) oid 370 current snap is 0 2024-09-21T18:08:15.487 INFO:tasks.rados.rados.0.smithi160.stdout:1247: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-21T18:08:15.506 INFO:tasks.rados.rados.0.smithi160.stdout:1247: writing smithi16030472-370 from 685917 to 1262824 tid 1 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1247: writing smithi16030472-370 from 1893043 to 2285917 tid 2 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1248: cache_try_flush oid 264 snap -1 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1249: cache_flush oid 328 snap -1 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1250: read oid 497 snap -1 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1250: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:08:15.517 INFO:tasks.rados.rados.0.smithi160.stdout:1251: write (excl) oid 298 current snap is 0 2024-09-21T18:08:15.518 INFO:tasks.rados.rados.0.smithi160.stdout:1251: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-21T18:08:15.519 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:15.517+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:15.529 INFO:tasks.rados.rados.0.smithi160.stdout:1251: writing smithi16030472-298 from 702724 to 1354880 tid 1 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1251: writing smithi16030472-298 from 1951525 to 2302724 tid 2 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1252: read oid 268 snap -1 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1252: expect (ObjNum 537 snap 0 seq_num 537) 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1253: read oid 124 snap -1 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1253: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1238: finishing copy_from racing read to smithi16030472-210 2024-09-21T18:08:15.538 INFO:tasks.rados.rados.0.smithi160.stdout:1245: expect (ObjNum 383 snap 0 seq_num 383) 2024-09-21T18:08:15.567 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.570 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:15.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:15.585 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:15.585 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:15.585 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:15.586 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:15.591 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:15.591 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:15.600 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:15.609 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:15.610 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:15.620 INFO:tasks.rados.rados.0.smithi160.stdout:1248: got (2) No such file or directory 2024-09-21T18:08:15.620 INFO:tasks.rados.rados.0.smithi160.stdout:1253: expect (ObjNum 123 snap 0 seq_num 123) 2024-09-21T18:08:15.620 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.621 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:15.613+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:15.630 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:15.631 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:15.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:15.659 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.667 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1238: finishing copy_from to smithi16030472-210 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 413 (ObjNum 676 snap 0 seq_num 676) dirty exists 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1238: done (15 left) 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1244: done (14 left) 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1245: done (13 left) 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1248: done (12 left) 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1253: done (11 left) 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1254: copy_from oid 384 from oid 444 current snap is 0 2024-09-21T18:08:15.681 INFO:tasks.rados.rados.0.smithi160.stdout:1255: write oid 378 current snap is 0 2024-09-21T18:08:15.682 INFO:tasks.rados.rados.0.smithi160.stdout:1255: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-21T18:08:15.685 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:1255: writing smithi16030472-378 from 719531 to 1446936 tid 1 2024-09-21T18:08:15.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:15.708 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1255: writing smithi16030472-378 from 1893654 to 2319531 tid 2 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1256: read oid 407 snap -1 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1256: expect deleted 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1257: cache_evict oid 259 snap -1 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1243: finishing write tid 1 to smithi16030472-166 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1258: read oid 168 snap -1 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1258: expect deleted 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1243: finishing write tid 2 to smithi16030472-166 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1243: finishing write tid 3 to smithi16030472-166 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1243: finishing write tid 4 to smithi16030472-166 2024-09-21T18:08:15.722 INFO:tasks.rados.rados.0.smithi160.stdout:1243: oid 166 updating version 0 to 551 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:1243: oid 166 version 551 is already newer than 550 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:1243: oid 166 updating version 551 to 552 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 552 (ObjNum 730 snap 0 seq_num 730) dirty exists 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:1243: left oid 166 (ObjNum 730 snap 0 seq_num 730) 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:1247: finishing write tid 1 to smithi16030472-370 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:1241: expect (ObjNum 392 snap 0 seq_num 392) 2024-09-21T18:08:15.723 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.723 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:15.713+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:15.724 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:15.724 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:15.724 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:15.724 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:15.724 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:15.730 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:15.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:15.749 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:15.749 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:15.769 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:15.769 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:15.788 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:15.788 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:15.822 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:15.817+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:15.825 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2414302 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: finishing write tid 2 to smithi16030472-370 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: finishing write tid 3 to smithi16030472-370 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: finishing write tid 4 to smithi16030472-370 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: oid 370 updating version 0 to 554 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: oid 370 updating version 554 to 555 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: oid 370 version 555 is already newer than 553 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 370 v 555 (ObjNum 731 snap 0 seq_num 731) dirty exists 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1247: left oid 370 (ObjNum 731 snap 0 seq_num 731) 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:1250: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:08:15.827 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:15.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:08:15.863 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:15.863 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:15.877 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:15.880 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:15.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:15.903 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:15.906 INFO:tasks.rados.rados.0.smithi160.stdout:1229: finishing write tid 3 to smithi16030472-79 2024-09-21T18:08:15.906 INFO:tasks.rados.rados.0.smithi160.stdout:1241: done (15 left) 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1243: done (14 left) 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1247: done (13 left) 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1250: done (12 left) 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1259: cache_try_flush oid 6 snap -1 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1229: finishing write tid 4 to smithi16030472-79 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1229: oid 79 updating version 0 to 521 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1229: oid 79 version 521 is already newer than 519 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1229: oid 79 version 521 is already newer than 520 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 521 (ObjNum 725 snap 0 seq_num 725) dirty exists 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1229: left oid 79 (ObjNum 725 snap 0 seq_num 725) 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1233: finishing write tid 1 to smithi16030472-472 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1233: finishing write tid 2 to smithi16030472-472 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1233: finishing write tid 3 to smithi16030472-472 2024-09-21T18:08:15.907 INFO:tasks.rados.rados.0.smithi160.stdout:1234: got (0) Success 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 0 (ObjNum 220 snap 0 seq_num 220) dirty exists 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1233: finishing write tid 4 to smithi16030472-472 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1233: oid 472 updating version 0 to 524 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1233: oid 472 version 524 is already newer than 523 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1233: oid 472 version 524 is already newer than 522 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 472 v 524 (ObjNum 726 snap 0 seq_num 726) dirty exists 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1233: left oid 472 (ObjNum 726 snap 0 seq_num 726) 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1246: got (0) Success 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 0 (ObjNum 675 snap 0 seq_num 675) dirty exists 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1249: got (0) Success 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1255: finishing write tid 1 to smithi16030472-378 2024-09-21T18:08:15.908 INFO:tasks.rados.rados.0.smithi160.stdout:1252: expect (ObjNum 537 snap 0 seq_num 537) 2024-09-21T18:08:15.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:15.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:15.923 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:15.921+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:15.925 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2194716 2024-09-21T18:08:15.952 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1408809 2024-09-21T18:08:15.991 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1206336 2024-09-21T18:08:16.009 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.009 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1259: got (0) Success 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 0 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1229: done (12 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1233: done (11 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1234: done (10 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1246: done (9 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1249: done (8 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1252: done (7 left) 2024-09-21T18:08:16.021 INFO:tasks.rados.rados.0.smithi160.stdout:1258: done (6 left) 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1259: done (5 left) 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1260: read oid 165 snap -1 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1260: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1261: delete oid 125 current snap is 0 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1257: got (0) Success 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: finishing write tid 1 to smithi16030472-298 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: finishing write tid 2 to smithi16030472-298 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: finishing write tid 3 to smithi16030472-298 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: finishing write tid 4 to smithi16030472-298 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: oid 298 updating version 0 to 476 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: oid 298 version 476 is already newer than 475 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: oid 298 version 476 is already newer than 474 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 298 v 476 (ObjNum 732 snap 0 seq_num 732) dirty exists 2024-09-21T18:08:16.022 INFO:tasks.rados.rados.0.smithi160.stdout:1251: left oid 298 (ObjNum 732 snap 0 seq_num 732) 2024-09-21T18:08:16.023 INFO:tasks.rados.rados.0.smithi160.stdout:1255: finishing write tid 2 to smithi16030472-378 2024-09-21T18:08:16.023 INFO:tasks.rados.rados.0.smithi160.stdout:1254: finishing copy_from racing read to smithi16030472-384 2024-09-21T18:08:16.023 INFO:tasks.rados.rados.0.smithi160.stdout:1254: finishing copy_from to smithi16030472-384 2024-09-21T18:08:16.023 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 384 v 317 (ObjNum 443 snap 0 seq_num 443) dirty exists 2024-09-21T18:08:16.024 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:16.021+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:16.025 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:1255: finishing write tid 3 to smithi16030472-378 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:1255: finishing write tid 4 to smithi16030472-378 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:1255: oid 378 updating version 0 to 509 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:1255: oid 378 updating version 509 to 556 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:1255: oid 378 updating version 556 to 557 2024-09-21T18:08:16.026 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 378 v 557 (ObjNum 733 snap 0 seq_num 733) dirty exists 2024-09-21T18:08:16.027 INFO:tasks.rados.rados.0.smithi160.stdout:1255: left oid 378 (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:08:16.125 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:16.121+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:16.155 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.155 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.170 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:16.226 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:16.221+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:16.292 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.292 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:16.322 INFO:tasks.rados.rados.0.smithi160.stdout:1260: expect (ObjNum 164 snap 0 seq_num 164) 2024-09-21T18:08:16.326 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:16.325+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:16.335 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2219377 2024-09-21T18:08:16.339 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:16.383 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2538048 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1251: done (6 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1254: done (5 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1255: done (4 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1256: done (3 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1257: done (2 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1260: done (1 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1261: done (0 left) 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1262: write (excl) oid 494 current snap is 0 2024-09-21T18:08:16.386 INFO:tasks.rados.rados.0.smithi160.stdout:1262: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-21T18:08:16.393 INFO:tasks.rados.rados.0.smithi160.stdout:1262: writing smithi16030472-494 from 753145 to 1231048 tid 1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1262: writing smithi16030472-494 from 2010618 to 2353145 tid 2 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1263: cache_try_flush oid 173 snap -1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1264: read oid 306 snap -1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1264: expect deleted 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1265: read oid 313 snap -1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1265: expect (ObjNum 312 snap 0 seq_num 312) 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1266: cache_evict oid 69 snap -1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1267: cache_flush oid 326 snap -1 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1268: write (excl) oid 139 current snap is 0 2024-09-21T18:08:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:1268: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-21T18:08:16.409 INFO:tasks.rados.rados.0.smithi160.stdout:1268: writing smithi16030472-139 from 769952 to 1323104 tid 1 2024-09-21T18:08:16.415 INFO:tasks.rados.rados.0.smithi160.stdout:1268: writing smithi16030472-139 from 2069100 to 2369952 tid 2 2024-09-21T18:08:16.415 INFO:tasks.rados.rados.0.smithi160.stdout:1269: cache_evict oid 436 snap -1 2024-09-21T18:08:16.415 INFO:tasks.rados.rados.0.smithi160.stdout:1270: write (excl) oid 315 current snap is 0 2024-09-21T18:08:16.415 INFO:tasks.rados.rados.0.smithi160.stdout:1270: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-21T18:08:16.423 INFO:tasks.rados.rados.0.smithi160.stdout:1270: writing smithi16030472-315 from 786759 to 1415160 tid 1 2024-09-21T18:08:16.427 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:16.425+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:16.428 INFO:tasks.rados.rados.0.smithi160.stdout:1270: writing smithi16030472-315 from 2011229 to 2386759 tid 2 2024-09-21T18:08:16.428 INFO:tasks.rados.rados.0.smithi160.stdout:1271: write (excl) oid 498 current snap is 0 2024-09-21T18:08:16.428 INFO:tasks.rados.rados.0.smithi160.stdout:1271: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-21T18:08:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:1271: writing smithi16030472-498 from 403566 to 823569 tid 1 2024-09-21T18:08:16.438 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.439 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1271: writing smithi16030472-498 from 1386064 to 1932982 tid 2 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1272: copy_from oid 197 from oid 102 current snap is 0 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1266: got (16) Device or resource busy 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1264: done (10 left) 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1266: done (9 left) 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1273: read oid 405 snap -1 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1273: expect deleted 2024-09-21T18:08:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:1274: cache_try_flush oid 130 snap -1 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1275: cache_flush oid 109 snap -1 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1276: read oid 421 snap -1 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1276: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1277: cache_evict oid 459 snap -1 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1278: delete oid 465 current snap is 0 2024-09-21T18:08:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:1274: got (2) No such file or directory 2024-09-21T18:08:16.444 INFO:tasks.rados.rados.0.smithi160.stdout:1277: got (0) Success 2024-09-21T18:08:16.445 INFO:tasks.rados.rados.0.smithi160.stdout:1274: done (14 left) 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1277: done (13 left) 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1278: done (12 left) 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1279: cache_flush oid 78 snap -1 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1280: read oid 252 snap -1 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1280: expect (ObjNum 251 snap 0 seq_num 251) 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1281: read oid 206 snap -1 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1281: expect (ObjNum 572 snap 0 seq_num 572) 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1282: write (excl) oid 376 current snap is 0 2024-09-21T18:08:16.446 INFO:tasks.rados.rados.0.smithi160.stdout:1282: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-21T18:08:16.451 INFO:tasks.rados.rados.0.smithi160.stdout:1282: writing smithi16030472-376 from 437180 to 1007681 tid 1 2024-09-21T18:08:16.454 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:1282: writing smithi16030472-376 from 1503028 to 2053615 tid 2 2024-09-21T18:08:16.457 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:1267: got (0) Success 2024-09-21T18:08:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 326 v 0 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:08:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:1267: done (15 left) 2024-09-21T18:08:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:1283: write (excl) oid 472 current snap is 0 2024-09-21T18:08:16.502 INFO:tasks.rados.rados.0.smithi160.stdout:1283: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-21T18:08:16.509 INFO:tasks.rados.rados.0.smithi160.stdout:1283: writing smithi16030472-472 from 453987 to 1099737 tid 1 2024-09-21T18:08:16.518 INFO:tasks.rados.rados.0.smithi160.stdout:1283: writing smithi16030472-472 from 1845157 to 2453987 tid 2 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout:1269: got (16) Device or resource busy 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout:1269: done (15 left) 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout:1284: read oid 231 snap -1 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout:1284: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-21T18:08:16.519 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.528 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:16.525+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:16.580 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.580 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:16.629 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:16.625+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:16.657 INFO:tasks.rados.rados.0.smithi160.stdout:1282: finishing write tid 1 to smithi16030472-376 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1279: got (0) Success 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 0 (ObjNum 723 snap 0 seq_num 723) dirty exists 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1279: done (15 left) 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1285: read oid 277 snap -1 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1285: expect (ObjNum 276 snap 0 seq_num 276) 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1282: finishing write tid 2 to smithi16030472-376 2024-09-21T18:08:16.676 INFO:tasks.rados.rados.0.smithi160.stdout:1282: finishing write tid 3 to smithi16030472-376 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: finishing write tid 4 to smithi16030472-376 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: oid 376 updating version 0 to 559 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: oid 376 updating version 559 to 560 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: oid 376 version 560 is already newer than 510 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 376 v 560 (ObjNum 740 snap 0 seq_num 740) dirty exists 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: left oid 376 (ObjNum 740 snap 0 seq_num 740) 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1282: done (15 left) 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout:1286: copy_from oid 117 from oid 454 current snap is 0 2024-09-21T18:08:16.677 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.709 INFO:tasks.rados.rados.0.smithi160.stdout:1271: finishing write tid 1 to smithi16030472-498 2024-09-21T18:08:16.710 INFO:tasks.rados.rados.0.smithi160.stdout:1271: finishing write tid 2 to smithi16030472-498 2024-09-21T18:08:16.710 INFO:tasks.rados.rados.0.smithi160.stdout:1271: finishing write tid 3 to smithi16030472-498 2024-09-21T18:08:16.710 INFO:tasks.rados.rados.0.smithi160.stdout:1271: finishing write tid 4 to smithi16030472-498 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1271: oid 498 updating version 0 to 527 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1271: oid 498 updating version 527 to 528 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1271: oid 498 version 528 is already newer than 526 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 498 v 528 (ObjNum 738 snap 0 seq_num 738) dirty exists 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1271: left oid 498 (ObjNum 738 snap 0 seq_num 738) 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1271: done (15 left) 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1287: write (excl) oid 122 current snap is 0 2024-09-21T18:08:16.711 INFO:tasks.rados.rados.0.smithi160.stdout:1287: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-21T18:08:16.713 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.713 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.723 INFO:tasks.rados.rados.0.smithi160.stdout:1287: writing smithi16030472-122 from 470794 to 1191793 tid 1 2024-09-21T18:08:16.730 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:16.725+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:16.731 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:16.735 INFO:tasks.rados.rados.0.smithi160.stdout:1287: writing smithi16030472-122 from 1903639 to 2457895 tid 2 2024-09-21T18:08:16.735 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.735 INFO:tasks.rados.rados.0.smithi160.stdout:1262: finishing write tid 1 to smithi16030472-494 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1263: got (0) Success 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 0 (ObjNum 605 snap 0 seq_num 605) dirty exists 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1275: got (0) Success 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 0 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1263: done (15 left) 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1275: done (14 left) 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1288: read oid 153 snap -1 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1288: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout:1289: copy_from oid 350 from oid 433 current snap is 0 2024-09-21T18:08:16.736 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:16.829 INFO:tasks.rados.rados.0.smithi160.stdout:1265: expect (ObjNum 312 snap 0 seq_num 312) 2024-09-21T18:08:16.830 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:16.825+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:16.856 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:16.856 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:16.870 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:16.923 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 758046 2024-09-21T18:08:16.931 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:16.929+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:16.955 INFO:tasks.rados.rados.0.smithi160.stdout:1281: expect (ObjNum 572 snap 0 seq_num 572) 2024-09-21T18:08:16.998 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:17.033 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:17.027+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:17.043 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.043 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.058 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:17.107 INFO:tasks.rados.rados.0.smithi160.stdout:1262: finishing write tid 2 to smithi16030472-494 2024-09-21T18:08:17.107 INFO:tasks.rados.rados.0.smithi160.stdout:1268: finishing write tid 1 to smithi16030472-139 2024-09-21T18:08:17.107 INFO:tasks.rados.rados.0.smithi160.stdout:1270: finishing write tid 1 to smithi16030472-315 2024-09-21T18:08:17.107 INFO:tasks.rados.rados.0.smithi160.stdout:1270: finishing write tid 2 to smithi16030472-315 2024-09-21T18:08:17.107 INFO:tasks.rados.rados.0.smithi160.stdout:1270: finishing write tid 3 to smithi16030472-315 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1270: finishing write tid 4 to smithi16030472-315 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1270: oid 315 updating version 0 to 479 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1270: oid 315 updating version 479 to 481 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1270: oid 315 version 481 is already newer than 480 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 315 v 481 (ObjNum 737 snap 0 seq_num 737) dirty exists 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1270: left oid 315 (ObjNum 737 snap 0 seq_num 737) 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: finishing write tid 2 to smithi16030472-139 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: finishing write tid 3 to smithi16030472-139 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: finishing write tid 4 to smithi16030472-139 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: oid 139 updating version 0 to 478 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: oid 139 updating version 478 to 482 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: oid 139 updating version 482 to 483 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 483 (ObjNum 736 snap 0 seq_num 736) dirty exists 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1268: left oid 139 (ObjNum 736 snap 0 seq_num 736) 2024-09-21T18:08:17.108 INFO:tasks.rados.rados.0.smithi160.stdout:1265: done (15 left) 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1268: done (14 left) 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1270: done (13 left) 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1273: done (12 left) 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1281: done (11 left) 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1290: write (excl) oid 299 current snap is 0 2024-09-21T18:08:17.109 INFO:tasks.rados.rados.0.smithi160.stdout:1290: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-21T18:08:17.126 INFO:tasks.rados.rados.0.smithi160.stdout:1290: writing smithi16030472-299 from 487601 to 1283849 tid 1 2024-09-21T18:08:17.134 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:17.129+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:17.145 INFO:tasks.rados.rados.0.smithi160.stdout:1290: writing smithi16030472-299 from 1962121 to 2487601 tid 2 2024-09-21T18:08:17.145 INFO:tasks.rados.rados.0.smithi160.stdout:1291: write (excl) oid 476 current snap is 0 2024-09-21T18:08:17.145 INFO:tasks.rados.rados.0.smithi160.stdout:1291: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-21T18:08:17.159 INFO:tasks.rados.rados.0.smithi160.stdout:1291: writing smithi16030472-476 from 504408 to 975905 tid 1 2024-09-21T18:08:17.173 INFO:tasks.rados.rados.0.smithi160.stdout:1291: writing smithi16030472-476 from 1620603 to 2178528 tid 2 2024-09-21T18:08:17.173 INFO:tasks.rados.rados.0.smithi160.stdout:1292: delete oid 206 current snap is 0 2024-09-21T18:08:17.173 INFO:tasks.rados.rados.0.smithi160.stdout:1287: finishing write tid 1 to smithi16030472-122 2024-09-21T18:08:17.173 INFO:tasks.rados.rados.0.smithi160.stdout:1280: expect (ObjNum 251 snap 0 seq_num 251) 2024-09-21T18:08:17.219 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.219 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:17.268 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:17.268 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:17.261+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:1291: finishing write tid 1 to smithi16030472-476 2024-09-21T18:08:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:1262: finishing write tid 3 to smithi16030472-494 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1262: finishing write tid 4 to smithi16030472-494 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1262: oid 494 updating version 0 to 525 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1262: oid 494 version 525 is already newer than 522 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1262: oid 494 updating version 525 to 526 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 526 (ObjNum 735 snap 0 seq_num 735) dirty exists 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1262: left oid 494 (ObjNum 735 snap 0 seq_num 735) 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1283: finishing write tid 1 to smithi16030472-472 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1272: finishing copy_from to smithi16030472-197 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 430 (ObjNum 566 snap 0 seq_num 566) dirty exists 2024-09-21T18:08:17.295 INFO:tasks.rados.rados.0.smithi160.stdout:1276: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:17.333 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1005860 2024-09-21T18:08:17.336 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:17.333+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:17.379 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.379 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.389 INFO:tasks.rados.rados.0.smithi160.stdout:1272: finishing copy_from racing read to smithi16030472-197 2024-09-21T18:08:17.390 INFO:tasks.rados.rados.0.smithi160.stdout:1284: expect (ObjNum 230 snap 0 seq_num 230) 2024-09-21T18:08:17.393 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1477038 2024-09-21T18:08:17.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:17.435 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2668268 2024-09-21T18:08:17.443 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:17.439+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:17.468 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3629704 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: finishing write tid 2 to smithi16030472-472 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: finishing write tid 3 to smithi16030472-472 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: finishing write tid 4 to smithi16030472-472 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: oid 472 updating version 0 to 528 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: oid 472 updating version 528 to 529 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: oid 472 version 529 is already newer than 527 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 472 v 529 (ObjNum 741 snap 0 seq_num 741) dirty exists 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1283: left oid 472 (ObjNum 741 snap 0 seq_num 741) 2024-09-21T18:08:17.476 INFO:tasks.rados.rados.0.smithi160.stdout:1288: expect (ObjNum 152 snap 0 seq_num 152) 2024-09-21T18:08:17.503 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 554365 2024-09-21T18:08:17.520 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.521 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:17.537 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:17.533+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: finishing write tid 2 to smithi16030472-476 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: finishing write tid 3 to smithi16030472-476 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: finishing write tid 4 to smithi16030472-476 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: oid 476 updating version 0 to 563 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: oid 476 version 563 is already newer than 562 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: oid 476 version 563 is already newer than 561 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 476 v 563 (ObjNum 744 snap 0 seq_num 744) dirty exists 2024-09-21T18:08:17.549 INFO:tasks.rados.rados.0.smithi160.stdout:1291: left oid 476 (ObjNum 744 snap 0 seq_num 744) 2024-09-21T18:08:17.550 INFO:tasks.rados.rados.0.smithi160.stdout:1286: finishing copy_from to smithi16030472-117 2024-09-21T18:08:17.550 INFO:tasks.rados.rados.0.smithi160.stdout:1286: got expected ENOENT (src dne) 2024-09-21T18:08:17.550 INFO:tasks.rados.rados.0.smithi160.stdout:1286: finishing copy_from racing read to smithi16030472-117 2024-09-21T18:08:17.550 INFO:tasks.rados.rados.0.smithi160.stdout:1287: finishing write tid 2 to smithi16030472-122 2024-09-21T18:08:17.550 INFO:tasks.rados.rados.0.smithi160.stdout:1287: finishing write tid 3 to smithi16030472-122 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: finishing write tid 1 to smithi16030472-299 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: finishing write tid 2 to smithi16030472-299 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1289: finishing copy_from to smithi16030472-350 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 425 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: finishing write tid 3 to smithi16030472-299 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: finishing write tid 4 to smithi16030472-299 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: oid 299 updating version 0 to 486 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: oid 299 version 486 is already newer than 484 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: oid 299 version 486 is already newer than 485 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 299 v 486 (ObjNum 743 snap 0 seq_num 743) dirty exists 2024-09-21T18:08:17.579 INFO:tasks.rados.rados.0.smithi160.stdout:1290: left oid 299 (ObjNum 743 snap 0 seq_num 743) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1289: finishing copy_from racing read to smithi16030472-350 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1262: done (13 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1272: done (12 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1276: done (11 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1280: done (10 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1283: done (9 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1284: done (8 left) 2024-09-21T18:08:17.585 INFO:tasks.rados.rados.0.smithi160.stdout:1286: done (7 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1288: done (6 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1289: done (5 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1290: done (4 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1291: done (3 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1292: done (2 left) 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1293: cache_evict oid 36 snap -1 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1294: write oid 471 current snap is 0 2024-09-21T18:08:17.586 INFO:tasks.rados.rados.0.smithi160.stdout:1294: seq_num 746 ranges {538022=738348,1737567=561594} 2024-09-21T18:08:17.597 INFO:tasks.rados.rados.0.smithi160.stdout:1294: writing smithi16030472-471 from 538022 to 1276370 tid 1 2024-09-21T18:08:17.607 INFO:tasks.rados.rados.0.smithi160.stdout:1294: writing smithi16030472-471 from 1737567 to 2299161 tid 2 2024-09-21T18:08:17.607 INFO:tasks.rados.rados.0.smithi160.stdout:1295: cache_try_flush oid 355 snap -1 2024-09-21T18:08:17.607 INFO:tasks.rados.rados.0.smithi160.stdout:1296: write oid 329 current snap is 0 2024-09-21T18:08:17.608 INFO:tasks.rados.rados.0.smithi160.stdout:1296: seq_num 747 ranges {554829=413597,1396049=705252} 2024-09-21T18:08:17.617 INFO:tasks.rados.rados.0.smithi160.stdout:1296: writing smithi16030472-329 from 554829 to 968426 tid 1 2024-09-21T18:08:17.628 INFO:tasks.rados.rados.0.smithi160.stdout:1296: writing smithi16030472-329 from 1396049 to 2101301 tid 2 2024-09-21T18:08:17.628 INFO:tasks.rados.rados.0.smithi160.stdout:1297: cache_evict oid 432 snap -1 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: finishing write tid 4 to smithi16030472-122 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: oid 122 updating version 0 to 549 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: oid 122 updating version 549 to 565 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: oid 122 version 565 is already newer than 564 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 122 v 565 (ObjNum 742 snap 0 seq_num 742) dirty exists 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: left oid 122 (ObjNum 742 snap 0 seq_num 742) 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1295: got (0) Success 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 0 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1287: done (6 left) 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1295: done (5 left) 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1298: write oid 346 current snap is 0 2024-09-21T18:08:17.629 INFO:tasks.rados.rados.0.smithi160.stdout:1298: seq_num 748 ranges {571636=488846,1854531=565263} 2024-09-21T18:08:17.638 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:17.637+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:1298: writing smithi16030472-346 from 571636 to 1060482 tid 1 2024-09-21T18:08:17.652 INFO:tasks.rados.rados.0.smithi160.stdout:1298: writing smithi16030472-346 from 1854531 to 2419794 tid 2 2024-09-21T18:08:17.652 INFO:tasks.rados.rados.0.smithi160.stdout:1299: write (excl) oid 32 current snap is 0 2024-09-21T18:08:17.652 INFO:tasks.rados.rados.0.smithi160.stdout:1299: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-21T18:08:17.662 INFO:tasks.rados.rados.0.smithi160.stdout:1299: writing smithi16030472-32 from 588443 to 1152538 tid 1 2024-09-21T18:08:17.672 INFO:tasks.rados.rados.0.smithi160.stdout:1299: writing smithi16030472-32 from 1796660 to 2505581 tid 2 2024-09-21T18:08:17.672 INFO:tasks.rados.rados.0.smithi160.stdout:1300: read oid 392 snap -1 2024-09-21T18:08:17.672 INFO:tasks.rados.rados.0.smithi160.stdout:1300: expect deleted 2024-09-21T18:08:17.673 INFO:tasks.rados.rados.0.smithi160.stdout:1301: write (excl) oid 142 current snap is 0 2024-09-21T18:08:17.673 INFO:tasks.rados.rados.0.smithi160.stdout:1301: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-21T18:08:17.681 INFO:tasks.rados.rados.0.smithi160.stdout:1301: writing smithi16030472-142 from 605250 to 1244594 tid 1 2024-09-21T18:08:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:1301: writing smithi16030472-142 from 1855142 to 2307721 tid 2 2024-09-21T18:08:17.688 INFO:tasks.rados.rados.0.smithi160.stdout:1302: delete oid 287 current snap is 0 2024-09-21T18:08:17.689 INFO:tasks.rados.rados.0.smithi160.stdout:1285: expect (ObjNum 276 snap 0 seq_num 276) 2024-09-21T18:08:17.704 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.704 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:17.738 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:17.737+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:17.768 INFO:tasks.rados.rados.0.smithi160.stdout:1293: got (0) Success 2024-09-21T18:08:17.791 INFO:tasks.rados.rados.0.smithi160.stdout:1297: got (0) Success 2024-09-21T18:08:17.805 INFO:tasks.rados.rados.0.smithi160.stdout:1294: finishing write tid 1 to smithi16030472-471 2024-09-21T18:08:17.825 INFO:tasks.rados.rados.0.smithi160.stdout:1294: finishing write tid 2 to smithi16030472-471 2024-09-21T18:08:17.839 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:17.835+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:17.849 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:17.849 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:17.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:17.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:08:17.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: finishing write tid 3 to smithi16030472-471 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: finishing write tid 4 to smithi16030472-471 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: oid 471 updating version 0 to 568 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: oid 471 version 568 is already newer than 566 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: oid 471 version 568 is already newer than 567 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 471 v 568 (ObjNum 746 snap 0 seq_num 746) dirty exists 2024-09-21T18:08:17.926 INFO:tasks.rados.rados.0.smithi160.stdout:1294: left oid 471 (ObjNum 746 snap 0 seq_num 746) 2024-09-21T18:08:17.933 INFO:tasks.rados.rados.0.smithi160.stdout:1296: finishing write tid 1 to smithi16030472-329 2024-09-21T18:08:17.939 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:17.937+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:17.942 INFO:tasks.rados.rados.0.smithi160.stdout:1298: finishing write tid 1 to smithi16030472-346 2024-09-21T18:08:18.034 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:18.034 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:18.040 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:18.035+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:18.043 INFO:tasks.rados.rados.0.smithi160.stdout:1298: finishing write tid 2 to smithi16030472-346 2024-09-21T18:08:18.043 INFO:tasks.rados.rados.0.smithi160.stdout:1298: finishing write tid 3 to smithi16030472-346 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:1298: finishing write tid 4 to smithi16030472-346 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:1298: oid 346 updating version 0 to 525 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:1298: oid 346 version 525 is already newer than 523 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:1298: oid 346 version 525 is already newer than 524 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 346 v 525 (ObjNum 748 snap 0 seq_num 748) dirty exists 2024-09-21T18:08:18.044 INFO:tasks.rados.rados.0.smithi160.stdout:1298: left oid 346 (ObjNum 748 snap 0 seq_num 748) 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: finishing write tid 2 to smithi16030472-329 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: finishing write tid 3 to smithi16030472-329 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: finishing write tid 4 to smithi16030472-329 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: oid 329 updating version 0 to 531 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: oid 329 version 531 is already newer than 530 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:1296: oid 329 version 531 is already newer than 529 2024-09-21T18:08:18.045 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 329 v 531 (ObjNum 747 snap 0 seq_num 747) dirty exists 2024-09-21T18:08:18.046 INFO:tasks.rados.rados.0.smithi160.stdout:1296: left oid 329 (ObjNum 747 snap 0 seq_num 747) 2024-09-21T18:08:18.050 INFO:tasks.rados.rados.0.smithi160.stdout:1299: finishing write tid 1 to smithi16030472-32 2024-09-21T18:08:18.050 INFO:tasks.rados.rados.0.smithi160.stdout:1299: finishing write tid 2 to smithi16030472-32 2024-09-21T18:08:18.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:18.053 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:18.074 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:18.075 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:18.095 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:18.096 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:18.116 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:18.117 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:18.133 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:18.133 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:18.141 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:18.141+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:18.150 INFO:tasks.rados.rados.0.smithi160.stdout:1299: finishing write tid 3 to smithi16030472-32 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1299: finishing write tid 4 to smithi16030472-32 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1299: oid 32 updating version 0 to 532 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1299: oid 32 updating version 532 to 534 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1299: oid 32 version 534 is already newer than 533 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 32 v 534 (ObjNum 749 snap 0 seq_num 749) dirty exists 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1299: left oid 32 (ObjNum 749 snap 0 seq_num 749) 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1301: finishing write tid 1 to smithi16030472-142 2024-09-21T18:08:18.151 INFO:tasks.rados.rados.0.smithi160.stdout:1301: finishing write tid 2 to smithi16030472-142 2024-09-21T18:08:18.154 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:18.154 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 64.960853505000003, 2024-09-21T18:08:18.173 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:18.175 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.176 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.179 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.180 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.181 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.416915955999997, 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.182 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.183 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.184 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.187 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.255992491999997, 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.188 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.190 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.255317896999998, 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.193 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:08:18.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.196 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.196 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:08:18.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.255291358999997, 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.199 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:08:18.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:08:18.201 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.245457617, 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.202 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:08:18.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:08:18.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.233045828999998, 2024-09-21T18:08:18.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:08:18.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:08:18.209 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.210 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:08:18.212 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.224155758999999, 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.213 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:08:18.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:08:18.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:08:18.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:18.219 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.223343741999997, 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:08:18.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:08:18.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:08:18.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.223235529999997, 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:08:18.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:08:18.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:08:18.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:08:18.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.122964383999999, 2024-09-21T18:08:18.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:08:18.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:08:18.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.235 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.122068124999998, 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.121586512, 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:08:18.243 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.121134087000002, 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.244 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.121083761000001, 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.248 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.995350981000001, 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.252 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:08:18.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.991072924000001, 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:08:18.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:08:18.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:08:18.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.264 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.990702683999999, 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.265 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:08:18.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:08:18.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:08:18.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:08:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.990632953999999, 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:08:18.271 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:08:18.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:08:18.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 46.399055963999999, 2024-09-21T18:08:18.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:18.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:18.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:18.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:18.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:18.282 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:18.283 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:18.284 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:18.284 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:18.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:18.286 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:18.286 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:18.287 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:18.287 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:18.288 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:18.288 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:18.288 INFO:tasks.rados.rados.0.smithi160.stdout:1301: finishing write tid 3 to smithi16030472-142 2024-09-21T18:08:18.288 INFO:tasks.rados.rados.0.smithi160.stdout:1301: finishing write tid 4 to smithi16030472-142 2024-09-21T18:08:18.288 INFO:tasks.rados.rados.0.smithi160.stdout:1301: oid 142 updating version 0 to 535 2024-09-21T18:08:18.288 INFO:tasks.rados.rados.0.smithi160.stdout:1301: oid 142 updating version 535 to 536 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1301: oid 142 updating version 536 to 537 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 537 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1301: left oid 142 (ObjNum 750 snap 0 seq_num 750) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1285: done (9 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1293: done (8 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1294: done (7 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1296: done (6 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1297: done (5 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1298: done (4 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1299: done (3 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1300: done (2 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1301: done (1 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1302: done (0 left) 2024-09-21T18:08:18.289 INFO:tasks.rados.rados.0.smithi160.stdout:1303: copy_from oid 245 from oid 326 current snap is 0 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1304: cache_try_flush oid 185 snap -1 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1305: write (excl) oid 146 current snap is 0 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1305: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1305: writing smithi16030472-146 from 638864 to 1428706 tid 1 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1305: writing smithi16030472-146 from 1972106 to 2428354 tid 2 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1306: write (excl) oid 151 current snap is 0 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1306: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1306: writing smithi16030472-151 from 655671 to 1237115 tid 1 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1306: writing smithi16030472-151 from 2030588 to 2655671 tid 2 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1307: copy_from oid 63 from oid 416 current snap is 0 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1308: read oid 334 snap -1 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1308: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1309: cache_evict oid 358 snap -1 2024-09-21T18:08:18.290 INFO:tasks.rados.rados.0.smithi160.stdout:1310: write (excl) oid 198 current snap is 0 2024-09-21T18:08:18.291 INFO:tasks.rados.rados.0.smithi160.stdout:1310: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-21T18:08:18.291 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:18.291 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:18.291 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:18.291 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:18.292 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:18.281+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:18.292 INFO:tasks.rados.rados.0.smithi160.stdout:1310: writing smithi16030472-198 from 672478 to 1329171 tid 1 2024-09-21T18:08:18.300 INFO:tasks.rados.rados.0.smithi160.stdout:1310: writing smithi16030472-198 from 2089070 to 2548987 tid 2 2024-09-21T18:08:18.300 INFO:tasks.rados.rados.0.smithi160.stdout:1311: write oid 26 current snap is 0 2024-09-21T18:08:18.300 INFO:tasks.rados.rados.0.smithi160.stdout:1311: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-21T18:08:18.301 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:18.301 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:18.309 INFO:tasks.rados.rados.0.smithi160.stdout:1311: writing smithi16030472-26 from 689285 to 1421227 tid 1 2024-09-21T18:08:18.317 INFO:tasks.rados.rados.0.smithi160.stdout:1311: writing smithi16030472-26 from 2147552 to 2689285 tid 2 2024-09-21T18:08:18.317 INFO:tasks.rados.rados.0.smithi160.stdout:1312: read oid 446 snap -1 2024-09-21T18:08:18.317 INFO:tasks.rados.rados.0.smithi160.stdout:1312: expect (ObjNum 538 snap 0 seq_num 538) 2024-09-21T18:08:18.317 INFO:tasks.rados.rados.0.smithi160.stdout:1313: write oid 394 current snap is 0 2024-09-21T18:08:18.317 INFO:tasks.rados.rados.0.smithi160.stdout:1313: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-21T18:08:18.323 INFO:tasks.rados.rados.0.smithi160.stdout:1313: writing smithi16030472-394 from 706092 to 1113283 tid 1 2024-09-21T18:08:18.323 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:18.330 INFO:tasks.rados.rados.0.smithi160.stdout:1313: writing smithi16030472-394 from 1806034 to 2269620 tid 2 2024-09-21T18:08:18.330 INFO:tasks.rados.rados.0.smithi160.stdout:1314: copy_from oid 363 from oid 28 current snap is 0 2024-09-21T18:08:18.330 INFO:tasks.rados.rados.0.smithi160.stdout:1315: delete oid 135 current snap is 0 2024-09-21T18:08:18.343 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:18.343 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:18.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:18.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:18.353 INFO:tasks.rados.rados.0.smithi160.stdout:1304: got (0) Success 2024-09-21T18:08:18.353 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 185 v 0 (ObjNum 33 snap 0 seq_num 3131250336) clean dne 2024-09-21T18:08:18.359 INFO:tasks.rados.rados.0.smithi160.stdout:1303: finishing copy_from racing read to smithi16030472-245 2024-09-21T18:08:18.384 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:18.381+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:18.436 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:18.436 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:18.436 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:18.436 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:18.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:1303: finishing copy_from to smithi16030472-245 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 245 v 274 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:1303: done (12 left) 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:1304: done (11 left) 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:1315: done (10 left) 2024-09-21T18:08:18.465 INFO:tasks.rados.rados.0.smithi160.stdout:1316: write oid 35 current snap is 0 2024-09-21T18:08:18.466 INFO:tasks.rados.rados.0.smithi160.stdout:1316: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-21T18:08:18.473 INFO:tasks.rados.rados.0.smithi160.stdout:1316: writing smithi16030472-35 from 739706 to 1297395 tid 1 2024-09-21T18:08:18.479 INFO:tasks.rados.rados.0.smithi160.stdout:1316: writing smithi16030472-35 from 1806645 to 2273900 tid 2 2024-09-21T18:08:18.479 INFO:tasks.rados.rados.0.smithi160.stdout:1317: cache_flush oid 117 snap -1 2024-09-21T18:08:18.479 INFO:tasks.rados.rados.0.smithi160.stdout:1318: write oid 56 current snap is 0 2024-09-21T18:08:18.479 INFO:tasks.rados.rados.0.smithi160.stdout:1318: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-21T18:08:18.486 INFO:tasks.rados.rados.0.smithi160.stdout:1318: writing smithi16030472-56 from 756513 to 1389451 tid 1 2024-09-21T18:08:18.493 INFO:tasks.rados.rados.0.smithi160.stdout:1318: writing smithi16030472-56 from 1865127 to 2592393 tid 2 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout:1319: copy_from oid 48 from oid 81 current snap is 0 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout:1316: finishing write tid 1 to smithi16030472-35 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout:1320: read oid 302 snap -1 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout:1320: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout:1321: copy_from oid 228 from oid 301 current snap is 0 2024-09-21T18:08:18.494 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:18.495 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:18.491+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:18.577 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:18.577 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:18.577 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:18.577 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:18.587 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:18.581+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:18.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:18.687 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:18.685+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:18.698 INFO:tasks.rados.rados.0.smithi160.stdout:1316: finishing write tid 2 to smithi16030472-35 2024-09-21T18:08:18.698 INFO:tasks.rados.rados.0.smithi160.stdout:1318: finishing write tid 1 to smithi16030472-56 2024-09-21T18:08:18.705 INFO:tasks.rados.rados.0.smithi160.stdout:1320: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:08:18.705 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:18.760 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:18.761 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:18.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:18.761 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:18.761 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:18.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:18.788 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:18.785+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:18.871 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:08:18.886 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 454861 2024-09-21T18:08:18.889 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:18.889+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:18.899 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:18.899 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:18.899 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:18.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:18.945 INFO:tasks.rados.rados.0.smithi160.stdout:1318: finishing write tid 2 to smithi16030472-56 2024-09-21T18:08:18.945 INFO:tasks.rados.rados.0.smithi160.stdout:1316: finishing write tid 3 to smithi16030472-35 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: finishing write tid 4 to smithi16030472-35 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: oid 35 updating version 0 to 531 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: oid 35 updating version 531 to 532 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: oid 35 version 532 is already newer than 530 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 532 (ObjNum 758 snap 0 seq_num 758) dirty exists 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: left oid 35 (ObjNum 758 snap 0 seq_num 758) 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1318: finishing write tid 3 to smithi16030472-56 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1305: finishing write tid 1 to smithi16030472-146 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1316: done (15 left) 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1320: done (14 left) 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1322: cache_evict oid 182 snap -1 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1305: finishing write tid 2 to smithi16030472-146 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1305: finishing write tid 3 to smithi16030472-146 2024-09-21T18:08:18.946 INFO:tasks.rados.rados.0.smithi160.stdout:1305: finishing write tid 4 to smithi16030472-146 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:1305: oid 146 updating version 0 to 570 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:1305: oid 146 version 570 is already newer than 569 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:1305: oid 146 updating version 570 to 571 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 571 (ObjNum 752 snap 0 seq_num 752) dirty exists 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:1305: left oid 146 (ObjNum 752 snap 0 seq_num 752) 2024-09-21T18:08:18.947 INFO:tasks.rados.rados.0.smithi160.stdout:1308: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:08:18.954 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2215386 2024-09-21T18:08:18.990 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:18.989+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:18.996 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:19.026 INFO:tasks.rados.rados.0.smithi160.stdout:1310: finishing write tid 1 to smithi16030472-198 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1307: finishing copy_from racing read to smithi16030472-63 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1318: finishing write tid 4 to smithi16030472-56 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1318: oid 56 updating version 0 to 528 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1318: oid 56 version 528 is already newer than 527 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1318: oid 56 updating version 528 to 529 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 529 (ObjNum 759 snap 0 seq_num 759) dirty exists 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1318: left oid 56 (ObjNum 759 snap 0 seq_num 759) 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1317: got (0) Success 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 117 v 0 (ObjNum 268 snap 0 seq_num 268) dirty exists 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1306: finishing write tid 1 to smithi16030472-151 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1306: finishing write tid 2 to smithi16030472-151 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1305: done (14 left) 2024-09-21T18:08:19.027 INFO:tasks.rados.rados.0.smithi160.stdout:1308: done (13 left) 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1317: done (12 left) 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1318: done (11 left) 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1323: delete oid 163 current snap is 0 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1322: got (2) No such file or directory 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: finishing write tid 3 to smithi16030472-151 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: finishing write tid 4 to smithi16030472-151 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: oid 151 updating version 0 to 483 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: oid 151 version 483 is already newer than 482 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: oid 151 version 483 is already newer than 481 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 151 v 483 (ObjNum 753 snap 0 seq_num 753) dirty exists 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1306: left oid 151 (ObjNum 753 snap 0 seq_num 753) 2024-09-21T18:08:19.028 INFO:tasks.rados.rados.0.smithi160.stdout:1312: expect (ObjNum 538 snap 0 seq_num 538) 2024-09-21T18:08:19.046 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2364068 2024-09-21T18:08:19.083 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:19.084 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:19.084 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:19.084 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:19.084 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:19.091 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:19.089+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:19.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:19.139 INFO:tasks.rados.rados.0.smithi160.stdout:1309: got (0) Success 2024-09-21T18:08:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:1310: finishing write tid 2 to smithi16030472-198 2024-09-21T18:08:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:1310: finishing write tid 3 to smithi16030472-198 2024-09-21T18:08:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:1310: finishing write tid 4 to smithi16030472-198 2024-09-21T18:08:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:1310: oid 198 updating version 0 to 550 2024-09-21T18:08:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:1310: oid 198 updating version 550 to 573 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1310: oid 198 version 573 is already newer than 572 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 198 v 573 (ObjNum 754 snap 0 seq_num 754) dirty exists 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1310: left oid 198 (ObjNum 754 snap 0 seq_num 754) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1306: done (11 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1309: done (10 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1310: done (9 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1312: done (8 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1322: done (7 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1323: done (6 left) 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1324: copy_from oid 499 from oid 1 current snap is 0 2024-09-21T18:08:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:1307: finishing copy_from to smithi16030472-63 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 63 v 285 (ObjNum 415 snap 0 seq_num 415) dirty exists 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:1307: done (6 left) 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:1325: copy_from oid 97 from oid 347 current snap is 0 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:1326: cache_flush oid 155 snap -1 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:1327: write oid 124 current snap is 0 2024-09-21T18:08:19.142 INFO:tasks.rados.rados.0.smithi160.stdout:1327: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-21T18:08:19.146 INFO:tasks.rados.rados.0.smithi160.stdout:1327: writing smithi16030472-124 from 790127 to 1289916 tid 1 2024-09-21T18:08:19.152 INFO:tasks.rados.rados.0.smithi160.stdout:1327: writing smithi16030472-124 from 1698444 to 2429379 tid 2 2024-09-21T18:08:19.152 INFO:tasks.rados.rados.0.smithi160.stdout:1328: copy_from oid 446 from oid 42 current snap is 0 2024-09-21T18:08:19.153 INFO:tasks.rados.rados.0.smithi160.stdout:1329: write (excl) oid 243 current snap is 0 2024-09-21T18:08:19.153 INFO:tasks.rados.rados.0.smithi160.stdout:1329: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-21T18:08:19.158 INFO:tasks.rados.rados.0.smithi160.stdout:1329: writing smithi16030472-243 from 406934 to 981972 tid 1 2024-09-21T18:08:19.164 INFO:tasks.rados.rados.0.smithi160.stdout:1329: writing smithi16030472-243 from 1640573 to 2115166 tid 2 2024-09-21T18:08:19.168 INFO:tasks.rados.rados.0.smithi160.stdout:1329: writing smithi16030472-243 from 2642791 to 2806934 tid 3 2024-09-21T18:08:19.168 INFO:tasks.rados.rados.0.smithi160.stdout:1330: delete oid 497 current snap is 0 2024-09-21T18:08:19.183 INFO:tasks.rados.rados.0.smithi160.stdout:1311: finishing write tid 1 to smithi16030472-26 2024-09-21T18:08:19.191 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:19.187+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:19.200 INFO:tasks.rados.rados.0.smithi160.stdout:1321: finishing copy_from racing read to smithi16030472-228 2024-09-21T18:08:19.200 INFO:tasks.rados.rados.0.smithi160.stdout:1319: finishing copy_from racing read to smithi16030472-48 2024-09-21T18:08:19.200 INFO:tasks.rados.rados.0.smithi160.stdout:1311: finishing write tid 2 to smithi16030472-26 2024-09-21T18:08:19.200 INFO:tasks.rados.rados.0.smithi160.stdout:1311: finishing write tid 3 to smithi16030472-26 2024-09-21T18:08:19.201 INFO:tasks.rados.rados.0.smithi160.stdout:1311: finishing write tid 4 to smithi16030472-26 2024-09-21T18:08:19.201 INFO:tasks.rados.rados.0.smithi160.stdout:1311: oid 26 updating version 0 to 575 2024-09-21T18:08:19.201 INFO:tasks.rados.rados.0.smithi160.stdout:1311: oid 26 updating version 575 to 576 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1311: oid 26 version 576 is already newer than 574 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 576 (ObjNum 755 snap 0 seq_num 755) dirty exists 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1311: left oid 26 (ObjNum 755 snap 0 seq_num 755) 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: finishing write tid 1 to smithi16030472-394 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: finishing write tid 2 to smithi16030472-394 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: finishing write tid 3 to smithi16030472-394 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: finishing write tid 4 to smithi16030472-394 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: oid 394 updating version 0 to 577 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: oid 394 updating version 577 to 579 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: oid 394 version 579 is already newer than 578 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 394 v 579 (ObjNum 756 snap 0 seq_num 756) dirty exists 2024-09-21T18:08:19.202 INFO:tasks.rados.rados.0.smithi160.stdout:1313: left oid 394 (ObjNum 756 snap 0 seq_num 756) 2024-09-21T18:08:19.203 INFO:tasks.rados.rados.0.smithi160.stdout:1314: finishing copy_from racing read to smithi16030472-363 2024-09-21T18:08:19.204 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:19.195+0000 7fb924fb6640 -1 osd.8 74 handle_osd_map injecting map crc failure 2024-09-21T18:08:19.216 INFO:tasks.rados.rados.0.smithi160.stdout:1328: finishing copy_from racing read to smithi16030472-446 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 93.345608999999996, 2024-09-21T18:08:19.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:19.247 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 92.260216611000004, 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:19.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:08:19.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:08:19.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:08:19.251 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:19.252 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:19.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:19.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:19.252 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:19.252 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:19.254 INFO:tasks.rados.rados.0.smithi160.stdout:1325: finishing copy_from racing read to smithi16030472-97 2024-09-21T18:08:19.258 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:19.269 INFO:tasks.rados.rados.0.smithi160.stdout:1326: got (2) No such file or directory 2024-09-21T18:08:19.269 INFO:tasks.rados.rados.0.smithi160.stdout:1324: finishing copy_from to smithi16030472-499 2024-09-21T18:08:19.269 INFO:tasks.rados.rados.0.smithi160.stdout:1324: got expected ENOENT (src dne) 2024-09-21T18:08:19.269 INFO:tasks.rados.rados.0.smithi160.stdout:1324: finishing copy_from racing read to smithi16030472-499 2024-09-21T18:08:19.307 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:19.305+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:19.312 INFO:tasks.rados.rados.0.smithi160.stdout:1325: finishing copy_from to smithi16030472-97 2024-09-21T18:08:19.312 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 97 v 125 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:08:19.340 INFO:tasks.rados.rados.0.smithi160.stdout:1327: finishing write tid 1 to smithi16030472-124 2024-09-21T18:08:19.354 INFO:tasks.rados.rados.0.smithi160.stdout:1327: finishing write tid 2 to smithi16030472-124 2024-09-21T18:08:19.365 INFO:tasks.rados.rados.0.smithi160.stdout:1327: finishing write tid 3 to smithi16030472-124 2024-09-21T18:08:19.365 INFO:tasks.rados.rados.0.smithi160.stdout:1327: finishing write tid 4 to smithi16030472-124 2024-09-21T18:08:19.365 INFO:tasks.rados.rados.0.smithi160.stdout:1327: oid 124 updating version 0 to 539 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:1327: oid 124 updating version 539 to 540 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:1327: oid 124 updating version 540 to 541 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 124 v 541 (ObjNum 761 snap 0 seq_num 761) dirty exists 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:1327: left oid 124 (ObjNum 761 snap 0 seq_num 761) 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:1329: finishing write tid 1 to smithi16030472-243 2024-09-21T18:08:19.366 INFO:tasks.rados.rados.0.smithi160.stdout:1329: finishing write tid 2 to smithi16030472-243 2024-09-21T18:08:19.394 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:19.387+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:19.396 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:19.396 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:19.396 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:19.396 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:19.405 INFO:tasks.rados.rados.0.smithi160.stdout:1314: finishing copy_from to smithi16030472-363 2024-09-21T18:08:19.405 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:08:19.411 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:19.495 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:19.491+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:19.505 INFO:tasks.rados.rados.0.smithi160.stdout:1319: finishing copy_from to smithi16030472-48 2024-09-21T18:08:19.505 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 48 v 57 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-09-21T18:08:19.505 INFO:tasks.rados.rados.0.smithi160.stdout:1321: finishing copy_from to smithi16030472-228 2024-09-21T18:08:19.506 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 228 v 253 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-21T18:08:19.506 INFO:tasks.rados.rados.0.smithi160.stdout:1311: done (11 left) 2024-09-21T18:08:19.506 INFO:tasks.rados.rados.0.smithi160.stdout:1313: done (10 left) 2024-09-21T18:08:19.506 INFO:tasks.rados.rados.0.smithi160.stdout:1314: done (9 left) 2024-09-21T18:08:19.506 INFO:tasks.rados.rados.0.smithi160.stdout:1319: done (8 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1321: done (7 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1324: done (6 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1325: done (5 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1326: done (4 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1327: done (3 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1330: done (2 left) 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1331: write oid 436 current snap is 0 2024-09-21T18:08:19.507 INFO:tasks.rados.rados.0.smithi160.stdout:1331: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-21T18:08:19.516 INFO:tasks.rados.rados.0.smithi160.stdout:1331: writing smithi16030472-436 from 440548 to 1166084 tid 1 2024-09-21T18:08:19.526 INFO:tasks.rados.rados.0.smithi160.stdout:1331: writing smithi16030472-436 from 1757537 to 2235799 tid 2 2024-09-21T18:08:19.531 INFO:tasks.rados.rados.0.smithi160.stdout:1331: writing smithi16030472-436 from 2813990 to 2840548 tid 3 2024-09-21T18:08:19.531 INFO:tasks.rados.rados.0.smithi160.stdout:1332: cache_flush oid 351 snap -1 2024-09-21T18:08:19.531 INFO:tasks.rados.rados.0.smithi160.stdout:1333: delete oid 194 current snap is 0 2024-09-21T18:08:19.531 INFO:tasks.rados.rados.0.smithi160.stdout:1328: finishing copy_from to smithi16030472-446 2024-09-21T18:08:19.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 348 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:08:19.543 INFO:tasks.rados.rados.0.smithi160.stdout:1329: finishing write tid 3 to smithi16030472-243 2024-09-21T18:08:19.571 INFO:tasks.rados.rados.0.smithi160.stdout:1332: got (2) No such file or directory 2024-09-21T18:08:19.580 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:19.581 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:19.595 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:19.596 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:19.589+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: finishing write tid 4 to smithi16030472-243 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: finishing write tid 5 to smithi16030472-243 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: oid 243 updating version 0 to 543 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: oid 243 updating version 543 to 545 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: oid 243 version 545 is already newer than 544 2024-09-21T18:08:19.618 INFO:tasks.rados.rados.0.smithi160.stdout:1329: oid 243 version 545 is already newer than 542 2024-09-21T18:08:19.619 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 545 (ObjNum 762 snap 0 seq_num 762) dirty exists 2024-09-21T18:08:19.619 INFO:tasks.rados.rados.0.smithi160.stdout:1329: left oid 243 (ObjNum 762 snap 0 seq_num 762) 2024-09-21T18:08:19.636 INFO:tasks.rados.rados.0.smithi160.stdout:1328: done (4 left) 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1329: done (3 left) 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1332: done (2 left) 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1333: done (1 left) 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1334: read oid 305 snap -1 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1334: expect deleted 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1335: cache_try_flush oid 95 snap -1 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1336: cache_flush oid 303 snap -1 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1337: write oid 90 current snap is 0 2024-09-21T18:08:19.637 INFO:tasks.rados.rados.0.smithi160.stdout:1337: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-21T18:08:19.646 INFO:tasks.rados.rados.0.smithi160.stdout:1337: writing smithi16030472-90 from 474162 to 950196 tid 1 2024-09-21T18:08:19.654 INFO:tasks.rados.rados.0.smithi160.stdout:1337: writing smithi16030472-90 from 1358148 to 1840079 tid 2 2024-09-21T18:08:19.664 INFO:tasks.rados.rados.0.smithi160.stdout:1337: writing smithi16030472-90 from 2352483 to 2817786 tid 3 2024-09-21T18:08:19.664 INFO:tasks.rados.rados.0.smithi160.stdout:1338: read oid 342 snap -1 2024-09-21T18:08:19.665 INFO:tasks.rados.rados.0.smithi160.stdout:1338: expect (ObjNum 465 snap 0 seq_num 465) 2024-09-21T18:08:19.665 INFO:tasks.rados.rados.0.smithi160.stdout:1339: read oid 97 snap -1 2024-09-21T18:08:19.665 INFO:tasks.rados.rados.0.smithi160.stdout:1339: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:08:19.665 INFO:tasks.rados.rados.0.smithi160.stdout:1340: write (excl) oid 416 current snap is 0 2024-09-21T18:08:19.665 INFO:tasks.rados.rados.0.smithi160.stdout:1340: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-21T18:08:19.675 INFO:tasks.rados.rados.0.smithi160.stdout:1340: writing smithi16030472-416 from 490969 to 1042252 tid 1 2024-09-21T18:08:19.688 INFO:tasks.rados.rados.0.smithi160.stdout:1340: writing smithi16030472-416 from 1816630 to 2442219 tid 2 2024-09-21T18:08:19.688 INFO:tasks.rados.rados.0.smithi160.stdout:1341: write (excl) oid 62 current snap is 0 2024-09-21T18:08:19.688 INFO:tasks.rados.rados.0.smithi160.stdout:1341: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-21T18:08:19.696 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:19.691+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:19.699 INFO:tasks.rados.rados.0.smithi160.stdout:1341: writing smithi16030472-62 from 507776 to 1250661 tid 1 2024-09-21T18:08:19.708 INFO:tasks.rados.rados.0.smithi160.stdout:1341: writing smithi16030472-62 from 1991465 to 2477065 tid 2 2024-09-21T18:08:19.708 INFO:tasks.rados.rados.0.smithi160.stdout:1342: read oid 254 snap -1 2024-09-21T18:08:19.708 INFO:tasks.rados.rados.0.smithi160.stdout:1342: expect (ObjNum 253 snap 0 seq_num 253) 2024-09-21T18:08:19.708 INFO:tasks.rados.rados.0.smithi160.stdout:1343: copy_from oid 286 from oid 371 current snap is 0 2024-09-21T18:08:19.708 INFO:tasks.rados.rados.0.smithi160.stdout:1344: cache_try_flush oid 165 snap -1 2024-09-21T18:08:19.709 INFO:tasks.rados.rados.0.smithi160.stdout:1345: read oid 327 snap -1 2024-09-21T18:08:19.709 INFO:tasks.rados.rados.0.smithi160.stdout:1345: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:19.709 INFO:tasks.rados.rados.0.smithi160.stdout:1346: delete oid 349 current snap is 0 2024-09-21T18:08:19.726 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:19.726 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:19.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684463735999998, 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:19.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:08:19.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684224112000003, 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:08:19.732 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:08:19.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:08:19.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684188704999997, 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:19.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:08:19.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684155531999998, 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684125971999997, 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.747 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:08:19.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:08:19.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.751 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.684096316000002, 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:08:19.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:08:19.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.683610059999999, 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:08:19.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:08:19.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.683576467000002, 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:08:19.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:08:19.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:08:19.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.583358775000001, 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.771 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.582759871, 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:08:19.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:08:19.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.582045254000001, 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.777 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.782 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.572792522, 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:08:19.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.572527821000001, 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.572433132, 2024-09-21T18:08:19.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:19.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:08:19.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:19.797 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.572391842000002, 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:19.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:08:19.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.561121939000003, 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.802 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:08:19.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.552005706000003, 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:08:19.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:08:19.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.550942339000002, 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.813 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:08:19.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:08:19.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 48.550902029, 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.818 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:08:19.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:08:19.822 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 47.788264187000003, 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:08:19.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:08:19.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:08:19.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.828 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:19.828 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:19.828 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:19.829 INFO:tasks.rados.rados.0.smithi160.stdout:1335: got (0) Success 2024-09-21T18:08:19.829 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 95 v 0 (ObjNum 94 snap 0 seq_num 94) dirty exists 2024-09-21T18:08:19.829 INFO:tasks.rados.rados.0.smithi160.stdout:1338: expect (ObjNum 465 snap 0 seq_num 465) 2024-09-21T18:08:19.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:19.830 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:19.825+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:19.866 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1079812 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:1336: got (0) Success 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 303 v 0 (ObjNum 577 snap 0 seq_num 577) dirty exists 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:1331: finishing write tid 1 to smithi16030472-436 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:1337: finishing write tid 1 to smithi16030472-90 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:1337: finishing write tid 2 to smithi16030472-90 2024-09-21T18:08:19.895 INFO:tasks.rados.rados.0.smithi160.stdout:1339: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:08:19.928 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 18247 2024-09-21T18:08:19.929 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:19.921+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:19.967 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1204725 2024-09-21T18:08:19.987 INFO:tasks.rados.rados.0.smithi160.stdout:1331: finishing write tid 2 to smithi16030472-436 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: finishing write tid 3 to smithi16030472-436 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: finishing write tid 4 to smithi16030472-436 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: finishing write tid 5 to smithi16030472-436 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: oid 436 updating version 0 to 548 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: oid 436 version 548 is already newer than 547 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: oid 436 updating version 548 to 549 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: oid 436 version 549 is already newer than 546 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 436 v 549 (ObjNum 764 snap 0 seq_num 764) dirty exists 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1331: left oid 436 (ObjNum 764 snap 0 seq_num 764) 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1341: finishing write tid 1 to smithi16030472-62 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1341: finishing write tid 2 to smithi16030472-62 2024-09-21T18:08:19.988 INFO:tasks.rados.rados.0.smithi160.stdout:1341: finishing write tid 3 to smithi16030472-62 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1341: finishing write tid 4 to smithi16030472-62 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1341: oid 62 updating version 0 to 551 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1341: oid 62 updating version 551 to 552 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1341: oid 62 version 552 is already newer than 550 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 62 v 552 (ObjNum 768 snap 0 seq_num 768) dirty exists 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1341: left oid 62 (ObjNum 768 snap 0 seq_num 768) 2024-09-21T18:08:19.989 INFO:tasks.rados.rados.0.smithi160.stdout:1342: expect (ObjNum 253 snap 0 seq_num 253) 2024-09-21T18:08:20.001 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:20.002 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:20.002 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:20.002 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:20.015 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1723294 2024-09-21T18:08:20.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:20.026 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:20.030 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:20.025+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:20.050 INFO:tasks.rados.rados.0.smithi160.stdout:1340: finishing write tid 1 to smithi16030472-416 2024-09-21T18:08:20.050 INFO:tasks.rados.rados.0.smithi160.stdout:1340: finishing write tid 2 to smithi16030472-416 2024-09-21T18:08:20.058 INFO:tasks.rados.rados.0.smithi160.stdout:1337: finishing write tid 3 to smithi16030472-90 2024-09-21T18:08:20.059 INFO:tasks.rados.rados.0.smithi160.stdout:1337: finishing write tid 4 to smithi16030472-90 2024-09-21T18:08:20.084 INFO:tasks.rados.rados.0.smithi160.stdout:1340: finishing write tid 3 to smithi16030472-416 2024-09-21T18:08:20.131 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:20.129+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:20.140 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:20.140 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:20.141 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:20.141 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:20.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:20.159 INFO:tasks.rados.rados.0.smithi160.stdout:1337: finishing write tid 5 to smithi16030472-90 2024-09-21T18:08:20.159 INFO:tasks.rados.rados.0.smithi160.stdout:1337: oid 90 updating version 0 to 582 2024-09-21T18:08:20.160 INFO:tasks.rados.rados.0.smithi160.stdout:1337: oid 90 updating version 582 to 583 2024-09-21T18:08:20.160 INFO:tasks.rados.rados.0.smithi160.stdout:1337: oid 90 updating version 583 to 584 2024-09-21T18:08:20.160 INFO:tasks.rados.rados.0.smithi160.stdout:1337: oid 90 version 584 is already newer than 581 2024-09-21T18:08:20.160 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 90 v 584 (ObjNum 766 snap 0 seq_num 766) dirty exists 2024-09-21T18:08:20.160 INFO:tasks.rados.rados.0.smithi160.stdout:1337: left oid 90 (ObjNum 766 snap 0 seq_num 766) 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1340: finishing write tid 4 to smithi16030472-416 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1340: oid 416 updating version 0 to 534 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1340: oid 416 version 534 is already newer than 533 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1340: oid 416 updating version 534 to 535 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 535 (ObjNum 767 snap 0 seq_num 767) dirty exists 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1340: left oid 416 (ObjNum 767 snap 0 seq_num 767) 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:1344: got (0) Success 2024-09-21T18:08:20.185 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 165 v 0 (ObjNum 164 snap 0 seq_num 164) dirty exists 2024-09-21T18:08:20.196 INFO:tasks.rados.rados.0.smithi160.stdout:1345: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:20.196 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 168302 2024-09-21T18:08:20.233 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:20.229+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:20.244 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1694799 2024-09-21T18:08:20.282 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:20.282 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:20.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 95.979847269999993, 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:20.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:08:20.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.287 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 95.978633211000002, 2024-09-21T18:08:20.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:20.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:08:20.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:08:20.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 95.977282290999995, 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:08:20.293 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:08:20.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 89.689701654000004, 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:20.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:08:20.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:08:20.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 67.591109024000005, 2024-09-21T18:08:20.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:08:20.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:08:20.306 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 67.590631814000005, 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:08:20.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 67.590196860999995, 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:08:20.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:08:20.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.290462413999997, 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:08:20.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.189893022, 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.325 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.189534954999999, 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:08:20.326 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.327 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:08:20.328 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:08:20.329 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.330 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.189498122000003, 2024-09-21T18:08:20.331 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.332 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.333 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.334 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.335 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.18903839, 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.336 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:08:20.337 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.338 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:08:20.339 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.188606862999997, 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:08:20.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.187670248000003, 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:20.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:08:20.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.187631564999997, 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:08:20.351 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.354 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.186344912999999, 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:08:20.356 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:08:20.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:08:20.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.360 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.084200449000001, 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.365 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 49.083476656000002, 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 40.957766417000002, 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.370 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:08:20.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 16.317971929999999, 2024-09-21T18:08:20.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:08:20.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:20.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:20.381 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:20.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:20.385 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:20.385 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:20.381+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:20.388 INFO:tasks.rados.rados.0.smithi160.stdout:1331: done (13 left) 2024-09-21T18:08:20.388 INFO:tasks.rados.rados.0.smithi160.stdout:1334: done (12 left) 2024-09-21T18:08:20.388 INFO:tasks.rados.rados.0.smithi160.stdout:1335: done (11 left) 2024-09-21T18:08:20.388 INFO:tasks.rados.rados.0.smithi160.stdout:1336: done (10 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1337: done (9 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1338: done (8 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1339: done (7 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1340: done (6 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1341: done (5 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1342: done (4 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1344: done (3 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1345: done (2 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1346: done (1 left) 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1347: cache_flush oid 481 snap -1 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1348: copy_from oid 91 from oid 444 current snap is 0 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1349: write oid 466 current snap is 0 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1349: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2024-09-21T18:08:20.389 INFO:tasks.rados.rados.0.smithi160.stdout:1349: writing smithi16030472-466 from 541390 to 1034773 tid 1 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1349: writing smithi16030472-466 from 1592076 to 2081345 tid 2 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1349: writing smithi16030472-466 from 2862175 to 2941390 tid 3 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1350: delete oid 45 current snap is 0 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1349: finishing write tid 1 to smithi16030472-466 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1350: done (4 left) 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1351: copy_from oid 182 from oid 210 current snap is 0 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1352: write oid 12 current snap is 0 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1352: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1352: writing smithi16030472-12 from 575004 to 1218885 tid 1 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1352: writing smithi16030472-12 from 1709040 to 2201978 tid 2 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1352: writing smithi16030472-12 from 2917021 to 2975004 tid 3 2024-09-21T18:08:20.390 INFO:tasks.rados.rados.0.smithi160.stdout:1353: read oid 456 snap -1 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1353: expect deleted 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1354: cache_evict oid 113 snap -1 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1355: write (excl) oid 347 current snap is 0 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1355: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1355: writing smithi16030472-347 from 591811 to 1310941 tid 1 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1355: writing smithi16030472-347 from 1767522 to 2404118 tid 2 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1355: writing smithi16030472-347 from 2828091 to 2991811 tid 3 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1356: write oid 281 current snap is 0 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1356: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-21T18:08:20.391 INFO:tasks.rados.rados.0.smithi160.stdout:1356: writing smithi16030472-281 from 608618 to 1402997 tid 1 2024-09-21T18:08:20.392 INFO:tasks.rados.rados.0.smithi160.stdout:1356: writing smithi16030472-281 from 2109651 to 2889905 tid 2 2024-09-21T18:08:20.392 INFO:tasks.rados.rados.0.smithi160.stdout:1357: write (excl) oid 36 current snap is 0 2024-09-21T18:08:20.392 INFO:tasks.rados.rados.0.smithi160.stdout:1357: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-21T18:08:20.395 INFO:tasks.rados.rados.0.smithi160.stdout:1357: writing smithi16030472-36 from 625425 to 1095053 tid 1 2024-09-21T18:08:20.403 INFO:tasks.rados.rados.0.smithi160.stdout:1357: writing smithi16030472-36 from 1768133 to 2408398 tid 2 2024-09-21T18:08:20.403 INFO:tasks.rados.rados.0.smithi160.stdout:1358: cache_flush oid 31 snap -1 2024-09-21T18:08:20.403 INFO:tasks.rados.rados.0.smithi160.stdout:1359: cache_evict oid 56 snap -1 2024-09-21T18:08:20.403 INFO:tasks.rados.rados.0.smithi160.stdout:1360: cache_evict oid 50 snap -1 2024-09-21T18:08:20.404 INFO:tasks.rados.rados.0.smithi160.stdout:1361: copy_from oid 189 from oid 37 current snap is 0 2024-09-21T18:08:20.404 INFO:tasks.rados.rados.0.smithi160.stdout:1362: write (excl) oid 225 current snap is 0 2024-09-21T18:08:20.404 INFO:tasks.rados.rados.0.smithi160.stdout:1362: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-21T18:08:20.410 INFO:tasks.rados.rados.0.smithi160.stdout:1362: writing smithi16030472-225 from 642232 to 1303462 tid 1 2024-09-21T18:08:20.418 INFO:tasks.rados.rados.0.smithi160.stdout:1362: writing smithi16030472-225 from 1942968 to 2726891 tid 2 2024-09-21T18:08:20.418 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: finishing write tid 2 to smithi16030472-466 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1354: got (16) Device or resource busy 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: finishing write tid 3 to smithi16030472-466 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: finishing write tid 4 to smithi16030472-466 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: finishing write tid 5 to smithi16030472-466 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: oid 466 updating version 0 to 556 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: oid 466 version 556 is already newer than 553 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: oid 466 version 556 is already newer than 554 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: oid 466 version 556 is already newer than 555 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 466 v 556 (ObjNum 770 snap 0 seq_num 770) dirty exists 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: left oid 466 (ObjNum 770 snap 0 seq_num 770) 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1349: done (15 left) 2024-09-21T18:08:20.473 INFO:tasks.rados.rados.0.smithi160.stdout:1354: done (14 left) 2024-09-21T18:08:20.474 INFO:tasks.rados.rados.0.smithi160.stdout:1363: write oid 448 current snap is 0 2024-09-21T18:08:20.474 INFO:tasks.rados.rados.0.smithi160.stdout:1363: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-21T18:08:20.480 INFO:tasks.rados.rados.0.smithi160.stdout:1363: writing smithi16030472-448 from 659039 to 1395518 tid 1 2024-09-21T18:08:20.483 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:20.481+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:20.487 INFO:tasks.rados.rados.0.smithi160.stdout:1363: writing smithi16030472-448 from 2001450 to 2645384 tid 2 2024-09-21T18:08:20.487 INFO:tasks.rados.rados.0.smithi160.stdout:1364: read oid 269 snap -1 2024-09-21T18:08:20.487 INFO:tasks.rados.rados.0.smithi160.stdout:1364: expect deleted 2024-09-21T18:08:20.487 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.533 INFO:tasks.rados.rados.0.smithi160.stdout:1355: finishing write tid 1 to smithi16030472-347 2024-09-21T18:08:20.533 INFO:tasks.rados.rados.0.smithi160.stdout:1355: finishing write tid 2 to smithi16030472-347 2024-09-21T18:08:20.533 INFO:tasks.rados.rados.0.smithi160.stdout:1355: finishing write tid 3 to smithi16030472-347 2024-09-21T18:08:20.535 INFO:tasks.rados.rados.0.smithi160.stdout:1355: finishing write tid 4 to smithi16030472-347 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: finishing write tid 5 to smithi16030472-347 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: oid 347 updating version 0 to 557 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: oid 347 updating version 557 to 560 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: oid 347 version 560 is already newer than 559 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: oid 347 version 560 is already newer than 558 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 560 (ObjNum 773 snap 0 seq_num 773) dirty exists 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: left oid 347 (ObjNum 773 snap 0 seq_num 773) 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1355: done (15 left) 2024-09-21T18:08:20.536 INFO:tasks.rados.rados.0.smithi160.stdout:1365: read oid 364 snap -1 2024-09-21T18:08:20.537 INFO:tasks.rados.rados.0.smithi160.stdout:1365: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:20.537 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.538 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:20.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:20.545 INFO:tasks.rados.rados.0.smithi160.stdout:1365: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:20.556 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:20.556 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:20.556 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:20.556 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:20.558 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:20.559 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:20.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:20.579 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:20.579 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:20.585 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:20.577+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:20.602 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:20.602 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:20.621 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:20.621 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:20.640 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:20.640 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:20.663 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:20.663 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:20.675 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1442382 2024-09-21T18:08:20.682 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:20.685 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:20.681+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:20.701 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:20.709 INFO:tasks.rados.rados.0.smithi160.stdout:1356: finishing write tid 1 to smithi16030472-281 2024-09-21T18:08:20.709 INFO:tasks.rados.rados.0.smithi160.stdout:1356: finishing write tid 2 to smithi16030472-281 2024-09-21T18:08:20.709 INFO:tasks.rados.rados.0.smithi160.stdout:1360: got (16) Device or resource busy 2024-09-21T18:08:20.709 INFO:tasks.rados.rados.0.smithi160.stdout:1356: finishing write tid 3 to smithi16030472-281 2024-09-21T18:08:20.710 INFO:tasks.rados.rados.0.smithi160.stdout:1360: done (15 left) 2024-09-21T18:08:20.710 INFO:tasks.rados.rados.0.smithi160.stdout:1365: done (14 left) 2024-09-21T18:08:20.710 INFO:tasks.rados.rados.0.smithi160.stdout:1366: write oid 207 current snap is 0 2024-09-21T18:08:20.710 INFO:tasks.rados.rados.0.smithi160.stdout:1366: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-21T18:08:20.715 INFO:tasks.rados.rados.0.smithi160.stdout:1366: writing smithi16030472-207 from 675846 to 1087574 tid 1 2024-09-21T18:08:20.722 INFO:tasks.rados.rados.0.smithi160.stdout:1366: writing smithi16030472-207 from 1543579 to 2331171 tid 2 2024-09-21T18:08:20.723 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:20.726 INFO:tasks.rados.rados.0.smithi160.stdout:1366: writing smithi16030472-207 from 2848853 to 3075846 tid 3 2024-09-21T18:08:20.726 INFO:tasks.rados.rados.0.smithi160.stdout:1367: write oid 162 current snap is 0 2024-09-21T18:08:20.726 INFO:tasks.rados.rados.0.smithi160.stdout:1367: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-21T18:08:20.733 INFO:tasks.rados.rados.0.smithi160.stdout:1367: writing smithi16030472-162 from 692653 to 1179630 tid 1 2024-09-21T18:08:20.739 INFO:tasks.rados.rados.0.smithi160.stdout:1367: writing smithi16030472-162 from 1602061 to 2249664 tid 2 2024-09-21T18:08:20.743 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:20.743 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:1367: writing smithi16030472-162 from 2992629 to 3092653 tid 3 2024-09-21T18:08:20.744 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.758 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:20.794 INFO:tasks.rados.rados.0.smithi160.stdout:1356: finishing write tid 4 to smithi16030472-281 2024-09-21T18:08:20.794 INFO:tasks.rados.rados.0.smithi160.stdout:1356: oid 281 updating version 0 to 588 2024-09-21T18:08:20.794 INFO:tasks.rados.rados.0.smithi160.stdout:1356: oid 281 version 588 is already newer than 586 2024-09-21T18:08:20.794 INFO:tasks.rados.rados.0.smithi160.stdout:1356: oid 281 version 588 is already newer than 587 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 281 v 588 (ObjNum 774 snap 0 seq_num 774) dirty exists 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1356: left oid 281 (ObjNum 774 snap 0 seq_num 774) 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1356: done (15 left) 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1368: copy_from oid 146 from oid 79 current snap is 0 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1364: done (15 left) 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1369: read oid 318 snap -1 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout:1369: expect (ObjNum 431 snap 0 seq_num 431) 2024-09-21T18:08:20.795 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.799 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:20.797+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:20.803 INFO:tasks.rados.rados.0.smithi160.stdout:1367: finishing write tid 1 to smithi16030472-162 2024-09-21T18:08:20.805 INFO:tasks.rados.rados.0.smithi160.stdout:1367: finishing write tid 2 to smithi16030472-162 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1343: finishing copy_from racing read to smithi16030472-286 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1347: got (0) Success 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 0 (ObjNum 480 snap 0 seq_num 480) dirty exists 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1347: done (15 left) 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1370: cache_flush oid 452 snap -1 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1343: finishing copy_from to smithi16030472-286 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 286 v 454 (ObjNum 595 snap 0 seq_num 595) dirty exists 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1359: got (0) Success 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1357: finishing write tid 1 to smithi16030472-36 2024-09-21T18:08:20.811 INFO:tasks.rados.rados.0.smithi160.stdout:1343: done (15 left) 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout:1353: done (14 left) 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout:1359: done (13 left) 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout:1371: cache_flush oid 15 snap -1 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout:1372: cache_try_flush oid 166 snap -1 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout:1373: cache_try_flush oid 227 snap -1 2024-09-21T18:08:20.812 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.824 INFO:tasks.rados.rados.0.smithi160.stdout:1357: finishing write tid 2 to smithi16030472-36 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: finishing write tid 3 to smithi16030472-36 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: finishing write tid 4 to smithi16030472-36 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: oid 36 updating version 0 to 535 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: oid 36 version 535 is already newer than 534 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: oid 36 updating version 535 to 536 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 536 (ObjNum 775 snap 0 seq_num 775) dirty exists 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: left oid 36 (ObjNum 775 snap 0 seq_num 775) 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1357: done (15 left) 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout:1374: cache_try_flush oid 295 snap -1 2024-09-21T18:08:20.825 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.830 INFO:tasks.rados.rados.0.smithi160.stdout:1361: finishing copy_from racing read to smithi16030472-189 2024-09-21T18:08:20.839 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:20.858 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:20.858 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:20.882 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:20.883 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:20.886 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:20.879+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:20.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:08:20.903 INFO:tasks.rados.rados.0.smithi160.stdout:1358: got (0) Success 2024-09-21T18:08:20.904 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 0 (ObjNum 662 snap 0 seq_num 662) dirty exists 2024-09-21T18:08:20.904 INFO:tasks.rados.rados.0.smithi160.stdout:1358: done (15 left) 2024-09-21T18:08:20.904 INFO:tasks.rados.rados.0.smithi160.stdout:1375: cache_flush oid 277 snap -1 2024-09-21T18:08:20.904 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:20.906 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:20.917 INFO:tasks.rados.rados.0.smithi160.stdout:1352: finishing write tid 1 to smithi16030472-12 2024-09-21T18:08:20.917 INFO:tasks.rados.rados.0.smithi160.stdout:1352: finishing write tid 2 to smithi16030472-12 2024-09-21T18:08:20.917 INFO:tasks.rados.rados.0.smithi160.stdout:1352: finishing write tid 3 to smithi16030472-12 2024-09-21T18:08:20.918 INFO:tasks.rados.rados.0.smithi160.stdout:1369: expect (ObjNum 431 snap 0 seq_num 431) 2024-09-21T18:08:20.927 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:20.928 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:20.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:20.961 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:20.961 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:20.965 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:20.982 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:21.011 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:21.007+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:21.033 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:21.070 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.070 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1369: done (15 left) 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1376: delete oid 406 current snap is 0 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1367: finishing write tid 3 to smithi16030472-162 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1351: finishing copy_from to smithi16030472-182 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 182 v 413 (ObjNum 676 snap 0 seq_num 676) dirty exists 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1348: finishing copy_from racing read to smithi16030472-91 2024-09-21T18:08:21.074 INFO:tasks.rados.rados.0.smithi160.stdout:1363: finishing write tid 1 to smithi16030472-448 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: finishing write tid 4 to smithi16030472-162 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: finishing write tid 5 to smithi16030472-162 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: oid 162 updating version 0 to 590 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: oid 162 updating version 590 to 591 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: oid 162 updating version 591 to 592 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: oid 162 version 592 is already newer than 589 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 162 v 592 (ObjNum 779 snap 0 seq_num 779) dirty exists 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1367: left oid 162 (ObjNum 779 snap 0 seq_num 779) 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1368: finishing copy_from racing read to smithi16030472-146 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1374: got (0) Success 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 685 snap 0 seq_num 685) dirty exists 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1352: finishing write tid 4 to smithi16030472-12 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1352: finishing write tid 5 to smithi16030472-12 2024-09-21T18:08:21.075 INFO:tasks.rados.rados.0.smithi160.stdout:1352: oid 12 updating version 0 to 538 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1352: oid 12 version 538 is already newer than 537 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1352: oid 12 version 538 is already newer than 536 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1352: oid 12 updating version 538 to 539 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 12 v 539 (ObjNum 772 snap 0 seq_num 772) dirty exists 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1352: left oid 12 (ObjNum 772 snap 0 seq_num 772) 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1361: finishing copy_from to smithi16030472-189 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 189 v 465 (ObjNum 624 snap 0 seq_num 624) dirty exists 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1375: got (0) Success 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 277 v 0 (ObjNum 276 snap 0 seq_num 276) dirty exists 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1362: finishing write tid 1 to smithi16030472-225 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1362: finishing write tid 2 to smithi16030472-225 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1363: finishing write tid 2 to smithi16030472-448 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1351: finishing copy_from racing read to smithi16030472-182 2024-09-21T18:08:21.076 INFO:tasks.rados.rados.0.smithi160.stdout:1362: finishing write tid 3 to smithi16030472-225 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:1362: finishing write tid 4 to smithi16030472-225 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:1362: oid 225 updating version 0 to 490 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:1362: oid 225 version 490 is already newer than 489 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:1362: oid 225 version 490 is already newer than 488 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 490 (ObjNum 776 snap 0 seq_num 776) dirty exists 2024-09-21T18:08:21.077 INFO:tasks.rados.rados.0.smithi160.stdout:1362: left oid 225 (ObjNum 776 snap 0 seq_num 776) 2024-09-21T18:08:21.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:21.088 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:21.085+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:21.114 INFO:tasks.rados.rados.0.smithi160.stdout:1368: finishing copy_from to smithi16030472-146 2024-09-21T18:08:21.114 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 521 (ObjNum 725 snap 0 seq_num 725) dirty exists 2024-09-21T18:08:21.121 INFO:tasks.rados.rados.0.smithi160.stdout:1372: got (0) Success 2024-09-21T18:08:21.121 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 0 (ObjNum 730 snap 0 seq_num 730) dirty exists 2024-09-21T18:08:21.140 INFO:tasks.rados.rados.0.smithi160.stdout:1371: got (0) Success 2024-09-21T18:08:21.140 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 0 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-21T18:08:21.150 INFO:tasks.rados.rados.0.smithi160.stdout:1351: done (15 left) 2024-09-21T18:08:21.150 INFO:tasks.rados.rados.0.smithi160.stdout:1352: done (14 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1361: done (13 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1362: done (12 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1367: done (11 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1368: done (10 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1371: done (9 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1372: done (8 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1374: done (7 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1375: done (6 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1376: done (5 left) 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1377: copy_from oid 149 from oid 136 current snap is 0 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1378: cache_try_flush oid 407 snap -1 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1379: cache_flush oid 129 snap -1 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1380: cache_try_flush oid 341 snap -1 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1381: cache_evict oid 165 snap -1 2024-09-21T18:08:21.151 INFO:tasks.rados.rados.0.smithi160.stdout:1382: read oid 158 snap -1 2024-09-21T18:08:21.152 INFO:tasks.rados.rados.0.smithi160.stdout:1382: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-21T18:08:21.152 INFO:tasks.rados.rados.0.smithi160.stdout:1383: write oid 234 current snap is 0 2024-09-21T18:08:21.152 INFO:tasks.rados.rados.0.smithi160.stdout:1383: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-21T18:08:21.168 INFO:tasks.rados.rados.0.smithi160.stdout:1383: writing smithi16030472-234 from 726267 to 1363742 tid 1 2024-09-21T18:08:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:1383: writing smithi16030472-234 from 2119025 to 2770297 tid 2 2024-09-21T18:08:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:1384: write oid 432 current snap is 0 2024-09-21T18:08:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:1378: got (0) Success 2024-09-21T18:08:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 0 (ObjNum 129 snap 0 seq_num 3138372680) clean dne 2024-09-21T18:08:21.187 INFO:tasks.rados.rados.0.smithi160.stdout:1384: seq_num 782 ranges {743074=712724,2061154=794930} 2024-09-21T18:08:21.188 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:21.185+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:21.206 INFO:tasks.rados.rados.0.smithi160.stdout:1384: writing smithi16030472-432 from 743074 to 1455798 tid 1 2024-09-21T18:08:21.211 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.212 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.222 INFO:tasks.rados.rados.0.smithi160.stdout:1384: writing smithi16030472-432 from 2061154 to 2856084 tid 2 2024-09-21T18:08:21.222 INFO:tasks.rados.rados.0.smithi160.stdout:1378: done (12 left) 2024-09-21T18:08:21.222 INFO:tasks.rados.rados.0.smithi160.stdout:1385: write oid 496 current snap is 0 2024-09-21T18:08:21.223 INFO:tasks.rados.rados.0.smithi160.stdout:1385: seq_num 783 ranges {759881=787973,2119636=654941} 2024-09-21T18:08:21.226 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:21.236 INFO:tasks.rados.rados.0.smithi160.stdout:1385: writing smithi16030472-496 from 759881 to 1547854 tid 1 2024-09-21T18:08:21.246 INFO:tasks.rados.rados.0.smithi160.stdout:1385: writing smithi16030472-496 from 2119636 to 2774577 tid 2 2024-09-21T18:08:21.246 INFO:tasks.rados.rados.0.smithi160.stdout:1386: write (excl) oid 193 current snap is 0 2024-09-21T18:08:21.246 INFO:tasks.rados.rados.0.smithi160.stdout:1386: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-21T18:08:21.256 INFO:tasks.rados.rados.0.smithi160.stdout:1386: writing smithi16030472-193 from 776688 to 1356263 tid 1 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1386: writing smithi16030472-193 from 1894471 to 2693070 tid 2 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1387: read oid 73 snap -1 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1387: expect (ObjNum 657 snap 0 seq_num 657) 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1388: cache_flush oid 372 snap -1 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1348: finishing copy_from to smithi16030472-91 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 317 (ObjNum 443 snap 0 seq_num 443) dirty exists 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1366: finishing write tid 1 to smithi16030472-207 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1366: finishing write tid 2 to smithi16030472-207 2024-09-21T18:08:21.265 INFO:tasks.rados.rados.0.smithi160.stdout:1366: finishing write tid 3 to smithi16030472-207 2024-09-21T18:08:21.266 INFO:tasks.rados.rados.0.smithi160.stdout:1366: finishing write tid 4 to smithi16030472-207 2024-09-21T18:08:21.266 INFO:tasks.rados.rados.0.smithi160.stdout:1348: done (15 left) 2024-09-21T18:08:21.266 INFO:tasks.rados.rados.0.smithi160.stdout:1389: copy_from oid 198 from oid 133 current snap is 0 2024-09-21T18:08:21.266 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:21.311 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:21.309+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:21.390 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:21.383+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:21.392 INFO:tasks.rados.rados.0.smithi160.stdout:1383: finishing write tid 1 to smithi16030472-234 2024-09-21T18:08:21.392 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.392 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: finishing write tid 5 to smithi16030472-207 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: oid 207 updating version 0 to 493 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: oid 207 version 493 is already newer than 492 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: oid 207 version 493 is already newer than 491 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: oid 207 updating version 493 to 494 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 207 v 494 (ObjNum 778 snap 0 seq_num 778) dirty exists 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1366: left oid 207 (ObjNum 778 snap 0 seq_num 778) 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1363: finishing write tid 3 to smithi16030472-448 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1363: finishing write tid 4 to smithi16030472-448 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1363: oid 448 updating version 0 to 486 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1363: oid 448 updating version 486 to 487 2024-09-21T18:08:21.448 INFO:tasks.rados.rados.0.smithi160.stdout:1363: oid 448 updating version 487 to 495 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 495 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1363: left oid 448 (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: finishing write tid 1 to smithi16030472-432 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: finishing write tid 2 to smithi16030472-432 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: finishing write tid 3 to smithi16030472-432 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1363: done (15 left) 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1366: done (14 left) 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1390: write oid 79 current snap is 0 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: finishing write tid 4 to smithi16030472-432 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: oid 432 updating version 0 to 489 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: oid 432 updating version 489 to 490 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:1384: oid 432 version 490 is already newer than 488 2024-09-21T18:08:21.449 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 432 v 490 (ObjNum 782 snap 0 seq_num 782) dirty exists 2024-09-21T18:08:21.450 INFO:tasks.rados.rados.0.smithi160.stdout:1384: left oid 432 (ObjNum 782 snap 0 seq_num 782) 2024-09-21T18:08:21.450 INFO:tasks.rados.rados.0.smithi160.stdout:1390: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-21T18:08:21.460 INFO:tasks.rados.rados.0.smithi160.stdout:1390: writing smithi16030472-79 from 793495 to 1448319 tid 1 2024-09-21T18:08:21.472 INFO:tasks.rados.rados.0.smithi160.stdout:1390: writing smithi16030472-79 from 1952953 to 2611563 tid 2 2024-09-21T18:08:21.478 INFO:tasks.rados.rados.0.smithi160.stdout:1390: writing smithi16030472-79 from 3157167 to 3193495 tid 3 2024-09-21T18:08:21.478 INFO:tasks.rados.rados.0.smithi160.stdout:1384: done (14 left) 2024-09-21T18:08:21.478 INFO:tasks.rados.rados.0.smithi160.stdout:1391: cache_try_flush oid 393 snap -1 2024-09-21T18:08:21.478 INFO:tasks.rados.rados.0.smithi160.stdout:1382: expect (ObjNum 157 snap 0 seq_num 157) 2024-09-21T18:08:21.490 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:21.487+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:21.528 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:21.541 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.542 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.556 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:21.576 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:21.591 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:21.585+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:21.607 INFO:tasks.rados.rados.0.smithi160.stdout:1386: finishing write tid 1 to smithi16030472-193 2024-09-21T18:08:21.608 INFO:tasks.rados.rados.0.smithi160.stdout:1387: expect (ObjNum 657 snap 0 seq_num 657) 2024-09-21T18:08:21.679 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.679 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.688 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:21.692 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:21.689+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:21.693 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:21.736 INFO:tasks.rados.rados.0.smithi160.stdout:1388: got (0) Success 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 0 (ObjNum 474 snap 0 seq_num 474) dirty exists 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: finishing write tid 2 to smithi16030472-234 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1370: got (0) Success 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 452 v 0 (ObjNum 451 snap 0 seq_num 451) dirty exists 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: finishing write tid 3 to smithi16030472-234 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: finishing write tid 4 to smithi16030472-234 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: oid 234 updating version 0 to 564 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: oid 234 version 564 is already newer than 563 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: oid 234 version 564 is already newer than 562 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 234 v 564 (ObjNum 781 snap 0 seq_num 781) dirty exists 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1383: left oid 234 (ObjNum 781 snap 0 seq_num 781) 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1389: finishing copy_from racing read to smithi16030472-198 2024-09-21T18:08:21.737 INFO:tasks.rados.rados.0.smithi160.stdout:1373: got (0) Success 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 227 v 0 (ObjNum 626 snap 0 seq_num 626) dirty exists 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1370: done (14 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1373: done (13 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1382: done (12 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1383: done (11 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1387: done (10 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1388: done (9 left) 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1392: read oid 497 snap -1 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1392: expect deleted 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1393: cache_try_flush oid 241 snap -1 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1394: write (excl) oid 81 current snap is 0 2024-09-21T18:08:21.738 INFO:tasks.rados.rados.0.smithi160.stdout:1394: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-21T18:08:21.750 INFO:tasks.rados.rados.0.smithi160.stdout:1394: writing smithi16030472-81 from 410302 to 1140375 tid 1 2024-09-21T18:08:21.764 INFO:tasks.rados.rados.0.smithi160.stdout:1394: writing smithi16030472-81 from 1895082 to 2297350 tid 2 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1394: writing smithi16030472-81 from 2951884 to 3210302 tid 3 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1395: copy_from oid 479 from oid 343 current snap is 0 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1391: got (0) Success 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 0 (ObjNum 392 snap 0 seq_num 392) dirty exists 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1389: finishing copy_from to smithi16030472-198 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 198 v 82 (ObjNum 132 snap 0 seq_num 132) dirty exists 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1377: finishing copy_from to smithi16030472-149 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 425 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1389: done (12 left) 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1391: done (11 left) 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1392: done (10 left) 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1396: copy_from oid 231 from oid 448 current snap is 0 2024-09-21T18:08:21.776 INFO:tasks.rados.rados.0.smithi160.stdout:1397: delete oid 130 current snap is 0 2024-09-21T18:08:21.781 INFO:tasks.rados.rados.0.smithi160.stdout:1381: got (0) Success 2024-09-21T18:08:21.793 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:21.787+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:21.811 INFO:tasks.rados.rados.0.smithi160.stdout:1380: got (0) Success 2024-09-21T18:08:21.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 0 (ObjNum 705 snap 0 seq_num 705) dirty exists 2024-09-21T18:08:21.815 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.815 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:21.882 INFO:tasks.rados.rados.0.smithi160.stdout:1385: finishing write tid 1 to smithi16030472-496 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: finishing write tid 2 to smithi16030472-496 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: finishing write tid 3 to smithi16030472-496 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: finishing write tid 4 to smithi16030472-496 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: oid 496 updating version 0 to 541 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: oid 496 updating version 541 to 542 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: oid 496 version 542 is already newer than 540 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 542 (ObjNum 783 snap 0 seq_num 783) dirty exists 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1385: left oid 496 (ObjNum 783 snap 0 seq_num 783) 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1390: finishing write tid 1 to smithi16030472-79 2024-09-21T18:08:21.883 INFO:tasks.rados.rados.0.smithi160.stdout:1390: finishing write tid 2 to smithi16030472-79 2024-09-21T18:08:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:08:21.886 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:21.877+0000 7f94db546640 -1 osd.15 75 handle_osd_map injecting map crc failure 2024-09-21T18:08:21.886 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:21.881+0000 7f3ce4680640 -1 osd.6 75 handle_osd_map injecting map crc failure 2024-09-21T18:08:21.893 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:21.889+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:21.912 INFO:tasks.rados.rados.0.smithi160.stdout:1379: got (0) Success 2024-09-21T18:08:21.912 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:08:21.954 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:21.955 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:21.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:21.983 INFO:tasks.rados.rados.0.smithi160.stdout:1393: got (0) Success 2024-09-21T18:08:21.983 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 0 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-21T18:08:21.983 INFO:tasks.rados.rados.0.smithi160.stdout:1386: finishing write tid 2 to smithi16030472-193 2024-09-21T18:08:22.008 INFO:tasks.rados.rados.0.smithi160.stdout:1396: finishing copy_from racing read to smithi16030472-231 2024-09-21T18:08:22.012 INFO:tasks.rados.rados.0.smithi160.stdout:1377: finishing copy_from racing read to smithi16030472-149 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1377: done (11 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1379: done (10 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1380: done (9 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1381: done (8 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1385: done (7 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1393: done (6 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1397: done (5 left) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1398: read oid 366 snap -1 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1398: expect (ObjNum 551 snap 0 seq_num 551) 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1399: write (excl) oid 407 current snap is 0 2024-09-21T18:08:22.013 INFO:tasks.rados.rados.0.smithi160.stdout:1399: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-21T18:08:22.014 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:22.013+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:22.026 INFO:tasks.rados.rados.0.smithi160.stdout:1399: writing smithi16030472-407 from 443916 to 924487 tid 1 2024-09-21T18:08:22.042 INFO:tasks.rados.rados.0.smithi160.stdout:1399: writing smithi16030472-407 from 1612046 to 2017983 tid 2 2024-09-21T18:08:22.057 INFO:tasks.rados.rados.0.smithi160.stdout:1399: writing smithi16030472-407 from 2606730 to 3101782 tid 3 2024-09-21T18:08:22.057 INFO:tasks.rados.rados.0.smithi160.stdout:1400: write (excl) oid 149 current snap is 0 2024-09-21T18:08:22.058 INFO:tasks.rados.rados.0.smithi160.stdout:1395: finishing copy_from to smithi16030472-479 2024-09-21T18:08:22.058 INFO:tasks.rados.rados.0.smithi160.stdout:1395: got expected ENOENT (src dne) 2024-09-21T18:08:22.058 INFO:tasks.rados.rados.0.smithi160.stdout:1395: finishing copy_from racing read to smithi16030472-479 2024-09-21T18:08:22.058 INFO:tasks.rados.rados.0.smithi160.stdout:1398: expect (ObjNum 551 snap 0 seq_num 551) 2024-09-21T18:08:22.092 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.092 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.096 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:22.091+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:22.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:22.197 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:22.197+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:22.211 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2273995 2024-09-21T18:08:22.224 INFO:tasks.rados.rados.0.smithi160.stdout:1399: finishing write tid 1 to smithi16030472-407 2024-09-21T18:08:22.224 INFO:tasks.rados.rados.0.smithi160.stdout:1400: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-09-21T18:08:22.230 INFO:tasks.rados.rados.0.smithi160.stdout:1400: writing smithi16030472-149 from 460723 to 1016543 tid 1 2024-09-21T18:08:22.236 INFO:tasks.rados.rados.0.smithi160.stdout:1400: writing smithi16030472-149 from 1670528 to 2220123 tid 2 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1400: writing smithi16030472-149 from 2634153 to 3260723 tid 3 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1395: done (7 left) 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1398: done (6 left) 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1401: read oid 109 snap -1 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1401: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1402: cache_flush oid 257 snap -1 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1399: finishing write tid 2 to smithi16030472-407 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1399: finishing write tid 3 to smithi16030472-407 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1386: finishing write tid 3 to smithi16030472-193 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1386: finishing write tid 4 to smithi16030472-193 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1386: oid 193 updating version 0 to 545 2024-09-21T18:08:22.251 INFO:tasks.rados.rados.0.smithi160.stdout:1386: oid 193 updating version 545 to 546 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1386: oid 193 version 546 is already newer than 538 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 193 v 546 (ObjNum 784 snap 0 seq_num 784) dirty exists 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1386: left oid 193 (ObjNum 784 snap 0 seq_num 784) 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: finishing write tid 3 to smithi16030472-79 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1386: done (7 left) 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1403: cache_evict oid 290 snap -1 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: finishing write tid 4 to smithi16030472-79 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: finishing write tid 5 to smithi16030472-79 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: oid 79 updating version 0 to 547 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: oid 79 updating version 547 to 548 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: oid 79 version 548 is already newer than 543 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: oid 79 version 548 is already newer than 544 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 548 (ObjNum 785 snap 0 seq_num 785) dirty exists 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1390: left oid 79 (ObjNum 785 snap 0 seq_num 785) 2024-09-21T18:08:22.252 INFO:tasks.rados.rados.0.smithi160.stdout:1394: finishing write tid 1 to smithi16030472-81 2024-09-21T18:08:22.253 INFO:tasks.rados.rados.0.smithi160.stdout:1394: finishing write tid 2 to smithi16030472-81 2024-09-21T18:08:22.253 INFO:tasks.rados.rados.0.smithi160.stdout:1394: finishing write tid 3 to smithi16030472-81 2024-09-21T18:08:22.253 INFO:tasks.rados.rados.0.smithi160.stdout:1390: done (7 left) 2024-09-21T18:08:22.253 INFO:tasks.rados.rados.0.smithi160.stdout:1404: write (excl) oid 166 current snap is 0 2024-09-21T18:08:22.253 INFO:tasks.rados.rados.0.smithi160.stdout:1404: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-21T18:08:22.256 INFO:tasks.rados.rados.0.smithi160.stdout:1404: writing smithi16030472-166 from 477530 to 1108599 tid 1 2024-09-21T18:08:22.261 INFO:tasks.rados.rados.0.smithi160.stdout:1404: writing smithi16030472-166 from 1612657 to 2022263 tid 2 2024-09-21T18:08:22.267 INFO:tasks.rados.rados.0.smithi160.stdout:1404: writing smithi16030472-166 from 2545223 to 3062819 tid 3 2024-09-21T18:08:22.268 INFO:tasks.rados.rados.0.smithi160.stdout:1405: delete oid 484 current snap is 0 2024-09-21T18:08:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.288 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:22.331 INFO:tasks.rados.rados.0.smithi160.stdout:1399: finishing write tid 4 to smithi16030472-407 2024-09-21T18:08:22.331 INFO:tasks.rados.rados.0.smithi160.stdout:1399: finishing write tid 5 to smithi16030472-407 2024-09-21T18:08:22.331 INFO:tasks.rados.rados.0.smithi160.stdout:1399: oid 407 updating version 0 to 565 2024-09-21T18:08:22.332 INFO:tasks.rados.rados.0.smithi160.stdout:1399: oid 407 updating version 565 to 567 2024-09-21T18:08:22.332 INFO:tasks.rados.rados.0.smithi160.stdout:1399: oid 407 updating version 567 to 568 2024-09-21T18:08:22.332 INFO:tasks.rados.rados.0.smithi160.stdout:1399: oid 407 version 568 is already newer than 566 2024-09-21T18:08:22.332 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 568 (ObjNum 788 snap 0 seq_num 788) dirty exists 2024-09-21T18:08:22.332 INFO:tasks.rados.rados.0.smithi160.stdout:1399: left oid 407 (ObjNum 788 snap 0 seq_num 788) 2024-09-21T18:08:22.333 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:22.333+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: finishing write tid 4 to smithi16030472-81 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: finishing write tid 5 to smithi16030472-81 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: oid 81 updating version 0 to 498 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: oid 81 version 498 is already newer than 497 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: oid 81 version 498 is already newer than 496 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: oid 81 updating version 498 to 499 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 81 v 499 (ObjNum 786 snap 0 seq_num 786) dirty exists 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1394: left oid 81 (ObjNum 786 snap 0 seq_num 786) 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1396: finishing copy_from to smithi16030472-231 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 231 v 495 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:22.355 INFO:tasks.rados.rados.0.smithi160.stdout:1403: got (2) No such file or directory 2024-09-21T18:08:22.356 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:22.353+0000 7f4976023640 -1 osd.5 75 handle_osd_map injecting map crc failure 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1394: done (8 left) 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1396: done (7 left) 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1399: done (6 left) 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1403: done (5 left) 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1405: done (4 left) 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1406: cache_evict oid 35 snap -1 2024-09-21T18:08:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:1407: copy_from oid 452 from oid 159 current snap is 0 2024-09-21T18:08:22.368 INFO:tasks.rados.rados.0.smithi160.stdout:1408: read oid 365 snap -1 2024-09-21T18:08:22.368 INFO:tasks.rados.rados.0.smithi160.stdout:1408: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-21T18:08:22.368 INFO:tasks.rados.rados.0.smithi160.stdout:1409: write (excl) oid 305 current snap is 0 2024-09-21T18:08:22.368 INFO:tasks.rados.rados.0.smithi160.stdout:1409: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-21T18:08:22.377 INFO:tasks.rados.rados.0.smithi160.stdout:1409: writing smithi16030472-305 from 511144 to 1009064 tid 1 2024-09-21T18:08:22.384 INFO:tasks.rados.rados.0.smithi160.stdout:1409: writing smithi16030472-305 from 1445974 to 1859249 tid 2 2024-09-21T18:08:22.394 INFO:tasks.rados.rados.0.smithi160.stdout:1409: writing smithi16030472-305 from 2316422 to 2972915 tid 3 2024-09-21T18:08:22.394 INFO:tasks.rados.rados.0.smithi160.stdout:1410: read oid 325 snap -1 2024-09-21T18:08:22.394 INFO:tasks.rados.rados.0.smithi160.stdout:1410: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-21T18:08:22.394 INFO:tasks.rados.rados.0.smithi160.stdout:1411: copy_from oid 312 from oid 162 current snap is 0 2024-09-21T18:08:22.394 INFO:tasks.rados.rados.0.smithi160.stdout:1408: expect (ObjNum 364 snap 0 seq_num 364) 2024-09-21T18:08:22.399 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:22.395+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:22.454 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:22.457 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.457 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.472 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:22.495 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 648852 2024-09-21T18:08:22.500 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:22.493+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:22.513 INFO:tasks.rados.rados.0.smithi160.stdout:1401: expect (ObjNum 108 snap 0 seq_num 108) 2024-09-21T18:08:22.530 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3128313 2024-09-21T18:08:22.537 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:22.596 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.596 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.601 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:22.597+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:22.612 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:22.612 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:22.609+0000 7f0b6a212640 -1 osd.9 75 handle_osd_map injecting map crc failure 2024-09-21T18:08:22.614 INFO:tasks.rados.rados.0.smithi160.stdout:1402: got (2) No such file or directory 2024-09-21T18:08:22.614 INFO:tasks.rados.rados.0.smithi160.stdout:1406: got (16) Device or resource busy 2024-09-21T18:08:22.614 INFO:tasks.rados.rados.0.smithi160.stdout:1407: finishing copy_from to smithi16030472-452 2024-09-21T18:08:22.614 INFO:tasks.rados.rados.0.smithi160.stdout:1407: got expected ENOENT (src dne) 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1407: finishing copy_from racing read to smithi16030472-452 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: finishing write tid 1 to smithi16030472-149 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: finishing write tid 2 to smithi16030472-149 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: finishing write tid 3 to smithi16030472-149 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: finishing write tid 4 to smithi16030472-149 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: finishing write tid 5 to smithi16030472-149 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: oid 149 updating version 0 to 595 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: oid 149 updating version 595 to 596 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: oid 149 version 596 is already newer than 594 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: oid 149 version 596 is already newer than 593 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 596 (ObjNum 789 snap 0 seq_num 789) dirty exists 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: left oid 149 (ObjNum 789 snap 0 seq_num 789) 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1404: finishing write tid 1 to smithi16030472-166 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1404: finishing write tid 2 to smithi16030472-166 2024-09-21T18:08:22.615 INFO:tasks.rados.rados.0.smithi160.stdout:1400: done (9 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1401: done (8 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1402: done (7 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1406: done (6 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1407: done (5 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1408: done (4 left) 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1412: delete oid 93 current snap is 0 2024-09-21T18:08:22.616 INFO:tasks.rados.rados.0.smithi160.stdout:1404: finishing write tid 3 to smithi16030472-166 2024-09-21T18:08:22.622 INFO:tasks.rados.rados.0.smithi160.stdout:1410: expect (ObjNum 324 snap 0 seq_num 324) 2024-09-21T18:08:22.656 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3006223 2024-09-21T18:08:22.675 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2383389 2024-09-21T18:08:22.702 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:22.695+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:22.733 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.734 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.748 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:22.764 INFO:tasks.rados.rados.0.smithi160.stdout:1409: finishing write tid 1 to smithi16030472-305 2024-09-21T18:08:22.766 INFO:tasks.rados.rados.0.smithi160.stdout:1409: finishing write tid 2 to smithi16030472-305 2024-09-21T18:08:22.799 INFO:tasks.rados.rados.0.smithi160.stdout:1404: finishing write tid 4 to smithi16030472-166 2024-09-21T18:08:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:1404: finishing write tid 5 to smithi16030472-166 2024-09-21T18:08:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:1404: oid 166 updating version 0 to 600 2024-09-21T18:08:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:1404: oid 166 version 600 is already newer than 598 2024-09-21T18:08:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:1404: oid 166 version 600 is already newer than 599 2024-09-21T18:08:22.800 INFO:tasks.rados.rados.0.smithi160.stdout:1404: oid 166 version 600 is already newer than 597 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 600 (ObjNum 790 snap 0 seq_num 790) dirty exists 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1404: left oid 166 (ObjNum 790 snap 0 seq_num 790) 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1404: done (4 left) 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1410: done (3 left) 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1412: done (2 left) 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1413: read oid 256 snap -1 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1413: expect (ObjNum 255 snap 0 seq_num 255) 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1414: cache_evict oid 397 snap -1 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1415: write (excl) oid 201 current snap is 0 2024-09-21T18:08:22.801 INFO:tasks.rados.rados.0.smithi160.stdout:1415: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-21T18:08:22.803 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:22.797+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:22.819 INFO:tasks.rados.rados.0.smithi160.stdout:1415: writing smithi16030472-201 from 544758 to 1193176 tid 1 2024-09-21T18:08:22.837 INFO:tasks.rados.rados.0.smithi160.stdout:1415: writing smithi16030472-201 from 1846585 to 2263529 tid 2 2024-09-21T18:08:22.853 INFO:tasks.rados.rados.0.smithi160.stdout:1415: writing smithi16030472-201 from 2771268 to 3344758 tid 3 2024-09-21T18:08:22.853 INFO:tasks.rados.rados.0.smithi160.stdout:1416: write oid 294 current snap is 0 2024-09-21T18:08:22.853 INFO:tasks.rados.rados.0.smithi160.stdout:1416: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-21T18:08:22.866 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:22.866 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:22.867 INFO:tasks.rados.rados.0.smithi160.stdout:1416: writing smithi16030472-294 from 561565 to 1285232 tid 1 2024-09-21T18:08:22.878 INFO:tasks.rados.rados.0.smithi160.stdout:1416: writing smithi16030472-294 from 1905067 to 2465669 tid 2 2024-09-21T18:08:22.880 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1416: writing smithi16030472-294 from 3082338 to 3361565 tid 3 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1417: copy_from oid 184 from oid 414 current snap is 0 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1411: finishing copy_from racing read to smithi16030472-312 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1418: cache_try_flush oid 90 snap -1 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1411: finishing copy_from to smithi16030472-312 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 312 v 592 (ObjNum 779 snap 0 seq_num 779) dirty exists 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1414: got (2) No such file or directory 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1409: finishing write tid 3 to smithi16030472-305 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1411: done (7 left) 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1414: done (6 left) 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1419: write (excl) oid 230 current snap is 0 2024-09-21T18:08:22.887 INFO:tasks.rados.rados.0.smithi160.stdout:1419: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-21T18:08:22.897 INFO:tasks.rados.rados.0.smithi160.stdout:1419: writing smithi16030472-230 from 578372 to 1377288 tid 1 2024-09-21T18:08:22.906 INFO:tasks.rados.rados.0.smithi160.stdout:1419: writing smithi16030472-230 from 1963549 to 2384162 tid 2 2024-09-21T18:08:22.914 INFO:tasks.rados.rados.0.smithi160.stdout:1419: writing smithi16030472-230 from 2826114 to 3244048 tid 3 2024-09-21T18:08:22.914 INFO:tasks.rados.rados.0.smithi160.stdout:1420: cache_try_flush oid 348 snap -1 2024-09-21T18:08:22.914 INFO:tasks.rados.rados.0.smithi160.stdout:1421: delete oid 132 current snap is 0 2024-09-21T18:08:22.925 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:22.925+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:22.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:22.980 INFO:tasks.rados.rados.0.smithi160.stdout:1416: finishing write tid 1 to smithi16030472-294 2024-09-21T18:08:23.004 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:23.005 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:23.011 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:23.007+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:23.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1416: finishing write tid 2 to smithi16030472-294 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: finishing write tid 4 to smithi16030472-305 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1416: finishing write tid 3 to smithi16030472-294 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: finishing write tid 5 to smithi16030472-305 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: oid 305 updating version 0 to 539 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: oid 305 updating version 539 to 541 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: oid 305 version 541 is already newer than 540 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: oid 305 updating version 541 to 542 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 305 v 542 (ObjNum 792 snap 0 seq_num 792) dirty exists 2024-09-21T18:08:23.081 INFO:tasks.rados.rados.0.smithi160.stdout:1409: left oid 305 (ObjNum 792 snap 0 seq_num 792) 2024-09-21T18:08:23.105 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:23.101+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:23.125 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:23.125 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:23.179 INFO:tasks.rados.rados.0.smithi160.stdout:1416: finishing write tid 4 to smithi16030472-294 2024-09-21T18:08:23.179 INFO:tasks.rados.rados.0.smithi160.stdout:1416: finishing write tid 5 to smithi16030472-294 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:1416: oid 294 updating version 0 to 593 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:1416: oid 294 updating version 593 to 594 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:1416: oid 294 updating version 594 to 596 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:1416: oid 294 version 596 is already newer than 595 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 596 (ObjNum 795 snap 0 seq_num 795) dirty exists 2024-09-21T18:08:23.180 INFO:tasks.rados.rados.0.smithi160.stdout:1416: left oid 294 (ObjNum 795 snap 0 seq_num 795) 2024-09-21T18:08:23.181 INFO:tasks.rados.rados.0.smithi160.stdout:1413: expect (ObjNum 255 snap 0 seq_num 255) 2024-09-21T18:08:23.184 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:23.184 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:23.200 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:23.205 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:23.201+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:23.217 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:23.248 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:23.248 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1798613 2024-09-21T18:08:23.273 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:23.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1415: finishing write tid 1 to smithi16030472-201 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1415: finishing write tid 2 to smithi16030472-201 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1419: finishing write tid 1 to smithi16030472-230 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1409: done (8 left) 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1413: done (7 left) 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1416: done (6 left) 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1421: done (5 left) 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1422: copy_from oid 461 from oid 465 current snap is 0 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1423: write oid 209 current snap is 0 2024-09-21T18:08:23.297 INFO:tasks.rados.rados.0.smithi160.stdout:1423: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-09-21T18:08:23.300 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:23.301 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:23.306 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:23.305+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:23.316 INFO:tasks.rados.rados.0.smithi160.stdout:1423: writing smithi16030472-209 from 611986 to 1161400 tid 1 2024-09-21T18:08:23.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:23.327 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:23.327 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:08:23.327 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:23.329 INFO:tasks.rados.rados.0.smithi160.stdout:1423: writing smithi16030472-209 from 1564160 to 1988442 tid 2 2024-09-21T18:08:23.342 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:23.345 INFO:tasks.rados.rados.0.smithi160.stdout:1423: writing smithi16030472-209 from 2764607 to 3205085 tid 3 2024-09-21T18:08:23.345 INFO:tasks.rados.rados.0.smithi160.stdout:1424: write (excl) oid 328 current snap is 0 2024-09-21T18:08:23.345 INFO:tasks.rados.rados.0.smithi160.stdout:1424: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-09-21T18:08:23.355 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:23.356 INFO:tasks.rados.rados.0.smithi160.stdout:1424: writing smithi16030472-328 from 628793 to 1369809 tid 1 2024-09-21T18:08:23.367 INFO:tasks.rados.rados.0.smithi160.stdout:1424: writing smithi16030472-328 from 2138995 to 2706935 tid 2 2024-09-21T18:08:23.372 INFO:tasks.rados.rados.0.smithi160.stdout:1424: writing smithi16030472-328 from 3192030 to 3428793 tid 3 2024-09-21T18:08:23.373 INFO:tasks.rados.rados.0.smithi160.stdout:1425: write (excl) oid 332 current snap is 0 2024-09-21T18:08:23.373 INFO:tasks.rados.rados.0.smithi160.stdout:1425: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-21T18:08:23.377 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:23.377 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:23.381 INFO:tasks.rados.rados.0.smithi160.stdout:1425: writing smithi16030472-332 from 645600 to 1061865 tid 1 2024-09-21T18:08:23.390 INFO:tasks.rados.rados.0.smithi160.stdout:1425: writing smithi16030472-332 from 1797477 to 2509075 tid 2 2024-09-21T18:08:23.396 INFO:tasks.rados.rados.0.smithi160.stdout:1425: writing smithi16030472-332 from 3219453 to 3445600 tid 3 2024-09-21T18:08:23.397 INFO:tasks.rados.rados.0.smithi160.stdout:1426: write oid 164 current snap is 0 2024-09-21T18:08:23.397 INFO:tasks.rados.rados.0.smithi160.stdout:1426: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-09-21T18:08:23.403 INFO:tasks.rados.rados.0.smithi160.stdout:1426: writing smithi16030472-164 from 662407 to 1153921 tid 1 2024-09-21T18:08:23.404 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:23.404 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:23.407 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:23.401+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:23.410 INFO:tasks.rados.rados.0.smithi160.stdout:1426: writing smithi16030472-164 from 1855959 to 2427568 tid 2 2024-09-21T18:08:23.416 INFO:tasks.rados.rados.0.smithi160.stdout:1426: writing smithi16030472-164 from 2846876 to 3462407 tid 3 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1427: read oid 338 snap -1 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1427: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1428: write oid 372 current snap is 0 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1422: finishing copy_from to smithi16030472-461 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1422: got expected ENOENT (src dne) 2024-09-21T18:08:23.417 INFO:tasks.rados.rados.0.smithi160.stdout:1428: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-21T18:08:23.423 INFO:tasks.rados.rados.0.smithi160.stdout:1428: writing smithi16030472-372 from 679214 to 1245977 tid 1 2024-09-21T18:08:23.430 INFO:tasks.rados.rados.0.smithi160.stdout:1428: writing smithi16030472-372 from 1798088 to 2513355 tid 2 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1428: writing smithi16030472-372 from 3157946 to 3479214 tid 3 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1429: read oid 486 snap -1 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1429: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1430: read oid 353 snap -1 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1430: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-21T18:08:23.436 INFO:tasks.rados.rados.0.smithi160.stdout:1431: write (excl) oid 51 current snap is 0 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1422: finishing copy_from racing read to smithi16030472-461 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1420: got (0) Success 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 348 v 0 (ObjNum 506 snap 0 seq_num 506) dirty exists 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1418: got (0) Success 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 90 v 0 (ObjNum 766 snap 0 seq_num 766) dirty exists 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1417: finishing copy_from to smithi16030472-184 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 184 v 298 (ObjNum 413 snap 0 seq_num 413) dirty exists 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1417: finishing copy_from racing read to smithi16030472-184 2024-09-21T18:08:23.437 INFO:tasks.rados.rados.0.smithi160.stdout:1431: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-21T18:08:23.443 INFO:tasks.rados.rados.0.smithi160.stdout:1431: writing smithi16030472-51 from 696021 to 1338033 tid 1 2024-09-21T18:08:23.449 INFO:tasks.rados.rados.0.smithi160.stdout:1431: writing smithi16030472-51 from 1856570 to 2431848 tid 2 2024-09-21T18:08:23.455 INFO:tasks.rados.rados.0.smithi160.stdout:1431: writing smithi16030472-51 from 3185369 to 3496021 tid 3 2024-09-21T18:08:23.456 INFO:tasks.rados.rados.0.smithi160.stdout:1417: done (14 left) 2024-09-21T18:08:23.456 INFO:tasks.rados.rados.0.smithi160.stdout:1418: done (13 left) 2024-09-21T18:08:23.456 INFO:tasks.rados.rados.0.smithi160.stdout:1420: done (12 left) 2024-09-21T18:08:23.456 INFO:tasks.rados.rados.0.smithi160.stdout:1422: done (11 left) 2024-09-21T18:08:23.456 INFO:tasks.rados.rados.0.smithi160.stdout:1432: delete oid 223 current snap is 0 2024-09-21T18:08:23.462 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:23.462 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:23.462 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:23.462 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:23.462 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:23.477 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:23.522 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:23.517+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:23.545 INFO:tasks.rados.rados.0.smithi160.stdout:1430: expect (ObjNum 352 snap 0 seq_num 352) 2024-09-21T18:08:23.577 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:23.577 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:23.588 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:23.608 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:23.605+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:23.630 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:23.648 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:23.648 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:23.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:23.648 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:08:23.648 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:23.662 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:1419: finishing write tid 2 to smithi16030472-230 2024-09-21T18:08:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:1419: finishing write tid 3 to smithi16030472-230 2024-09-21T18:08:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:1419: finishing write tid 4 to smithi16030472-230 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: finishing write tid 5 to smithi16030472-230 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: oid 230 updating version 0 to 603 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: oid 230 version 603 is already newer than 602 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: oid 230 version 603 is already newer than 553 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: oid 230 version 603 is already newer than 601 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 603 (ObjNum 796 snap 0 seq_num 796) dirty exists 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1419: left oid 230 (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: finishing write tid 1 to smithi16030472-209 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: finishing write tid 2 to smithi16030472-209 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: finishing write tid 3 to smithi16030472-209 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: finishing write tid 4 to smithi16030472-209 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: finishing write tid 5 to smithi16030472-209 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: oid 209 updating version 0 to 606 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: oid 209 updating version 606 to 607 2024-09-21T18:08:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:1423: oid 209 version 607 is already newer than 605 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1423: oid 209 version 607 is already newer than 604 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 209 v 607 (ObjNum 798 snap 0 seq_num 798) dirty exists 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1423: left oid 209 (ObjNum 798 snap 0 seq_num 798) 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1425: finishing write tid 1 to smithi16030472-332 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1425: finishing write tid 2 to smithi16030472-332 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: finishing write tid 3 to smithi16030472-201 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: finishing write tid 4 to smithi16030472-201 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: finishing write tid 5 to smithi16030472-201 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: oid 201 updating version 0 to 545 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: oid 201 version 545 is already newer than 544 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: oid 201 version 545 is already newer than 543 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: oid 201 updating version 545 to 546 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 201 v 546 (ObjNum 794 snap 0 seq_num 794) dirty exists 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1415: left oid 201 (ObjNum 794 snap 0 seq_num 794) 2024-09-21T18:08:23.676 INFO:tasks.rados.rados.0.smithi160.stdout:1424: finishing write tid 1 to smithi16030472-328 2024-09-21T18:08:23.684 INFO:tasks.rados.rados.0.smithi160.stdout:1431: finishing write tid 1 to smithi16030472-51 2024-09-21T18:08:23.685 INFO:tasks.rados.rados.0.smithi160.stdout:1431: finishing write tid 2 to smithi16030472-51 2024-09-21T18:08:23.709 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:23.705+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:23.753 INFO:tasks.rados.rados.0.smithi160.stdout:1425: finishing write tid 3 to smithi16030472-332 2024-09-21T18:08:23.754 INFO:tasks.rados.rados.0.smithi160.stdout:1425: finishing write tid 4 to smithi16030472-332 2024-09-21T18:08:23.754 INFO:tasks.rados.rados.0.smithi160.stdout:1425: finishing write tid 5 to smithi16030472-332 2024-09-21T18:08:23.754 INFO:tasks.rados.rados.0.smithi160.stdout:1425: oid 332 updating version 0 to 609 2024-09-21T18:08:23.754 INFO:tasks.rados.rados.0.smithi160.stdout:1425: oid 332 updating version 609 to 610 2024-09-21T18:08:23.755 INFO:tasks.rados.rados.0.smithi160.stdout:1425: oid 332 updating version 610 to 611 2024-09-21T18:08:23.755 INFO:tasks.rados.rados.0.smithi160.stdout:1425: oid 332 version 611 is already newer than 608 2024-09-21T18:08:23.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 611 (ObjNum 800 snap 0 seq_num 800) dirty exists 2024-09-21T18:08:23.755 INFO:tasks.rados.rados.0.smithi160.stdout:1425: left oid 332 (ObjNum 800 snap 0 seq_num 800) 2024-09-21T18:08:23.755 INFO:tasks.rados.rados.0.smithi160.stdout:1428: finishing write tid 1 to smithi16030472-372 2024-09-21T18:08:23.775 INFO:tasks.rados.rados.0.smithi160.stdout:1424: finishing write tid 2 to smithi16030472-328 2024-09-21T18:08:23.779 INFO:tasks.rados.rados.0.smithi160.stdout:1427: expect (ObjNum 337 snap 0 seq_num 337) 2024-09-21T18:08:23.791 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:23.872 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:23.873 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.11.0:955 4.1 e76/33)", 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:23.797565+0000", 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.001091056, 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0011230389999999999, 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797565+0000", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797565+0000", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797567+0000", 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.671e-06 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797572+0000", 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.429e-06 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797585+0000", 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2993000000000001e-05 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797593+0000", 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3559999999999993e-06 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.11.0:957 4.1 e76/33)", 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:23.797891+0000", 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.00076528800000000004, 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00086722000000000004, 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797891+0000", 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797891+0000", 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797892+0000", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0550000000000001e-06 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797896+0000", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8870000000000002e-06 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797907+0000", 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1276e-05 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:23.797914+0000", 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0530000000000001e-06 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 2 2024-09-21T18:08:23.880 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:23.880 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:23.881 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:23.881 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:23.869+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:23.882 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:23.883 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:23.898 INFO:tasks.rados.rados.0.smithi160.stdout:1415: done (11 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1419: done (10 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1423: done (9 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1425: done (8 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1427: done (7 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1430: done (6 left) 2024-09-21T18:08:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:1432: done (5 left) 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1433: write oid 4 current snap is 0 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1426: finishing write tid 1 to smithi16030472-164 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1426: finishing write tid 2 to smithi16030472-164 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1424: finishing write tid 3 to smithi16030472-328 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1428: finishing write tid 2 to smithi16030472-372 2024-09-21T18:08:23.900 INFO:tasks.rados.rados.0.smithi160.stdout:1433: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-21T18:08:23.906 INFO:tasks.rados.rados.0.smithi160.stdout:1433: writing smithi16030472-4 from 729635 to 1522145 tid 1 2024-09-21T18:08:23.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:08:23.912 INFO:tasks.rados.rados.0.smithi160.stdout:1433: writing smithi16030472-4 from 1973534 to 2552481 tid 2 2024-09-21T18:08:23.916 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1433: writing smithi16030472-4 from 3240215 to 3529635 tid 3 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1434: cache_evict oid 337 snap -1 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: finishing write tid 3 to smithi16030472-372 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: finishing write tid 4 to smithi16030472-372 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1435: delete oid 209 current snap is 0 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: finishing write tid 5 to smithi16030472-372 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: oid 372 updating version 0 to 613 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: oid 372 updating version 613 to 614 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: oid 372 version 614 is already newer than 612 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:1428: oid 372 updating version 614 to 615 2024-09-21T18:08:23.918 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 615 (ObjNum 802 snap 0 seq_num 802) dirty exists 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1428: left oid 372 (ObjNum 802 snap 0 seq_num 802) 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: finishing write tid 3 to smithi16030472-164 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: finishing write tid 4 to smithi16030472-164 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: finishing write tid 5 to smithi16030472-164 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: oid 164 updating version 0 to 551 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: oid 164 version 551 is already newer than 549 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: oid 164 version 551 is already newer than 550 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: oid 164 updating version 551 to 552 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 552 (ObjNum 801 snap 0 seq_num 801) dirty exists 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1426: left oid 164 (ObjNum 801 snap 0 seq_num 801) 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: finishing write tid 3 to smithi16030472-51 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: finishing write tid 4 to smithi16030472-51 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: finishing write tid 5 to smithi16030472-51 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: oid 51 updating version 0 to 502 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: oid 51 version 502 is already newer than 500 2024-09-21T18:08:23.919 INFO:tasks.rados.rados.0.smithi160.stdout:1431: oid 51 version 502 is already newer than 501 2024-09-21T18:08:23.920 INFO:tasks.rados.rados.0.smithi160.stdout:1431: oid 51 updating version 502 to 503 2024-09-21T18:08:23.920 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 503 (ObjNum 803 snap 0 seq_num 803) dirty exists 2024-09-21T18:08:23.920 INFO:tasks.rados.rados.0.smithi160.stdout:1431: left oid 51 (ObjNum 803 snap 0 seq_num 803) 2024-09-21T18:08:23.920 INFO:tasks.rados.rados.0.smithi160.stdout:1434: got (16) Device or resource busy 2024-09-21T18:08:23.943 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:23.948 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:23.974 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:23.973+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:23.980 INFO:tasks.rados.rados.0.smithi160.stdout:1429: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:08:23.983 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1613311 2024-09-21T18:08:23.995 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:24.019 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 245430 2024-09-21T18:08:24.019 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:24.019 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:24.019 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:24.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: finishing write tid 4 to smithi16030472-328 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: finishing write tid 5 to smithi16030472-328 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: oid 328 updating version 0 to 553 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: oid 328 version 553 is already newer than 549 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: oid 328 version 553 is already newer than 548 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: oid 328 version 553 is already newer than 547 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 553 (ObjNum 799 snap 0 seq_num 799) dirty exists 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: left oid 328 (ObjNum 799 snap 0 seq_num 799) 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1433: finishing write tid 1 to smithi16030472-4 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1424: done (7 left) 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1426: done (6 left) 2024-09-21T18:08:24.041 INFO:tasks.rados.rados.0.smithi160.stdout:1428: done (5 left) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1429: done (4 left) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1431: done (3 left) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1434: done (2 left) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1435: done (1 left) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1436: read oid 38 snap -1 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1436: expect (ObjNum 500 snap 0 seq_num 500) 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1437: write (excl) oid 311 current snap is 0 2024-09-21T18:08:24.042 INFO:tasks.rados.rados.0.smithi160.stdout:1437: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-21T18:08:24.052 INFO:tasks.rados.rados.0.smithi160.stdout:1437: writing smithi16030472-311 from 763249 to 1422610 tid 1 2024-09-21T18:08:24.063 INFO:tasks.rados.rados.0.smithi160.stdout:1437: writing smithi16030472-311 from 2090498 to 2673114 tid 2 2024-09-21T18:08:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:1437: writing smithi16030472-311 from 3295061 to 3563249 tid 3 2024-09-21T18:08:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:1438: cache_flush oid 147 snap -1 2024-09-21T18:08:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:1439: cache_flush oid 414 snap -1 2024-09-21T18:08:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:1440: write oid 258 current snap is 0 2024-09-21T18:08:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:1440: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-21T18:08:24.075 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:24.073+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:24.084 INFO:tasks.rados.rados.0.smithi160.stdout:1440: writing smithi16030472-258 from 780056 to 1514666 tid 1 2024-09-21T18:08:24.095 INFO:tasks.rados.rados.0.smithi160.stdout:1440: writing smithi16030472-258 from 2148980 to 2875254 tid 2 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1440: writing smithi16030472-258 from 3322484 to 3580056 tid 3 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1441: read oid 296 snap -1 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1441: expect deleted 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1442: cache_flush oid 455 snap -1 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1439: got (0) Success 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 0 (ObjNum 413 snap 0 seq_num 413) dirty exists 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1439: done (7 left) 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1443: write (excl) oid 51 current snap is 0 2024-09-21T18:08:24.101 INFO:tasks.rados.rados.0.smithi160.stdout:1443: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-21T18:08:24.109 INFO:tasks.rados.rados.0.smithi160.stdout:1443: writing smithi16030472-51 from 796863 to 1206722 tid 1 2024-09-21T18:08:24.117 INFO:tasks.rados.rados.0.smithi160.stdout:1443: writing smithi16030472-51 from 1807462 to 2393747 tid 2 2024-09-21T18:08:24.126 INFO:tasks.rados.rados.0.smithi160.stdout:1443: writing smithi16030472-51 from 3066260 to 3596863 tid 3 2024-09-21T18:08:24.126 INFO:tasks.rados.rados.0.smithi160.stdout:1444: read oid 476 snap -1 2024-09-21T18:08:24.126 INFO:tasks.rados.rados.0.smithi160.stdout:1444: expect (ObjNum 744 snap 0 seq_num 744) 2024-09-21T18:08:24.126 INFO:tasks.rados.rados.0.smithi160.stdout:1445: delete oid 362 current snap is 0 2024-09-21T18:08:24.160 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:24.160 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 71.012269559999993, 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:08:24.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.162 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:08:24.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.167 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 71.012223250999995, 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 71.010127338000004, 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.173 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:24.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.176 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:08:24.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:08:24.179 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 71.009464222999995, 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.180 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:24.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:08:24.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:08:24.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:08:24.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:24.186 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 71.009427724000005, 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:08:24.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 70.875789552000001, 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.192 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:24.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:08:24.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:08:24.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 70.875077761, 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:08:24.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:08:24.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:08:24.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 70.875040197999994, 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.205 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:24.206 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:08:24.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.077654199000001, 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.214 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.077459652999998, 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.218 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.219 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.077173264999999, 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.220 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:08:24.221 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.222 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.223 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.077157184999997, 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:08:24.224 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:08:24.225 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.226 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.227 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.048229933000002, 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.228 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.229 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.230 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:08:24.231 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.042010857999998, 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:08:24.232 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.233 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:08:24.234 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:08:24.235 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.036909262999998, 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.237 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:08:24.238 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.239 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.036575315, 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.241 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.243 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:24.245 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 47.036554836000001, 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:24.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:08:24.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:08:24.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:24.249 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.950226309000001, 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:08:24.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:08:24.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.253 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.941766123000001, 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:08:24.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.935838418000003, 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:08:24.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.262 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:24.264 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:24.264 INFO:tasks.rados.rados.0.smithi160.stdout:1433: finishing write tid 2 to smithi16030472-4 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: finishing write tid 3 to smithi16030472-4 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: finishing write tid 4 to smithi16030472-4 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: finishing write tid 5 to smithi16030472-4 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: oid 4 updating version 0 to 597 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: oid 4 updating version 597 to 598 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: oid 4 updating version 598 to 599 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: oid 4 updating version 599 to 600 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 4 v 600 (ObjNum 805 snap 0 seq_num 805) dirty exists 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1433: left oid 4 (ObjNum 805 snap 0 seq_num 805) 2024-09-21T18:08:24.265 INFO:tasks.rados.rados.0.smithi160.stdout:1436: expect (ObjNum 500 snap 0 seq_num 500) 2024-09-21T18:08:24.265 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:24.257+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:24.276 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:24.330 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1980988 2024-09-21T18:08:24.347 INFO:tasks.rados.rados.0.smithi160.stdout:1444: expect (ObjNum 744 snap 0 seq_num 744) 2024-09-21T18:08:24.364 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:24.361+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:24.404 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:24.404 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:4435 4.2 4.697b435e (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e76)", 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:24.140012+0000", 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.26292108400000003, 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.26296040100000001, 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 4435 2024-09-21T18:08:24.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140012+0000", 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140012+0000", 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140013+0000", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.72e-07 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140015+0000", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140018+0000", 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.084e-06 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:24.140021+0000", 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-21T18:08:24.408 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:24.416 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:24.419 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:24.454 INFO:tasks.rados.rados.0.smithi160.stdout:1438: got (0) Success 2024-09-21T18:08:24.455 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 0 (ObjNum 146 snap 0 seq_num 146) dirty exists 2024-09-21T18:08:24.455 INFO:tasks.rados.rados.0.smithi160.stdout:1437: finishing write tid 1 to smithi16030472-311 2024-09-21T18:08:24.455 INFO:tasks.rados.rados.0.smithi160.stdout:1437: finishing write tid 2 to smithi16030472-311 2024-09-21T18:08:24.455 INFO:tasks.rados.rados.0.smithi160.stdout:1442: got (2) No such file or directory 2024-09-21T18:08:24.465 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:24.465+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:24.538 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:24.538 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:24.538 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:24.552 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:24.583 INFO:tasks.rados.rados.0.smithi160.stdout:1437: finishing write tid 3 to smithi16030472-311 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: finishing write tid 4 to smithi16030472-311 2024-09-21T18:08:24.584 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:24.577+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: finishing write tid 5 to smithi16030472-311 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: oid 311 updating version 0 to 506 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: oid 311 updating version 506 to 507 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: oid 311 version 507 is already newer than 505 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:1437: oid 311 version 507 is already newer than 504 2024-09-21T18:08:24.584 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 311 v 507 (ObjNum 807 snap 0 seq_num 807) dirty exists 2024-09-21T18:08:24.585 INFO:tasks.rados.rados.0.smithi160.stdout:1437: left oid 311 (ObjNum 807 snap 0 seq_num 807) 2024-09-21T18:08:24.667 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:24.663+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.710 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.033034960999998, 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.711 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:08:24.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.032964598999996, 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.716 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:08:24.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.021163164000001, 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.721 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.013296780999994, 2024-09-21T18:08:24.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:08:24.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.731 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.012911137000003, 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.732 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.736 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 66.012865688000005, 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.737 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:24.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:08:24.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:08:24.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:08:24.743 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 64.964967771000005, 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.744 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 64.964860129000002, 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:08:24.749 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:24.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.751 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:08:24.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 59.389077514, 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.754 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.755 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:08:24.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:08:24.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:08:24.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:08:24.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:08:24.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:24.764 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 59.389025578999998, 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:08:24.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:08:24.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.771 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.622321190999997, 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:08:24.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:08:24.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:08:24.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:08:24.778 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.393217933000003, 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.779 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:08:24.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.782 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:08:24.783 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:08:24.784 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.785 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.786 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.392789942999997, 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.787 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.788 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.789 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:08:24.790 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.791 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.792 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:08:24.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.794 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.392743373999998, 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.795 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:08:24.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:08:24.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 53.892578264000001, 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.802 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:24.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:08:24.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:08:24.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 53.892074313999998, 2024-09-21T18:08:24.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:24.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:08:24.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:24.816 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.884542518000003, 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:08:24.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.824 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.884468198, 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.825 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:08:24.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.831 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.482664952999997, 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.832 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 46.380315064999998, 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:24.837 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:24.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:24.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:08:24.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:24.844 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:24.846 INFO:tasks.rados.rados.0.smithi160.stdout:1440: finishing write tid 1 to smithi16030472-258 2024-09-21T18:08:24.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:24.847 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:24.843+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:24.890 INFO:tasks.rados.rados.0.smithi160.stdout:1440: finishing write tid 2 to smithi16030472-258 2024-09-21T18:08:24.890 INFO:tasks.rados.rados.0.smithi160.stdout:1440: finishing write tid 3 to smithi16030472-258 2024-09-21T18:08:24.890 INFO:tasks.rados.rados.0.smithi160.stdout:1440: finishing write tid 4 to smithi16030472-258 2024-09-21T18:08:24.903 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:08:24.946 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:24.939+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:24.984 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:24.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:24.985 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:24.985 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:24.991 INFO:tasks.rados.rados.0.smithi160.stdout:1440: finishing write tid 5 to smithi16030472-258 2024-09-21T18:08:24.991 INFO:tasks.rados.rados.0.smithi160.stdout:1440: oid 258 updating version 0 to 511 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1440: oid 258 version 511 is already newer than 510 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1440: oid 258 version 511 is already newer than 509 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1440: oid 258 version 511 is already newer than 508 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 511 (ObjNum 808 snap 0 seq_num 808) dirty exists 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1440: left oid 258 (ObjNum 808 snap 0 seq_num 808) 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1443: finishing write tid 1 to smithi16030472-51 2024-09-21T18:08:24.992 INFO:tasks.rados.rados.0.smithi160.stdout:1443: finishing write tid 2 to smithi16030472-51 2024-09-21T18:08:24.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:25.087 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:25.083+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:25.092 INFO:tasks.rados.rados.0.smithi160.stdout:1443: finishing write tid 3 to smithi16030472-51 2024-09-21T18:08:25.092 INFO:tasks.rados.rados.0.smithi160.stdout:1443: finishing write tid 4 to smithi16030472-51 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: finishing write tid 5 to smithi16030472-51 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: oid 51 updating version 0 to 515 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: oid 51 version 515 is already newer than 512 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: oid 51 version 515 is already newer than 513 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: oid 51 version 515 is already newer than 514 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 515 (ObjNum 809 snap 0 seq_num 809) dirty exists 2024-09-21T18:08:25.093 INFO:tasks.rados.rados.0.smithi160.stdout:1443: left oid 51 (ObjNum 809 snap 0 seq_num 809) 2024-09-21T18:08:25.147 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:25.145+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:25.168 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:25.168 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:25.168 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:25.168 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:25.168 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:25.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:25.192 INFO:tasks.rados.rados.0.smithi160.stdout:1433: done (9 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1436: done (8 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1437: done (7 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1438: done (6 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1440: done (5 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1441: done (4 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1442: done (3 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1443: done (2 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1444: done (1 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1445: done (0 left) 2024-09-21T18:08:25.193 INFO:tasks.rados.rados.0.smithi160.stdout:1446: delete oid 424 current snap is 0 2024-09-21T18:08:25.198 INFO:tasks.rados.rados.0.smithi160.stdout:1446: done (0 left) 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1447: copy_from oid 150 from oid 289 current snap is 0 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1448: copy_from oid 366 from oid 81 current snap is 0 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1449: copy_from oid 279 from oid 317 current snap is 0 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1450: read oid 87 snap -1 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1450: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1451: cache_flush oid 173 snap -1 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1452: cache_try_flush oid 459 snap -1 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1453: write oid 257 current snap is 0 2024-09-21T18:08:25.199 INFO:tasks.rados.rados.0.smithi160.stdout:1453: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-21T18:08:25.212 INFO:tasks.rados.rados.0.smithi160.stdout:1453: writing smithi16030472-257 from 447284 to 1082890 tid 1 2024-09-21T18:08:25.230 INFO:tasks.rados.rados.0.smithi160.stdout:1453: writing smithi16030472-257 from 1866555 to 2600167 tid 2 2024-09-21T18:08:25.243 INFO:tasks.rados.rados.0.smithi160.stdout:1453: writing smithi16030472-257 from 3315823 to 3647284 tid 3 2024-09-21T18:08:25.243 INFO:tasks.rados.rados.0.smithi160.stdout:1454: cache_try_flush oid 266 snap -1 2024-09-21T18:08:25.243 INFO:tasks.rados.rados.0.smithi160.stdout:1455: copy_from oid 476 from oid 181 current snap is 0 2024-09-21T18:08:25.243 INFO:tasks.rados.rados.0.smithi160.stdout:1456: write oid 250 current snap is 0 2024-09-21T18:08:25.243 INFO:tasks.rados.rados.0.smithi160.stdout:1456: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2024-09-21T18:08:25.248 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:25.245+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:25.256 INFO:tasks.rados.rados.0.smithi160.stdout:1456: writing smithi16030472-250 from 464091 to 1174946 tid 1 2024-09-21T18:08:25.267 INFO:tasks.rados.rados.0.smithi160.stdout:1456: writing smithi16030472-250 from 1925037 to 2402307 tid 2 2024-09-21T18:08:25.275 INFO:tasks.rados.rados.0.smithi160.stdout:1456: writing smithi16030472-250 from 2943246 to 3418216 tid 3 2024-09-21T18:08:25.276 INFO:tasks.rados.rados.0.smithi160.stdout:1457: read oid 172 snap -1 2024-09-21T18:08:25.276 INFO:tasks.rados.rados.0.smithi160.stdout:1457: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:25.276 INFO:tasks.rados.rados.0.smithi160.stdout:1458: write oid 407 current snap is 0 2024-09-21T18:08:25.276 INFO:tasks.rados.rados.0.smithi160.stdout:1458: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-21T18:08:25.282 INFO:tasks.rados.rados.0.smithi160.stdout:1458: writing smithi16030472-407 from 480898 to 983355 tid 1 2024-09-21T18:08:25.292 INFO:tasks.rados.rados.0.smithi160.stdout:1458: writing smithi16030472-407 from 1699872 to 2437153 tid 2 2024-09-21T18:08:25.299 INFO:tasks.rados.rados.0.smithi160.stdout:1458: writing smithi16030472-407 from 3087022 to 3680898 tid 3 2024-09-21T18:08:25.299 INFO:tasks.rados.rados.0.smithi160.stdout:1459: read oid 255 snap -1 2024-09-21T18:08:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:1459: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:08:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:1460: read oid 402 snap -1 2024-09-21T18:08:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:1460: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-21T18:08:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:1461: write oid 344 current snap is 0 2024-09-21T18:08:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:1461: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-21T18:08:25.306 INFO:tasks.rados.rados.0.smithi160.stdout:1461: writing smithi16030472-344 from 497705 to 1075411 tid 1 2024-09-21T18:08:25.312 INFO:tasks.rados.rados.0.smithi160.stdout:1461: writing smithi16030472-344 from 1642001 to 2122940 tid 2 2024-09-21T18:08:25.317 INFO:tasks.rados.rados.0.smithi160.stdout:1461: writing smithi16030472-344 from 2598092 to 3095606 tid 3 2024-09-21T18:08:25.317 INFO:tasks.rados.rados.0.smithi160.stdout:1462: cache_flush oid 161 snap -1 2024-09-21T18:08:25.317 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:25.349 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:25.345+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:25.358 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:25.358 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:25.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:25.359 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:25.359 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:25.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:25.400 INFO:tasks.rados.rados.0.smithi160.stdout:1451: got (0) Success 2024-09-21T18:08:25.400 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 0 (ObjNum 605 snap 0 seq_num 605) dirty exists 2024-09-21T18:08:25.400 INFO:tasks.rados.rados.0.smithi160.stdout:1451: done (15 left) 2024-09-21T18:08:25.400 INFO:tasks.rados.rados.0.smithi160.stdout:1463: delete oid 165 current snap is 0 2024-09-21T18:08:25.446 INFO:tasks.rados.rados.0.smithi160.stdout:1452: got (0) Success 2024-09-21T18:08:25.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 0 (ObjNum 458 snap 0 seq_num 458) dirty exists 2024-09-21T18:08:25.449 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:25.445+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:25.450 INFO:tasks.rados.rados.0.smithi160.stdout:1454: got (0) Success 2024-09-21T18:08:25.451 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 0 (ObjNum 265 snap 0 seq_num 265) dirty exists 2024-09-21T18:08:25.455 INFO:tasks.rados.rados.0.smithi160.stdout:1459: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:08:25.456 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4453 4.0 4:1c164a33:::smithi16030472-87:head [sparse-read 1438477~2006925] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.198321+0000", 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.30672991199999999, 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30680010400000002, 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:25.508 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4453 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198321+0000", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198321+0000", 2024-09-21T18:08:25.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198323+0000", 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.33e-06 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198328+0000", 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1830000000000002e-06 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198339+0000", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.147e-05 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198348+0000", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2630000000000003e-06 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198848+0000", 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00050014500000000002 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198901+0000", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3659000000000002e-05 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.328021+0000", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.12911913999999999 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.328180+0000", 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015917500000000001 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.512 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4454 4.0 4:1c164a33:::smithi16030472-87:head [read 1378393~2067009,checksum in=4b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e76)", 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.198388+0000", 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.30666356, 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30700858199999997, 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4454 2024-09-21T18:08:25.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198388+0000", 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198388+0000", 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198389+0000", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1570000000000001e-06 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198395+0000", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9000000000000003e-06 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198418+0000", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3371999999999999e-05 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198450+0000", 2024-09-21T18:08:25.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1757000000000001e-05 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198952+0000", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000502265 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.198992+0000", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9505999999999997e-05 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.328308+0000", 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.129315971 2024-09-21T18:08:25.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.328456+0000", 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014794100000000001 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4463 4.0 4:26764ef7:::smithi16030472-476:head [copy-from ver 140 in=56b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.343037+0000", 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.162013991, 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.16260530600000001, 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4463 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343037+0000", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343037+0000", 2024-09-21T18:08:25.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343040+0000", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2800000000000002e-06 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343049+0000", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4320000000000005e-06 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343060+0000", 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0621999999999999e-05 2024-09-21T18:08:25.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343066+0000", 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343379+0000", 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031297999999999997 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343473+0000", 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3183000000000006e-05 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4464 4.0 4:26764ef7:::smithi16030472-476:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.343074+0000", 2024-09-21T18:08:25.521 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.161977276, 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.16273723500000001, 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4464 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343074+0000", 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343074+0000", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343075+0000", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0800000000000004e-07 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343077+0000", 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3879999999999998e-06 2024-09-21T18:08:25.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343082+0000", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6160000000000004e-06 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343086+0000", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.177e-06 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343662+0000", 2024-09-21T18:08:25.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000576626 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.343745+0000", 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2149000000000002e-05 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4484 4.0 4:2f6cf193:::smithi16030472-344:head [write 497705~577706 in=577706b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.443321+0000", 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.061730421000000001, 2024-09-21T18:08:25.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.062611087999999995, 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4484 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443321+0000", 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443321+0000", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443324+0000", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3809999999999999e-06 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443864+0000", 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00053954199999999995 2024-09-21T18:08:25.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443879+0000", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.519e-05 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443889+0000", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0336000000000001e-05 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444980+0000", 2024-09-21T18:08:25.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00109033 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.445199+0000", 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021943599999999999 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.445390+0000", 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019077800000000001 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4485 4.0 4:2f6cf193:::smithi16030472-344:head [write 1642001~480939 in=480939b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.443908+0000", 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.061143031, 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.062250490999999998, 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4485 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443908+0000", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443908+0000", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.443910+0000", 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3239999999999999e-06 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444356+0000", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044679399999999998 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444369+0000", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2462e-05 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444377+0000", 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3119999999999994e-06 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.449571+0000", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0051933559999999997 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.449768+0000", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000197784 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.449979+0000", 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021033299999999999 2024-09-21T18:08:25.533 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4486 4.0 4:2f6cf193:::smithi16030472-344:head [write 2598092~497514 in=497514b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.444391+0000", 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.060659986999999999, 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.061890869000000001, 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.534 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4486 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444391+0000", 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444391+0000", 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444393+0000", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.249e-06 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444620+0000", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022720800000000001 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444631+0000", 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.098e-05 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444639+0000", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8169999999999999e-06 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.453404+0000", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0087653560000000002 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.453699+0000", 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029475900000000001 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.454051+0000", 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035257799999999998 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4487 4.0 4.c98f36f4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.444650+0000", 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.060401115999999998, 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.061731345999999999, 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.538 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4487 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444650+0000", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444650+0000", 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444651+0000", 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0330000000000001e-06 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444656+0000", 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5800000000000002e-06 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444662+0000", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2979999999999997e-06 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444667+0000", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1510000000000003e-06 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4488 4.0 4.c98f36f4 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.444675+0000", 2024-09-21T18:08:25.541 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.060376377000000002, 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.061779250000000001, 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4488 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444675+0000", 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444675+0000", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444676+0000", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1200000000000002e-07 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444679+0000", 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0910000000000001e-06 2024-09-21T18:08:25.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444685+0000", 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0990000000000004e-06 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444690+0000", 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3519999999999997e-06 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.544 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:4489 4.0 4.2fbf0320 (undecoded) ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.444698+0000", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.060353293000000002, 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.061829538000000003, 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4489 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444698+0000", 2024-09-21T18:08:25.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444698+0000", 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444699+0000", 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3900000000000004e-07 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444701+0000", 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6429999999999999e-06 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444707+0000", 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3839999999999996e-06 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.444712+0000", 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.809e-06 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(osd.11.22:1814 3.1 3:a40e217e:::smithi16030472-172:head [read 0~2873997 out=2873997b,checksum in=4b] snapc 0=[] ondisk+read+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e77)", 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:25.504980+0000", 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 7.1599000000000003e-05, 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0016291960000000001, 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.504980+0000", 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.504980+0000", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.504985+0000", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3329999999999996e-06 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.505001+0000", 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6305999999999999e-05 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.505035+0000", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3733999999999999e-05 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.505054+0000", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8816000000000001e-05 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.505237+0000", 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183234 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:25.505383+0000", 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014591100000000001 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 11 2024-09-21T18:08:25.551 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:25.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:25.553 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:25.547+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:25.555 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 360074 2024-09-21T18:08:25.579 INFO:tasks.rados.rados.0.smithi160.stdout:1448: finishing copy_from racing read to smithi16030472-366 2024-09-21T18:08:25.579 INFO:tasks.rados.rados.0.smithi160.stdout:1457: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:25.611 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:25.654 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:25.649+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:25.666 INFO:tasks.rados.rados.0.smithi160.stdout:1453: finishing write tid 1 to smithi16030472-257 2024-09-21T18:08:25.666 INFO:tasks.rados.rados.0.smithi160.stdout:1455: finishing copy_from racing read to smithi16030472-476 2024-09-21T18:08:25.667 INFO:tasks.rados.rados.0.smithi160.stdout:1462: got (0) Success 2024-09-21T18:08:25.667 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 161 v 0 (ObjNum 160 snap 0 seq_num 160) dirty exists 2024-09-21T18:08:25.667 INFO:tasks.rados.rados.0.smithi160.stdout:1450: expect (ObjNum 86 snap 0 seq_num 86) 2024-09-21T18:08:25.667 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:25.718 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:25.718 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:25.718 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:25.718 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:25.718 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:25.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:25.735 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1378393 2024-09-21T18:08:25.755 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:25.749+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:1449: finishing copy_from racing read to smithi16030472-279 2024-09-21T18:08:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:1456: finishing write tid 1 to smithi16030472-250 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: finishing write tid 2 to smithi16030472-250 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: finishing write tid 3 to smithi16030472-250 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: finishing write tid 4 to smithi16030472-250 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: finishing write tid 5 to smithi16030472-250 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: oid 250 updating version 0 to 603 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: oid 250 updating version 603 to 605 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: oid 250 version 605 is already newer than 602 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: oid 250 version 605 is already newer than 604 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 250 v 605 (ObjNum 813 snap 0 seq_num 813) dirty exists 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1456: left oid 250 (ObjNum 813 snap 0 seq_num 813) 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1458: finishing write tid 1 to smithi16030472-407 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1453: finishing write tid 2 to smithi16030472-257 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1453: finishing write tid 3 to smithi16030472-257 2024-09-21T18:08:25.763 INFO:tasks.rados.rados.0.smithi160.stdout:1453: finishing write tid 4 to smithi16030472-257 2024-09-21T18:08:25.796 INFO:tasks.rados.rados.0.smithi160.stdout:1461: finishing write tid 1 to smithi16030472-344 2024-09-21T18:08:25.796 INFO:tasks.rados.rados.0.smithi160.stdout:1461: finishing write tid 2 to smithi16030472-344 2024-09-21T18:08:25.796 INFO:tasks.rados.rados.0.smithi160.stdout:1461: finishing write tid 3 to smithi16030472-344 2024-09-21T18:08:25.796 INFO:tasks.rados.rados.0.smithi160.stdout:1461: finishing write tid 4 to smithi16030472-344 2024-09-21T18:08:25.806 INFO:tasks.rados.rados.0.smithi160.stdout:1458: finishing write tid 2 to smithi16030472-407 2024-09-21T18:08:25.806 INFO:tasks.rados.rados.0.smithi160.stdout:1458: finishing write tid 3 to smithi16030472-407 2024-09-21T18:08:25.807 INFO:tasks.rados.rados.0.smithi160.stdout:1447: finishing copy_from to smithi16030472-150 2024-09-21T18:08:25.807 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 150 v 177 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-21T18:08:25.811 INFO:tasks.rados.rados.0.smithi160.stdout:1447: finishing copy_from racing read to smithi16030472-150 2024-09-21T18:08:25.811 INFO:tasks.rados.rados.0.smithi160.stdout:1458: finishing write tid 4 to smithi16030472-407 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: finishing write tid 5 to smithi16030472-407 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: oid 407 updating version 0 to 606 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: oid 407 updating version 606 to 609 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: oid 407 version 609 is already newer than 607 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: oid 407 version 609 is already newer than 608 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 609 (ObjNum 814 snap 0 seq_num 814) dirty exists 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1458: left oid 407 (ObjNum 814 snap 0 seq_num 814) 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1448: finishing copy_from to smithi16030472-366 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 499 (ObjNum 786 snap 0 seq_num 786) dirty exists 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:1449: finishing copy_from to smithi16030472-279 2024-09-21T18:08:25.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 279 v 403 (ObjNum 536 snap 0 seq_num 536) dirty exists 2024-09-21T18:08:25.819 INFO:tasks.rados.rados.0.smithi160.stdout:1460: expect (ObjNum 401 snap 0 seq_num 401) 2024-09-21T18:08:25.855 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:25.849+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 72.651869199999993, 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.866 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:08:25.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.872 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 55.107931651000001, 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:08:25.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:08:25.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.947008187000002, 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:08:25.878 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:08:25.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:08:25.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.946333592000002, 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:08:25.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:08:25.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.946307054000002, 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:08:25.890 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.936473311999997, 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.891 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.924061524000003, 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.895 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:25.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:08:25.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.915171454000003, 2024-09-21T18:08:25.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.907 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.914359437000002, 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.913 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.914251225000001, 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.914 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:08:25.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:08:25.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.813980078999997, 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:08:25.919 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.920 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.921 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.922 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.923 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.813083820000003, 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.924 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.925 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.927 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.812602206999998, 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.928 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.812149781999999, 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.932 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:25.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:25.936 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.812099455999999, 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:08:25.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:08:25.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.686366675999999, 2024-09-21T18:08:25.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.945 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.946 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.682088618999998, 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.947 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.949 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.950 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.952 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.681718379000003, 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.953 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:08:25.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.955 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.956 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.681648649000003, 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.959 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 54.090071659000003, 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:25.964 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:25.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:25.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:25.969 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:25.970 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:25.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:25.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1284022 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: finishing write tid 5 to smithi16030472-344 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: oid 344 updating version 0 to 618 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: oid 344 updating version 618 to 620 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: oid 344 version 620 is already newer than 619 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: oid 344 version 620 is already newer than 617 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 344 v 620 (ObjNum 815 snap 0 seq_num 815) dirty exists 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1461: left oid 344 (ObjNum 815 snap 0 seq_num 815) 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1455: finishing copy_from to smithi16030472-476 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 476 v 140 (ObjNum 180 snap 0 seq_num 180) dirty exists 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1447: done (15 left) 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1448: done (14 left) 2024-09-21T18:08:25.973 INFO:tasks.rados.rados.0.smithi160.stdout:1449: done (13 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1450: done (12 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1452: done (11 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1454: done (10 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1455: done (9 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1456: done (8 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1457: done (7 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1458: done (6 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1459: done (5 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1460: done (4 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1461: done (3 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1462: done (2 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1463: done (1 left) 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1464: copy_from oid 496 from oid 137 current snap is 0 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1465: cache_try_flush oid 440 snap -1 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1466: copy_from oid 398 from oid 258 current snap is 0 2024-09-21T18:08:25.974 INFO:tasks.rados.rados.0.smithi160.stdout:1467: cache_evict oid 5 snap -1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1468: read oid 51 snap -1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1468: expect (ObjNum 809 snap 0 seq_num 809) 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1469: cache_evict oid 105 snap -1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1470: cache_flush oid 471 snap -1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1471: write oid 265 current snap is 0 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1471: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1471: writing smithi16030472-265 from 531319 to 1259523 tid 1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1471: writing smithi16030472-265 from 1758965 to 2243573 tid 2 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1471: writing smithi16030472-265 from 2652938 to 3172996 tid 3 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1472: read oid 370 snap -1 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1472: expect (ObjNum 731 snap 0 seq_num 731) 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1473: delete oid 167 current snap is 0 2024-09-21T18:08:25.975 INFO:tasks.rados.rados.0.smithi160.stdout:1468: expect (ObjNum 809 snap 0 seq_num 809) 2024-09-21T18:08:25.976 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:25.969+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:26.072 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:26.069+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:26.110 INFO:tasks.rados.rados.0.smithi160.stdout:1453: finishing write tid 5 to smithi16030472-257 2024-09-21T18:08:26.110 INFO:tasks.rados.rados.0.smithi160.stdout:1453: oid 257 updating version 0 to 553 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1453: oid 257 version 553 is already newer than 552 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1453: oid 257 version 553 is already newer than 550 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1453: oid 257 version 553 is already newer than 551 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 257 v 553 (ObjNum 812 snap 0 seq_num 812) dirty exists 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1453: left oid 257 (ObjNum 812 snap 0 seq_num 812) 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1465: got (2) No such file or directory 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1467: got (16) Device or resource busy 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1471: finishing write tid 1 to smithi16030472-265 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1464: finishing copy_from racing read to smithi16030472-496 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1466: finishing copy_from to smithi16030472-398 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 398 v 511 (ObjNum 808 snap 0 seq_num 808) dirty exists 2024-09-21T18:08:26.111 INFO:tasks.rados.rados.0.smithi160.stdout:1466: finishing copy_from racing read to smithi16030472-398 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: finishing write tid 2 to smithi16030472-265 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: finishing write tid 3 to smithi16030472-265 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: finishing write tid 4 to smithi16030472-265 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: finishing write tid 5 to smithi16030472-265 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: oid 265 updating version 0 to 494 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: oid 265 version 494 is already newer than 493 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: oid 265 updating version 494 to 495 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: oid 265 updating version 495 to 496 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 265 v 496 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:08:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:1471: left oid 265 (ObjNum 817 snap 0 seq_num 817) 2024-09-21T18:08:26.128 INFO:tasks.rados.rados.0.smithi160.stdout:1472: expect (ObjNum 731 snap 0 seq_num 731) 2024-09-21T18:08:26.155 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:26.155 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:26.155 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:26.155 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:26.162 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:26.162 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:26.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:26.171 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1625540 2024-09-21T18:08:26.172 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:26.165+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:26.183 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:26.183 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:26.204 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:26.223 INFO:tasks.rados.rados.0.smithi160.stdout:1470: got (0) Success 2024-09-21T18:08:26.223 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 471 v 0 (ObjNum 746 snap 0 seq_num 746) dirty exists 2024-09-21T18:08:26.223 INFO:tasks.rados.rados.0.smithi160.stdout:1464: finishing copy_from to smithi16030472-496 2024-09-21T18:08:26.223 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 383 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:08:26.224 INFO:tasks.rados.rados.0.smithi160.stdout:1469: got (0) Success 2024-09-21T18:08:26.224 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:26.224 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:26.232 INFO:tasks.rados.rados.0.smithi160.stdout:1453: done (10 left) 2024-09-21T18:08:26.232 INFO:tasks.rados.rados.0.smithi160.stdout:1464: done (9 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1465: done (8 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1466: done (7 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1467: done (6 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1468: done (5 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1469: done (4 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1470: done (3 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1471: done (2 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1472: done (1 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1473: done (0 left) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1474: cache_flush oid 129 snap -1 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1475: cache_try_flush oid 41 snap -1 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1476: read oid 369 snap -1 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1476: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1477: write oid 408 current snap is 0 2024-09-21T18:08:26.233 INFO:tasks.rados.rados.0.smithi160.stdout:1477: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-09-21T18:08:26.239 INFO:tasks.rados.rados.0.smithi160.stdout:1477: writing smithi16030472-408 from 564933 to 1043635 tid 1 2024-09-21T18:08:26.242 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:26.243 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:26.246 INFO:tasks.rados.rados.0.smithi160.stdout:1477: writing smithi16030472-408 from 1759576 to 2247853 tid 2 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:1477: writing smithi16030472-408 from 2991431 to 3650386 tid 3 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:1478: cache_evict oid 361 snap -1 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:1474: got (0) Success 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:1474: done (4 left) 2024-09-21T18:08:26.254 INFO:tasks.rados.rados.0.smithi160.stdout:1479: copy_from oid 10 from oid 413 current snap is 0 2024-09-21T18:08:26.255 INFO:tasks.rados.rados.0.smithi160.stdout:1476: expect (ObjNum 368 snap 0 seq_num 368) 2024-09-21T18:08:26.267 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:26.267 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:26.273 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:26.269+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:26.284 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 968805 2024-09-21T18:08:26.287 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:26.289 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:26.289 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:26.290 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:26.290 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:26.290 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:26.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:26.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:26.305 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:26.321 INFO:tasks.rados.rados.0.smithi160.stdout:1476: done (4 left) 2024-09-21T18:08:26.321 INFO:tasks.rados.rados.0.smithi160.stdout:1480: delete oid 321 current snap is 0 2024-09-21T18:08:26.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:26.346 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:26.347 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:26.349 INFO:tasks.rados.rados.0.smithi160.stdout:1478: got (16) Device or resource busy 2024-09-21T18:08:26.367 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:26.374 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:26.369+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:26.385 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:26.406 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:26.406 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:26.427 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:26.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:26.452 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:26.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:26.475 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:26.473+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1475: got (0) Success 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 41 v 0 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1475: done (4 left) 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1478: done (3 left) 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1480: done (2 left) 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1481: read oid 453 snap -1 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1481: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1482: cache_evict oid 43 snap -1 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1483: write (excl) oid 70 current snap is 0 2024-09-21T18:08:26.551 INFO:tasks.rados.rados.0.smithi160.stdout:1483: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-21T18:08:26.557 INFO:tasks.rados.rados.0.smithi160.stdout:1483: writing smithi16030472-70 from 598547 to 1227747 tid 1 2024-09-21T18:08:26.564 INFO:tasks.rados.rados.0.smithi160.stdout:1483: writing smithi16030472-70 from 1876540 to 2368486 tid 2 2024-09-21T18:08:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:26.571 INFO:tasks.rados.rados.0.smithi160.stdout:1483: writing smithi16030472-70 from 3046277 to 3727776 tid 3 2024-09-21T18:08:26.571 INFO:tasks.rados.rados.0.smithi160.stdout:1484: cache_try_flush oid 241 snap -1 2024-09-21T18:08:26.571 INFO:tasks.rados.rados.0.smithi160.stdout:1485: copy_from oid 56 from oid 234 current snap is 0 2024-09-21T18:08:26.571 INFO:tasks.rados.rados.0.smithi160.stdout:1486: copy_from oid 166 from oid 190 current snap is 0 2024-09-21T18:08:26.571 INFO:tasks.rados.rados.0.smithi160.stdout:1487: write oid 46 current snap is 0 2024-09-21T18:08:26.572 INFO:tasks.rados.rados.0.smithi160.stdout:1487: seq_num 822 ranges {615354=420802,1651375=751957,3190053=492771} 2024-09-21T18:08:26.576 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:26.571+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:26.582 INFO:tasks.rados.rados.0.smithi160.stdout:1487: writing smithi16030472-46 from 615354 to 1036156 tid 1 2024-09-21T18:08:26.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:26.594 INFO:tasks.rados.rados.0.smithi160.stdout:1487: writing smithi16030472-46 from 1651375 to 2403332 tid 2 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1487: writing smithi16030472-46 from 3190053 to 3682824 tid 3 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1488: copy_from oid 433 from oid 354 current snap is 0 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1487: finishing write tid 1 to smithi16030472-46 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1489: cache_try_flush oid 226 snap -1 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1490: cache_evict oid 489 snap -1 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1491: write (excl) oid 454 current snap is 0 2024-09-21T18:08:26.605 INFO:tasks.rados.rados.0.smithi160.stdout:1491: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-21T18:08:26.613 INFO:tasks.rados.rados.0.smithi160.stdout:1491: writing smithi16030472-454 from 632161 to 1128212 tid 1 2024-09-21T18:08:26.619 INFO:tasks.rados.rados.0.smithi160.stdout:1491: writing smithi16030472-454 from 1593504 to 2089119 tid 2 2024-09-21T18:08:26.625 INFO:tasks.rados.rados.0.smithi160.stdout:1491: writing smithi16030472-454 from 2701123 to 3121519 tid 3 2024-09-21T18:08:26.629 INFO:tasks.rados.rados.0.smithi160.stdout:1491: writing smithi16030472-454 from 3611645 to 3832161 tid 4 2024-09-21T18:08:26.629 INFO:tasks.rados.rados.0.smithi160.stdout:1492: cache_evict oid 130 snap -1 2024-09-21T18:08:26.630 INFO:tasks.rados.rados.0.smithi160.stdout:1493: read oid 378 snap -1 2024-09-21T18:08:26.630 INFO:tasks.rados.rados.0.smithi160.stdout:1493: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:08:26.630 INFO:tasks.rados.rados.0.smithi160.stdout:1494: cache_evict oid 148 snap -1 2024-09-21T18:08:26.630 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:26.677 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:26.669+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:26.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:26.778 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:26.773+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:26.879 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:26.875+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:26.892 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:26.892 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:26.908 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:26.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout:1477: finishing write tid 1 to smithi16030472-408 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout:1477: finishing write tid 2 to smithi16030472-408 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout:1482: got (16) Device or resource busy 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout:1482: done (15 left) 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout:1495: copy_from oid 177 from oid 293 current snap is 0 2024-09-21T18:08:26.958 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:26.959 INFO:tasks.rados.rados.0.smithi160.stdout:1484: got (0) Success 2024-09-21T18:08:26.959 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 0 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-21T18:08:26.959 INFO:tasks.rados.rados.0.smithi160.stdout:1484: done (15 left) 2024-09-21T18:08:26.959 INFO:tasks.rados.rados.0.smithi160.stdout:1496: write oid 303 current snap is 0 2024-09-21T18:08:26.959 INFO:tasks.rados.rados.0.smithi160.stdout:1496: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2024-09-21T18:08:26.969 INFO:tasks.rados.rados.0.smithi160.stdout:1496: writing smithi16030472-303 from 648968 to 1220268 tid 1 2024-09-21T18:08:26.977 INFO:tasks.rados.rados.0.smithi160.stdout:1496: writing smithi16030472-303 from 1651986 to 2291259 tid 2 2024-09-21T18:08:26.980 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:26.973+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:26.985 INFO:tasks.rados.rados.0.smithi160.stdout:1496: writing smithi16030472-303 from 2728546 to 3360214 tid 3 2024-09-21T18:08:26.985 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:26.985 INFO:tasks.rados.rados.0.smithi160.stdout:1481: expect (ObjNum 452 snap 0 seq_num 452) 2024-09-21T18:08:27.041 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:27.044 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.044 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.059 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:27.081 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:27.075+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:27.100 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2206547 2024-09-21T18:08:27.115 INFO:tasks.rados.rados.0.smithi160.stdout:1485: finishing copy_from racing read to smithi16030472-56 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1485: finishing copy_from to smithi16030472-56 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 564 (ObjNum 781 snap 0 seq_num 781) dirty exists 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1490: got (2) No such file or directory 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 1 to smithi16030472-454 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1494: got (2) No such file or directory 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1477: finishing write tid 3 to smithi16030472-408 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1477: finishing write tid 4 to smithi16030472-408 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1481: done (15 left) 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1485: done (14 left) 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1490: done (13 left) 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1494: done (12 left) 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1497: write oid 115 current snap is 0 2024-09-21T18:08:27.116 INFO:tasks.rados.rados.0.smithi160.stdout:1497: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-21T18:08:27.123 INFO:tasks.rados.rados.0.smithi160.stdout:1497: writing smithi16030472-115 from 665775 to 1312324 tid 1 2024-09-21T18:08:27.136 INFO:tasks.rados.rados.0.smithi160.stdout:1497: writing smithi16030472-115 from 2110468 to 2609752 tid 2 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1497: writing smithi16030472-115 from 3155969 to 3598909 tid 3 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1498: cache_try_flush oid 290 snap -1 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: finishing write tid 5 to smithi16030472-408 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: oid 408 updating version 0 to 557 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: oid 408 version 557 is already newer than 554 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: oid 408 version 557 is already newer than 556 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: oid 408 version 557 is already newer than 555 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 557 (ObjNum 819 snap 0 seq_num 819) dirty exists 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1477: left oid 408 (ObjNum 819 snap 0 seq_num 819) 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:1489: got (0) Success 2024-09-21T18:08:27.150 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 226 v 0 (ObjNum 225 snap 0 seq_num 225) dirty exists 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: finishing write tid 2 to smithi16030472-46 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: finishing write tid 3 to smithi16030472-46 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: finishing write tid 4 to smithi16030472-46 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1477: done (13 left) 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1489: done (12 left) 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1499: cache_try_flush oid 233 snap -1 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: finishing write tid 5 to smithi16030472-46 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: oid 46 updating version 0 to 625 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: oid 46 version 625 is already newer than 622 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: oid 46 version 625 is already newer than 624 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: oid 46 version 625 is already newer than 623 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 625 (ObjNum 822 snap 0 seq_num 822) dirty exists 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: left oid 46 (ObjNum 822 snap 0 seq_num 822) 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1492: got (0) Success 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1487: done (12 left) 2024-09-21T18:08:27.151 INFO:tasks.rados.rados.0.smithi160.stdout:1492: done (11 left) 2024-09-21T18:08:27.152 INFO:tasks.rados.rados.0.smithi160.stdout:1500: read oid 59 snap -1 2024-09-21T18:08:27.152 INFO:tasks.rados.rados.0.smithi160.stdout:1500: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:27.152 INFO:tasks.rados.rados.0.smithi160.stdout:1501: cache_evict oid 195 snap -1 2024-09-21T18:08:27.152 INFO:tasks.rados.rados.0.smithi160.stdout:1502: cache_try_flush oid 132 snap -1 2024-09-21T18:08:27.152 INFO:tasks.rados.rados.0.smithi160.stdout:1493: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:08:27.179 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.179 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.182 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:27.175+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:27.194 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:27.218 INFO:tasks.rados.rados.0.smithi160.stdout:1486: finishing copy_from racing read to smithi16030472-166 2024-09-21T18:08:27.218 INFO:tasks.rados.rados.0.smithi160.stdout:1500: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:27.218 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:27.251 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1258035 2024-09-21T18:08:27.273 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:27.282 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:27.277+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:27.303 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 2 to smithi16030472-454 2024-09-21T18:08:27.303 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 3 to smithi16030472-454 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1479: finishing copy_from racing read to smithi16030472-10 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1501: got (16) Device or resource busy 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1498: got (2) No such file or directory 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1483: finishing write tid 1 to smithi16030472-70 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1488: finishing copy_from racing read to smithi16030472-433 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1499: got (0) Success 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 233 v 0 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1483: finishing write tid 2 to smithi16030472-70 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1483: finishing write tid 3 to smithi16030472-70 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1502: got (0) Success 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 0 (ObjNum 49 snap 0 seq_num 3111024432) clean dne 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1488: finishing copy_from to smithi16030472-433 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 361 (ObjNum 476 snap 0 seq_num 476) dirty exists 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1495: finishing copy_from to smithi16030472-177 2024-09-21T18:08:27.304 INFO:tasks.rados.rados.0.smithi160.stdout:1495: got expected ENOENT (src dne) 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 4 to smithi16030472-454 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 5 to smithi16030472-454 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1495: finishing copy_from racing read to smithi16030472-177 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: finishing write tid 6 to smithi16030472-454 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: oid 454 updating version 0 to 568 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: oid 454 version 568 is already newer than 565 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: oid 454 updating version 568 to 569 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: oid 454 version 569 is already newer than 567 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: oid 454 version 569 is already newer than 566 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 454 v 569 (ObjNum 823 snap 0 seq_num 823) dirty exists 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1491: left oid 454 (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1479: finishing copy_from to smithi16030472-10 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 10 v 339 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1479: done (13 left) 2024-09-21T18:08:27.305 INFO:tasks.rados.rados.0.smithi160.stdout:1488: done (12 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1491: done (11 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1493: done (10 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1495: done (9 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1498: done (8 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1499: done (7 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1500: done (6 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1501: done (5 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1502: done (4 left) 2024-09-21T18:08:27.306 INFO:tasks.rados.rados.0.smithi160.stdout:1503: delete oid 386 current snap is 0 2024-09-21T18:08:27.307 INFO:tasks.rados.rados.0.smithi160.stdout:1496: finishing write tid 1 to smithi16030472-303 2024-09-21T18:08:27.307 INFO:tasks.rados.rados.0.smithi160.stdout:1496: finishing write tid 2 to smithi16030472-303 2024-09-21T18:08:27.307 INFO:tasks.rados.rados.0.smithi160.stdout:1483: finishing write tid 4 to smithi16030472-70 2024-09-21T18:08:27.307 INFO:tasks.rados.rados.0.smithi160.stdout:1483: finishing write tid 5 to smithi16030472-70 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:1483: oid 70 updating version 0 to 498 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:1483: oid 70 updating version 498 to 501 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:1483: oid 70 version 501 is already newer than 499 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:1483: oid 70 version 501 is already newer than 500 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 501 (ObjNum 821 snap 0 seq_num 821) dirty exists 2024-09-21T18:08:27.308 INFO:tasks.rados.rados.0.smithi160.stdout:1483: left oid 70 (ObjNum 821 snap 0 seq_num 821) 2024-09-21T18:08:27.314 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.314 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.329 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:27.383 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:27.380+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:27.408 INFO:tasks.rados.rados.0.smithi160.stdout:1486: finishing copy_from to smithi16030472-166 2024-09-21T18:08:27.408 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 143 (ObjNum 189 snap 0 seq_num 189) dirty exists 2024-09-21T18:08:27.408 INFO:tasks.rados.rados.0.smithi160.stdout:1496: finishing write tid 3 to smithi16030472-303 2024-09-21T18:08:27.408 INFO:tasks.rados.rados.0.smithi160.stdout:1496: finishing write tid 4 to smithi16030472-303 2024-09-21T18:08:27.410 INFO:tasks.rados.rados.0.smithi160.stdout:1497: finishing write tid 1 to smithi16030472-115 2024-09-21T18:08:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:1483: done (4 left) 2024-09-21T18:08:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:1486: done (3 left) 2024-09-21T18:08:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:1503: done (2 left) 2024-09-21T18:08:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:1504: write oid 76 current snap is 0 2024-09-21T18:08:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:1504: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-21T18:08:27.453 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.454 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.463 INFO:tasks.rados.rados.0.smithi160.stdout:1504: writing smithi16030472-76 from 699389 to 1496436 tid 1 2024-09-21T18:08:27.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:27.481 INFO:tasks.rados.rados.0.smithi160.stdout:1504: writing smithi16030472-76 from 2111079 to 2614032 tid 2 2024-09-21T18:08:27.483 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:27.477+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:27.496 INFO:tasks.rados.rados.0.smithi160.stdout:1504: writing smithi16030472-76 from 3094462 to 3676299 tid 3 2024-09-21T18:08:27.496 INFO:tasks.rados.rados.0.smithi160.stdout:1505: read oid 385 snap -1 2024-09-21T18:08:27.496 INFO:tasks.rados.rados.0.smithi160.stdout:1505: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:27.496 INFO:tasks.rados.rados.0.smithi160.stdout:1506: delete oid 190 current snap is 0 2024-09-21T18:08:27.496 INFO:tasks.rados.rados.0.smithi160.stdout:1497: finishing write tid 2 to smithi16030472-115 2024-09-21T18:08:27.497 INFO:tasks.rados.rados.0.smithi160.stdout:1497: finishing write tid 3 to smithi16030472-115 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: finishing write tid 5 to smithi16030472-303 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: oid 303 updating version 0 to 626 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: oid 303 updating version 626 to 627 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: oid 303 updating version 627 to 628 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: oid 303 updating version 628 to 629 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 303 v 629 (ObjNum 824 snap 0 seq_num 824) dirty exists 2024-09-21T18:08:27.509 INFO:tasks.rados.rados.0.smithi160.stdout:1496: left oid 303 (ObjNum 824 snap 0 seq_num 824) 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: finishing write tid 4 to smithi16030472-115 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: finishing write tid 5 to smithi16030472-115 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: oid 115 updating version 0 to 518 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: oid 115 updating version 518 to 519 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: oid 115 updating version 519 to 520 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:1497: oid 115 version 520 is already newer than 502 2024-09-21T18:08:27.546 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 115 v 520 (ObjNum 825 snap 0 seq_num 825) dirty exists 2024-09-21T18:08:27.547 INFO:tasks.rados.rados.0.smithi160.stdout:1497: left oid 115 (ObjNum 825 snap 0 seq_num 825) 2024-09-21T18:08:27.585 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:27.579+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:27.636 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.636 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.651 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:27.699 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:27.696+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:27.749 INFO:tasks.rados.rados.0.smithi160.stdout:1504: finishing write tid 1 to smithi16030472-76 2024-09-21T18:08:27.750 INFO:tasks.rados.rados.0.smithi160.stdout:1504: finishing write tid 2 to smithi16030472-76 2024-09-21T18:08:27.761 INFO:tasks.rados.rados.0.smithi160.stdout:1505: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:27.769 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 501679 2024-09-21T18:08:27.779 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.779 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.787 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:27.781+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:27.794 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:27.847 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 38065 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: finishing write tid 3 to smithi16030472-76 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: finishing write tid 4 to smithi16030472-76 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: finishing write tid 5 to smithi16030472-76 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: oid 76 updating version 0 to 521 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: oid 76 updating version 521 to 522 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: oid 76 updating version 522 to 524 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:1504: oid 76 version 524 is already newer than 523 2024-09-21T18:08:27.874 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 524 (ObjNum 827 snap 0 seq_num 827) dirty exists 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1504: left oid 76 (ObjNum 827 snap 0 seq_num 827) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1496: done (4 left) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1497: done (3 left) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1504: done (2 left) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1505: done (1 left) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1506: done (0 left) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1507: cache_evict oid 31 snap -1 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1508: copy_from oid 295 from oid 153 current snap is 0 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1509: cache_flush oid 30 snap -1 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1510: cache_flush oid 434 snap -1 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1511: read oid 143 snap -1 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1511: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1512: cache_evict oid 227 snap -1 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1513: write oid 99 current snap is 0 2024-09-21T18:08:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:1513: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-21T18:08:27.888 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:27.883+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:27.895 INFO:tasks.rados.rados.0.smithi160.stdout:1513: writing smithi16030472-99 from 733003 to 1396901 tid 1 2024-09-21T18:08:27.911 INFO:tasks.rados.rados.0.smithi160.stdout:1513: writing smithi16030472-99 from 1944396 to 2451018 tid 2 2024-09-21T18:08:27.915 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:08:27.925 INFO:tasks.rados.rados.0.smithi160.stdout:1513: writing smithi16030472-99 from 2865661 to 3470042 tid 3 2024-09-21T18:08:27.926 INFO:tasks.rados.rados.0.smithi160.stdout:1514: delete oid 69 current snap is 0 2024-09-21T18:08:27.926 INFO:tasks.rados.rados.0.smithi160.stdout:1509: got (0) Success 2024-09-21T18:08:27.926 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 30 v 0 (ObjNum 65 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:08:27.926 INFO:tasks.rados.rados.0.smithi160.stdout:1510: got (2) No such file or directory 2024-09-21T18:08:27.926 INFO:tasks.rados.rados.0.smithi160.stdout:1512: got (0) Success 2024-09-21T18:08:27.963 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:27.963 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:27.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:27.978 INFO:tasks.rados.rados.0.smithi160.stdout:1507: got (0) Success 2024-09-21T18:08:28.023 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:28.020+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:28.058 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:28.057+0000 7f3ce267c640 -1 osd.6 77 handle_osd_map injecting map crc failure 2024-09-21T18:08:28.079 INFO:tasks.rados.rados.0.smithi160.stdout:1507: done (7 left) 2024-09-21T18:08:28.079 INFO:tasks.rados.rados.0.smithi160.stdout:1509: done (6 left) 2024-09-21T18:08:28.079 INFO:tasks.rados.rados.0.smithi160.stdout:1510: done (5 left) 2024-09-21T18:08:28.079 INFO:tasks.rados.rados.0.smithi160.stdout:1512: done (4 left) 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1514: done (3 left) 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1515: read oid 169 snap -1 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1515: expect (ObjNum 168 snap 0 seq_num 168) 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1516: cache_flush oid 382 snap -1 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1517: write (excl) oid 328 current snap is 0 2024-09-21T18:08:28.080 INFO:tasks.rados.rados.0.smithi160.stdout:1517: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-21T18:08:28.089 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:28.084+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:28.094 INFO:tasks.rados.rados.0.smithi160.stdout:1517: writing smithi16030472-328 from 766617 to 1181013 tid 1 2024-09-21T18:08:28.105 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:28.105 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:28.109 INFO:tasks.rados.rados.0.smithi160.stdout:1517: writing smithi16030472-328 from 1945007 to 2455298 tid 2 2024-09-21T18:08:28.120 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:28.126 INFO:tasks.rados.rados.0.smithi160.stdout:1517: writing smithi16030472-328 from 3204154 to 3831079 tid 3 2024-09-21T18:08:28.126 INFO:tasks.rados.rados.0.smithi160.stdout:1518: cache_try_flush oid 242 snap -1 2024-09-21T18:08:28.126 INFO:tasks.rados.rados.0.smithi160.stdout:1519: write (excl) oid 36 current snap is 0 2024-09-21T18:08:28.127 INFO:tasks.rados.rados.0.smithi160.stdout:1519: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-21T18:08:28.139 INFO:tasks.rados.rados.0.smithi160.stdout:1519: writing smithi16030472-36 from 783424 to 1273069 tid 1 2024-09-21T18:08:28.151 INFO:tasks.rados.rados.0.smithi160.stdout:1519: writing smithi16030472-36 from 2003489 to 2657438 tid 2 2024-09-21T18:08:28.160 INFO:tasks.rados.rados.0.smithi160.stdout:1519: writing smithi16030472-36 from 3231577 to 3786127 tid 3 2024-09-21T18:08:28.160 INFO:tasks.rados.rados.0.smithi160.stdout:1520: read oid 464 snap -1 2024-09-21T18:08:28.160 INFO:tasks.rados.rados.0.smithi160.stdout:1520: expect (ObjNum 675 snap 0 seq_num 675) 2024-09-21T18:08:28.160 INFO:tasks.rados.rados.0.smithi160.stdout:1521: write (excl) oid 498 current snap is 0 2024-09-21T18:08:28.160 INFO:tasks.rados.rados.0.smithi160.stdout:1521: seq_num 833 ranges {400231=564894,1661971=338260} 2024-09-21T18:08:28.166 INFO:tasks.rados.rados.0.smithi160.stdout:1521: writing smithi16030472-498 from 400231 to 965125 tid 1 2024-09-21T18:08:28.174 INFO:tasks.rados.rados.0.smithi160.stdout:1521: writing smithi16030472-498 from 1661971 to 2000231 tid 2 2024-09-21T18:08:28.174 INFO:tasks.rados.rados.0.smithi160.stdout:1522: cache_try_flush oid 43 snap -1 2024-09-21T18:08:28.174 INFO:tasks.rados.rados.0.smithi160.stdout:1523: write (excl) oid 51 current snap is 0 2024-09-21T18:08:28.174 INFO:tasks.rados.rados.0.smithi160.stdout:1523: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-21T18:08:28.180 INFO:tasks.rados.rados.0.smithi160.stdout:1523: writing smithi16030472-51 from 417038 to 1057181 tid 1 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1523: writing smithi16030472-51 from 1720453 to 2017038 tid 2 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1524: read oid 308 snap -1 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1524: expect (ObjNum 567 snap 0 seq_num 567) 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1525: copy_from oid 103 from oid 485 current snap is 0 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1526: cache_flush oid 260 snap -1 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1527: write oid 39 current snap is 0 2024-09-21T18:08:28.187 INFO:tasks.rados.rados.0.smithi160.stdout:1527: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-21T18:08:28.190 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:28.189+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:28.193 INFO:tasks.rados.rados.0.smithi160.stdout:1527: writing smithi16030472-39 from 433845 to 1149237 tid 1 2024-09-21T18:08:28.198 INFO:tasks.rados.rados.0.smithi160.stdout:1527: writing smithi16030472-39 from 1662582 to 2033845 tid 2 2024-09-21T18:08:28.198 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:28.201 INFO:tasks.rados.rados.0.smithi160.stdout:1508: finishing copy_from racing read to smithi16030472-295 2024-09-21T18:08:28.291 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:28.291 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:28.291 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:28.289+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:28.306 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:28.331 INFO:tasks.rados.rados.0.smithi160.stdout:1508: finishing copy_from to smithi16030472-295 2024-09-21T18:08:28.331 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 94 (ObjNum 152 snap 0 seq_num 152) dirty exists 2024-09-21T18:08:28.331 INFO:tasks.rados.rados.0.smithi160.stdout:1508: done (15 left) 2024-09-21T18:08:28.331 INFO:tasks.rados.rados.0.smithi160.stdout:1528: delete oid 262 current snap is 0 2024-09-21T18:08:28.388 INFO:tasks.rados.rados.0.smithi160.stdout:1511: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:08:28.388 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:28.392 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:28.389+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:28.433 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:28.433 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:28.450 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:28.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:28.490 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:28.503 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:28.499+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:1516: got (2) No such file or directory 2024-09-21T18:08:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:1515: expect (ObjNum 168 snap 0 seq_num 168) 2024-09-21T18:08:28.545 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:28.575 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:28.586 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1871573 2024-09-21T18:08:28.589 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:28.593 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:28.587+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:28.597 INFO:tasks.rados.rados.0.smithi160.stdout:1518: got (0) Success 2024-09-21T18:08:28.597 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 242 v 0 (ObjNum 129 snap 0 seq_num 3126304760) clean dne 2024-09-21T18:08:28.597 INFO:tasks.rados.rados.0.smithi160.stdout:1523: finishing write tid 1 to smithi16030472-51 2024-09-21T18:08:28.597 INFO:tasks.rados.rados.0.smithi160.stdout:1523: finishing write tid 2 to smithi16030472-51 2024-09-21T18:08:28.597 INFO:tasks.rados.rados.0.smithi160.stdout:1524: expect (ObjNum 567 snap 0 seq_num 567) 2024-09-21T18:08:28.603 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1178844 2024-09-21T18:08:28.614 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:28.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:28.635 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:28.655 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:28.676 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:28.676 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:28.694 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:28.688+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:28.696 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:28.696 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:28.709 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:28.709 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:28.718 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:28.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:28.723 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:28.740 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:28.740 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:28.740 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1513: finishing write tid 1 to smithi16030472-99 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1513: finishing write tid 2 to smithi16030472-99 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1511: done (15 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1515: done (14 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1516: done (13 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1518: done (12 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1524: done (11 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1528: done (10 left) 2024-09-21T18:08:28.744 INFO:tasks.rados.rados.0.smithi160.stdout:1529: cache_try_flush oid 12 snap -1 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: finishing write tid 3 to smithi16030472-51 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: finishing write tid 4 to smithi16030472-51 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: oid 51 updating version 0 to 526 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: oid 51 version 526 is already newer than 525 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: oid 51 updating version 526 to 527 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 527 (ObjNum 834 snap 0 seq_num 834) dirty exists 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1523: left oid 51 (ObjNum 834 snap 0 seq_num 834) 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:1520: expect (ObjNum 675 snap 0 seq_num 675) 2024-09-21T18:08:28.745 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:28.759 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:28.760 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:28.781 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:28.795 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:28.789+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:28.803 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:28.826 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:28.826 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:28.846 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:28.855 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:28.855 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:28.855 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:28.855 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:28.869 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:28.869 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1517: finishing write tid 1 to smithi16030472-328 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1517: finishing write tid 2 to smithi16030472-328 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: finishing write tid 3 to smithi16030472-99 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: finishing write tid 4 to smithi16030472-99 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: finishing write tid 5 to smithi16030472-99 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: oid 99 updating version 0 to 612 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: oid 99 version 612 is already newer than 611 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: oid 99 version 612 is already newer than 610 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: oid 99 updating version 612 to 613 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 613 (ObjNum 829 snap 0 seq_num 829) dirty exists 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1513: left oid 99 (ObjNum 829 snap 0 seq_num 829) 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1521: finishing write tid 1 to smithi16030472-498 2024-09-21T18:08:28.880 INFO:tasks.rados.rados.0.smithi160.stdout:1521: finishing write tid 2 to smithi16030472-498 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: finishing write tid 3 to smithi16030472-498 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: finishing write tid 4 to smithi16030472-498 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: oid 498 updating version 0 to 614 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: oid 498 updating version 614 to 616 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: oid 498 version 616 is already newer than 615 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 498 v 616 (ObjNum 833 snap 0 seq_num 833) dirty exists 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1521: left oid 498 (ObjNum 833 snap 0 seq_num 833) 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: finishing write tid 3 to smithi16030472-328 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: finishing write tid 4 to smithi16030472-328 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1527: finishing write tid 1 to smithi16030472-39 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: finishing write tid 5 to smithi16030472-328 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: oid 328 updating version 0 to 558 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: oid 328 updating version 558 to 560 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: oid 328 updating version 560 to 561 2024-09-21T18:08:28.881 INFO:tasks.rados.rados.0.smithi160.stdout:1517: oid 328 version 561 is already newer than 559 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 561 (ObjNum 831 snap 0 seq_num 831) dirty exists 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1517: left oid 328 (ObjNum 831 snap 0 seq_num 831) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1513: done (10 left) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1517: done (9 left) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1520: done (8 left) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1521: done (7 left) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1523: done (6 left) 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1530: cache_flush oid 223 snap -1 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1531: cache_flush oid 416 snap -1 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1532: cache_try_flush oid 162 snap -1 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1533: cache_try_flush oid 85 snap -1 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1534: write oid 412 current snap is 0 2024-09-21T18:08:28.882 INFO:tasks.rados.rados.0.smithi160.stdout:1534: seq_num 837 ranges {467459=582243,1495899=404945} 2024-09-21T18:08:28.891 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:28.896 INFO:tasks.rados.rados.0.smithi160.stdout:1534: writing smithi16030472-412 from 467459 to 1049702 tid 1 2024-09-21T18:08:28.896 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:28.889+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:1534: writing smithi16030472-412 from 1495899 to 1900844 tid 2 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:1535: cache_evict oid 35 snap -1 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:1530: got (0) Success 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 223 v 0 (ObjNum 209 snap 0 seq_num 1587258860) clean dne 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:1533: got (0) Success 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 0 (ObjNum 84 snap 0 seq_num 84) dirty exists 2024-09-21T18:08:28.911 INFO:tasks.rados.rados.0.smithi160.stdout:1526: got (2) No such file or directory 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1519: finishing write tid 1 to smithi16030472-36 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1519: finishing write tid 2 to smithi16030472-36 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1519: finishing write tid 3 to smithi16030472-36 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1525: finishing copy_from to smithi16030472-103 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1525: got expected ENOENT (src dne) 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1525: finishing copy_from racing read to smithi16030472-103 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1525: done (11 left) 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1526: done (10 left) 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1530: done (9 left) 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1533: done (8 left) 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1536: cache_flush oid 52 snap -1 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1537: write (excl) oid 495 current snap is 0 2024-09-21T18:08:28.912 INFO:tasks.rados.rados.0.smithi160.stdout:1537: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-21T18:08:28.913 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:28.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:28.929 INFO:tasks.rados.rados.0.smithi160.stdout:1537: writing smithi16030472-495 from 484266 to 1141758 tid 1 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1537: writing smithi16030472-495 from 1554381 to 2084266 tid 2 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1538: read oid 474 snap -1 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1538: expect (ObjNum 473 snap 0 seq_num 473) 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1539: cache_evict oid 131 snap -1 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1537: finishing write tid 1 to smithi16030472-495 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1540: read oid 428 snap -1 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1540: expect (ObjNum 427 snap 0 seq_num 427) 2024-09-21T18:08:28.944 INFO:tasks.rados.rados.0.smithi160.stdout:1541: read oid 413 snap -1 2024-09-21T18:08:28.945 INFO:tasks.rados.rados.0.smithi160.stdout:1541: expect (ObjNum 412 snap 0 seq_num 412) 2024-09-21T18:08:28.945 INFO:tasks.rados.rados.0.smithi160.stdout:1542: cache_evict oid 479 snap -1 2024-09-21T18:08:28.945 INFO:tasks.rados.rados.0.smithi160.stdout:1543: delete oid 106 current snap is 0 2024-09-21T18:08:28.967 INFO:tasks.rados.rados.0.smithi160.stdout:1538: expect (ObjNum 473 snap 0 seq_num 473) 2024-09-21T18:08:28.977 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1388572 2024-09-21T18:08:28.992 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:28.992 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:28.992 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:28.992 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:28.993 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:28.997 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:28.991+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:29.007 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:29.040 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.088 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.098 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:29.092+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1527: finishing write tid 2 to smithi16030472-39 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1527: finishing write tid 3 to smithi16030472-39 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1542: got (16) Device or resource busy 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: finishing write tid 4 to smithi16030472-36 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: finishing write tid 5 to smithi16030472-36 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: oid 36 updating version 0 to 572 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: oid 36 version 572 is already newer than 571 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: oid 36 version 572 is already newer than 570 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: oid 36 updating version 572 to 573 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 573 (ObjNum 832 snap 0 seq_num 832) dirty exists 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1519: left oid 36 (ObjNum 832 snap 0 seq_num 832) 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1535: got (16) Device or resource busy 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1527: finishing write tid 4 to smithi16030472-39 2024-09-21T18:08:29.134 INFO:tasks.rados.rados.0.smithi160.stdout:1527: oid 39 updating version 0 to 617 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:1527: oid 39 updating version 617 to 619 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:1527: oid 39 version 619 is already newer than 618 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 619 (ObjNum 835 snap 0 seq_num 835) dirty exists 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:1527: left oid 39 (ObjNum 835 snap 0 seq_num 835) 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:1532: got (0) Success 2024-09-21T18:08:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 162 v 0 (ObjNum 779 snap 0 seq_num 779) dirty exists 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:29.138 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 70.804600084, 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:08:29.142 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.964748087999993, 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:29.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:08:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.146 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.861027640000003, 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:29.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:08:29.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:08:29.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.860252896999995, 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:08:29.151 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:08:29.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.859604877999999, 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:08:29.156 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:08:29.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:08:29.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.845733182999993, 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:29.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:08:29.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.842792166999999, 2024-09-21T18:08:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.839775318999997, 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.170 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:08:29.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.174 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.837563736999996, 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.732226644999997, 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:08:29.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:08:29.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:08:29.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.040804270999999, 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:08:29.183 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:08:29.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.037774271000004, 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.188 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:08:29.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:08:29.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 59.071210309000001, 2024-09-21T18:08:29.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:08:29.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 57.404374873999998, 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:08:29.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:08:29.201 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.100883793000001, 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:08:29.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:08:29.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.098586760000003, 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:08:29.206 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:08:29.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.096536311999998, 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:08:29.210 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:29.214 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.094380950999998, 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.218 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.093618192999998, 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:29.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:08:29.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 49.991713148000002, 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:08:29.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:08:29.226 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.227 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.227 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.227 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:29.227 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:29.227 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.228 INFO:tasks.rados.rados.0.smithi160.stdout:1522: got (0) Success 2024-09-21T18:08:29.228 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:08:29.229 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:29.229 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:29.225+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:29.234 INFO:tasks.rados.rados.0.smithi160.stdout:1534: finishing write tid 1 to smithi16030472-412 2024-09-21T18:08:29.234 INFO:tasks.rados.rados.0.smithi160.stdout:1534: finishing write tid 2 to smithi16030472-412 2024-09-21T18:08:29.235 INFO:tasks.rados.rados.0.smithi160.stdout:1536: got (0) Success 2024-09-21T18:08:29.235 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 52 v 0 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-21T18:08:29.235 INFO:tasks.rados.rados.0.smithi160.stdout:1539: got (2) No such file or directory 2024-09-21T18:08:29.252 INFO:tasks.rados.rados.0.smithi160.stdout:1537: finishing write tid 2 to smithi16030472-495 2024-09-21T18:08:29.252 INFO:tasks.rados.rados.0.smithi160.stdout:1537: finishing write tid 3 to smithi16030472-495 2024-09-21T18:08:29.254 INFO:tasks.rados.rados.0.smithi160.stdout:1540: expect (ObjNum 427 snap 0 seq_num 427) 2024-09-21T18:08:29.329 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:29.323+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1529: got (0) Success 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 12 v 0 (ObjNum 772 snap 0 seq_num 772) dirty exists 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1537: finishing write tid 4 to smithi16030472-495 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1537: oid 495 updating version 0 to 530 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1537: oid 495 version 530 is already newer than 528 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1537: oid 495 version 530 is already newer than 529 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 530 (ObjNum 838 snap 0 seq_num 838) dirty exists 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1537: left oid 495 (ObjNum 838 snap 0 seq_num 838) 2024-09-21T18:08:29.395 INFO:tasks.rados.rados.0.smithi160.stdout:1541: expect (ObjNum 412 snap 0 seq_num 412) 2024-09-21T18:08:29.396 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.396 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:29.396 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:29.396 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.402 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2207717 2024-09-21T18:08:29.410 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:29.416 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.430 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:29.425+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:29.447 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: finishing write tid 3 to smithi16030472-412 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: finishing write tid 4 to smithi16030472-412 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: oid 412 updating version 0 to 621 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: oid 412 updating version 621 to 622 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: oid 412 version 622 is already newer than 620 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 622 (ObjNum 837 snap 0 seq_num 837) dirty exists 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1534: left oid 412 (ObjNum 837 snap 0 seq_num 837) 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:1531: got (0) Success 2024-09-21T18:08:29.479 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 0 (ObjNum 767 snap 0 seq_num 767) dirty exists 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1519: done (15 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1522: done (14 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1527: done (13 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1529: done (12 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1531: done (11 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1532: done (10 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1534: done (9 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1535: done (8 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1536: done (7 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1537: done (6 left) 2024-09-21T18:08:29.523 INFO:tasks.rados.rados.0.smithi160.stdout:1538: done (5 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1539: done (4 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1540: done (3 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1541: done (2 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1542: done (1 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1543: done (0 left) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1544: cache_evict oid 332 snap -1 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1545: read oid 417 snap -1 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1545: expect (ObjNum 502 snap 0 seq_num 502) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1546: read oid 5 snap -1 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1546: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:08:29.524 INFO:tasks.rados.rados.0.smithi160.stdout:1547: delete oid 98 current snap is 0 2024-09-21T18:08:29.531 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:29.529+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:29.533 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.534 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:29.534 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:29.534 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:29.534 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:29.623 INFO:tasks.rados.rados.0.smithi160.stdout:1544: got (16) Device or resource busy 2024-09-21T18:08:29.625 INFO:tasks.rados.rados.0.smithi160.stdout:1546: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:08:29.625 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.632 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:29.627+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:29.710 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.710 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:29.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:29.710 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:08:29.710 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.716 INFO:tasks.rados.rados.0.smithi160.stdout:1545: expect (ObjNum 502 snap 0 seq_num 502) 2024-09-21T18:08:29.716 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:29.726 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:29.732 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:29.728+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1544: done (3 left) 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1545: done (2 left) 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1546: done (1 left) 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1547: done (0 left) 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1548: cache_try_flush oid 30 snap -1 2024-09-21T18:08:29.820 INFO:tasks.rados.rados.0.smithi160.stdout:1549: write oid 40 current snap is 0 2024-09-21T18:08:29.821 INFO:tasks.rados.rados.0.smithi160.stdout:1549: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-21T18:08:29.833 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:29.825+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:29.836 INFO:tasks.rados.rados.0.smithi160.stdout:1549: writing smithi16030472-40 from 534687 to 1017926 tid 1 2024-09-21T18:08:29.847 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:29.847 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.853 INFO:tasks.rados.rados.0.smithi160.stdout:1549: writing smithi16030472-40 from 1613474 to 2025757 tid 2 2024-09-21T18:08:29.853 INFO:tasks.rados.rados.0.smithi160.stdout:1550: write oid 397 current snap is 0 2024-09-21T18:08:29.853 INFO:tasks.rados.rados.0.smithi160.stdout:1550: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-21T18:08:29.862 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:29.871 INFO:tasks.rados.rados.0.smithi160.stdout:1550: writing smithi16030472-397 from 551494 to 1109982 tid 1 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1550: writing smithi16030472-397 from 1671956 to 2151494 tid 2 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1551: copy_from oid 153 from oid 37 current snap is 0 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1552: read oid 421 snap -1 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1552: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1553: write oid 56 current snap is 0 2024-09-21T18:08:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:1553: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-21T18:08:29.896 INFO:tasks.rados.rados.0.smithi160.stdout:1553: writing smithi16030472-56 from 568301 to 1202038 tid 1 2024-09-21T18:08:29.903 INFO:tasks.rados.rados.0.smithi160.stdout:1553: writing smithi16030472-56 from 1614085 to 2030037 tid 2 2024-09-21T18:08:29.904 INFO:tasks.rados.rados.0.smithi160.stdout:1554: copy_from oid 260 from oid 203 current snap is 0 2024-09-21T18:08:29.904 INFO:tasks.rados.rados.0.smithi160.stdout:1555: write (excl) oid 157 current snap is 0 2024-09-21T18:08:29.904 INFO:tasks.rados.rados.0.smithi160.stdout:1555: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-21T18:08:29.911 INFO:tasks.rados.rados.0.smithi160.stdout:1555: writing smithi16030472-157 from 585108 to 1010447 tid 1 2024-09-21T18:08:29.920 INFO:tasks.rados.rados.0.smithi160.stdout:1555: writing smithi16030472-157 from 1788920 to 2185108 tid 2 2024-09-21T18:08:29.920 INFO:tasks.rados.rados.0.smithi160.stdout:1556: write oid 193 current snap is 0 2024-09-21T18:08:29.920 INFO:tasks.rados.rados.0.smithi160.stdout:1556: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-21T18:08:29.927 INFO:tasks.rados.rados.0.smithi160.stdout:1556: writing smithi16030472-193 from 601915 to 1102503 tid 1 2024-09-21T18:08:29.934 INFO:tasks.rados.rados.0.smithi160.stdout:1556: writing smithi16030472-193 from 1847402 to 2201915 tid 2 2024-09-21T18:08:29.934 INFO:tasks.rados.rados.0.smithi160.stdout:1557: write oid 32 current snap is 0 2024-09-21T18:08:29.934 INFO:tasks.rados.rados.0.smithi160.stdout:1557: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-21T18:08:29.934 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:29.931+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:29.941 INFO:tasks.rados.rados.0.smithi160.stdout:1557: writing smithi16030472-32 from 618722 to 1194559 tid 1 2024-09-21T18:08:29.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:08:29.947 INFO:tasks.rados.rados.0.smithi160.stdout:1557: writing smithi16030472-32 from 1905884 to 2218722 tid 2 2024-09-21T18:08:29.947 INFO:tasks.rados.rados.0.smithi160.stdout:1558: cache_evict oid 208 snap -1 2024-09-21T18:08:29.947 INFO:tasks.rados.rados.0.smithi160.stdout:1559: write oid 333 current snap is 0 2024-09-21T18:08:29.947 INFO:tasks.rados.rados.0.smithi160.stdout:1559: seq_num 847 ranges {635529=651086,1848013=387516} 2024-09-21T18:08:29.954 INFO:tasks.rados.rados.0.smithi160.stdout:1559: writing smithi16030472-333 from 635529 to 1286615 tid 1 2024-09-21T18:08:29.959 INFO:tasks.rados.rados.0.smithi160.stdout:1559: writing smithi16030472-333 from 1848013 to 2235529 tid 2 2024-09-21T18:08:29.959 INFO:tasks.rados.rados.0.smithi160.stdout:1560: cache_flush oid 375 snap -1 2024-09-21T18:08:29.960 INFO:tasks.rados.rados.0.smithi160.stdout:1561: write (excl) oid 449 current snap is 0 2024-09-21T18:08:29.960 INFO:tasks.rados.rados.0.smithi160.stdout:1561: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-21T18:08:29.967 INFO:tasks.rados.rados.0.smithi160.stdout:1561: writing smithi16030472-449 from 652336 to 1378671 tid 1 2024-09-21T18:08:29.972 INFO:tasks.rados.rados.0.smithi160.stdout:1561: writing smithi16030472-449 from 1906495 to 2252336 tid 2 2024-09-21T18:08:29.972 INFO:tasks.rados.rados.0.smithi160.stdout:1562: write (excl) oid 235 current snap is 0 2024-09-21T18:08:29.972 INFO:tasks.rados.rados.0.smithi160.stdout:1562: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-21T18:08:29.978 INFO:tasks.rados.rados.0.smithi160.stdout:1562: writing smithi16030472-235 from 669143 to 1070727 tid 1 2024-09-21T18:08:29.979 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:29.979 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:29.979 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:29.980 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:29.980 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:29.983 INFO:tasks.rados.rados.0.smithi160.stdout:1562: writing smithi16030472-235 from 1564977 to 1991936 tid 2 2024-09-21T18:08:29.983 INFO:tasks.rados.rados.0.smithi160.stdout:1563: write oid 82 current snap is 0 2024-09-21T18:08:29.984 INFO:tasks.rados.rados.0.smithi160.stdout:1563: seq_num 850 ranges {685950=476833,1623459=570617} 2024-09-21T18:08:29.990 INFO:tasks.rados.rados.0.smithi160.stdout:1563: writing smithi16030472-82 from 685950 to 1162783 tid 1 2024-09-21T18:08:29.994 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1563: writing smithi16030472-82 from 1623459 to 2194076 tid 2 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: finishing write tid 1 to smithi16030472-40 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: finishing write tid 2 to smithi16030472-40 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: finishing write tid 3 to smithi16030472-40 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: finishing write tid 4 to smithi16030472-40 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: oid 40 updating version 0 to 631 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: oid 40 updating version 631 to 633 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:1549: oid 40 version 633 is already newer than 632 2024-09-21T18:08:29.996 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 633 (ObjNum 841 snap 0 seq_num 841) dirty exists 2024-09-21T18:08:29.997 INFO:tasks.rados.rados.0.smithi160.stdout:1549: left oid 40 (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:08:29.997 INFO:tasks.rados.rados.0.smithi160.stdout:1549: done (15 left) 2024-09-21T18:08:29.997 INFO:tasks.rados.rados.0.smithi160.stdout:1564: read oid 307 snap -1 2024-09-21T18:08:29.997 INFO:tasks.rados.rados.0.smithi160.stdout:1564: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:29.997 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.036 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:30.032+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:30.101 INFO:tasks.rados.rados.0.smithi160.stdout:1564: expect (ObjNum 646 snap 0 seq_num 646) 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 58.518671452, 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.117 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:08:30.121 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 36.088701505000003, 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:08:30.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:08:30.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:08:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.991181130000001, 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:08:30.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:08:30.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:08:30.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:30.130 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.990496880000002, 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:30.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:08:30.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.134 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.989663524999997, 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:30.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:08:30.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.138 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.986944895000001, 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:30.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:08:30.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:08:30.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:08:30.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.891136488999997, 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.888831948000004, 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.147 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:08:30.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:08:30.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:08:30.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2900 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'245, mlcod=59'234)", 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 35.788551661, 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22363195199999999, 2024-09-21T18:08:30.151 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325923+0000", 2024-09-21T18:08:30.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2929999999999999e-06 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325935+0000", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1763000000000001e-05 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325952+0000", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6722999999999999e-05 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325965+0000", 2024-09-21T18:08:30.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3743e-05 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545223+0000", 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21925772700000001 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545279+0000", 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6085999999999998e-05 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549487+0000", 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004207868 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549553+0000", 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5748999999999997e-05 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.911203327999999, 2024-09-21T18:08:30.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:08:30.159 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.810047646000001, 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:08:30.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:08:30.163 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.809251983999999, 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.808116164000001, 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:08:30.167 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:08:30.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.787398571000001, 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:08:30.171 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:08:30.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.778045887000001, 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:08:30.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.772671239000001, 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.179 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:08:30.183 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 16.770122963999999, 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:30.187 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 10.542422427, 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 10.541756134, 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:08:30.191 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:08:30.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 10.541664021999999, 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.195 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:08:30.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.199 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:30.200 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:30.201 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:30.201 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:30.197+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:30.235 INFO:tasks.rados.rados.0.smithi160.stdout:1555: finishing write tid 1 to smithi16030472-157 2024-09-21T18:08:30.236 INFO:tasks.rados.rados.0.smithi160.stdout:1564: done (15 left) 2024-09-21T18:08:30.236 INFO:tasks.rados.rados.0.smithi160.stdout:1565: write oid 148 current snap is 0 2024-09-21T18:08:30.236 INFO:tasks.rados.rados.0.smithi160.stdout:1565: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-21T18:08:30.242 INFO:tasks.rados.rados.0.smithi160.stdout:1565: writing smithi16030472-148 from 702757 to 1254839 tid 1 2024-09-21T18:08:30.247 INFO:tasks.rados.rados.0.smithi160.stdout:1565: writing smithi16030472-148 from 1681941 to 2112569 tid 2 2024-09-21T18:08:30.247 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.249 INFO:tasks.rados.rados.0.smithi160.stdout:1557: finishing write tid 1 to smithi16030472-32 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: finishing write tid 2 to smithi16030472-32 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: finishing write tid 3 to smithi16030472-32 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: finishing write tid 4 to smithi16030472-32 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: oid 32 updating version 0 to 625 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: oid 32 updating version 625 to 627 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: oid 32 version 627 is already newer than 626 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 32 v 627 (ObjNum 846 snap 0 seq_num 846) dirty exists 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: left oid 32 (ObjNum 846 snap 0 seq_num 846) 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1557: done (15 left) 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1566: read oid 377 snap -1 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout:1566: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:08:30.250 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout:1548: got (0) Success 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 30 v 0 (ObjNum 65 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout:1548: done (15 left) 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout:1567: read oid 222 snap -1 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout:1567: expect (ObjNum 221 snap 0 seq_num 221) 2024-09-21T18:08:30.293 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.297 INFO:tasks.rados.rados.0.smithi160.stdout:1552: expect (ObjNum 420 snap 0 seq_num 420) 2024-09-21T18:08:30.302 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:30.299+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:30.321 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:30.375 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:30.375 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:30.375 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:30.376 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1559: finishing write tid 1 to smithi16030472-333 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1559: finishing write tid 2 to smithi16030472-333 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1558: got (2) No such file or directory 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1560: got (2) No such file or directory 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1559: finishing write tid 3 to smithi16030472-333 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1559: finishing write tid 4 to smithi16030472-333 2024-09-21T18:08:30.389 INFO:tasks.rados.rados.0.smithi160.stdout:1559: oid 333 updating version 0 to 531 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1559: oid 333 updating version 531 to 532 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1559: oid 333 updating version 532 to 533 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 533 (ObjNum 847 snap 0 seq_num 847) dirty exists 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1559: left oid 333 (ObjNum 847 snap 0 seq_num 847) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1555: finishing write tid 2 to smithi16030472-157 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1550: finishing write tid 1 to smithi16030472-397 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1552: done (15 left) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1558: done (14 left) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1559: done (13 left) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1560: done (12 left) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1568: cache_evict oid 231 snap -1 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1569: read oid 281 snap -1 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1569: expect (ObjNum 774 snap 0 seq_num 774) 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1570: copy_from oid 456 from oid 359 current snap is 0 2024-09-21T18:08:30.390 INFO:tasks.rados.rados.0.smithi160.stdout:1571: cache_evict oid 107 snap -1 2024-09-21T18:08:30.391 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.391 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:30.402 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:30.395+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:30.456 INFO:tasks.rados.rados.0.smithi160.stdout:1566: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:08:30.456 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:30.503 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:30.497+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:30.510 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:30.510 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:30.510 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:30.510 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:30.512 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1909620 2024-09-21T18:08:30.525 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:30.527 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:30.563 INFO:tasks.rados.rados.0.smithi160.stdout:1550: finishing write tid 2 to smithi16030472-397 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1550: finishing write tid 3 to smithi16030472-397 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: finishing write tid 1 to smithi16030472-56 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1568: got (16) Device or resource busy 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1563: finishing write tid 1 to smithi16030472-82 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1563: finishing write tid 2 to smithi16030472-82 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: finishing write tid 2 to smithi16030472-56 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: finishing write tid 3 to smithi16030472-56 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: finishing write tid 4 to smithi16030472-56 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: oid 56 updating version 0 to 574 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: oid 56 version 574 is already newer than 573 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: oid 56 updating version 574 to 575 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 575 (ObjNum 843 snap 0 seq_num 843) dirty exists 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1553: left oid 56 (ObjNum 843 snap 0 seq_num 843) 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1556: finishing write tid 1 to smithi16030472-193 2024-09-21T18:08:30.564 INFO:tasks.rados.rados.0.smithi160.stdout:1556: finishing write tid 2 to smithi16030472-193 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: finishing write tid 3 to smithi16030472-193 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: finishing write tid 4 to smithi16030472-193 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: oid 193 updating version 0 to 575 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: oid 193 version 575 is already newer than 574 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: oid 193 updating version 575 to 576 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 193 v 576 (ObjNum 845 snap 0 seq_num 845) dirty exists 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: left oid 193 (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1553: done (15 left) 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1556: done (14 left) 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1566: done (13 left) 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1568: done (12 left) 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1572: cache_evict oid 169 snap -1 2024-09-21T18:08:30.565 INFO:tasks.rados.rados.0.smithi160.stdout:1573: delete oid 477 current snap is 0 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1550: finishing write tid 4 to smithi16030472-397 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1550: oid 397 updating version 0 to 628 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1550: oid 397 version 628 is already newer than 429 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1550: oid 397 updating version 628 to 629 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 397 v 629 (ObjNum 842 snap 0 seq_num 842) dirty exists 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1550: left oid 397 (ObjNum 842 snap 0 seq_num 842) 2024-09-21T18:08:30.566 INFO:tasks.rados.rados.0.smithi160.stdout:1572: got (2) No such file or directory 2024-09-21T18:08:30.584 INFO:tasks.rados.rados.0.smithi160.stdout:1570: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:08:30.588 INFO:tasks.rados.rados.0.smithi160.stdout:1570: finishing copy_from to smithi16030472-456 2024-09-21T18:08:30.588 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 436 (ObjNum 577 snap 0 seq_num 577) dirty exists 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: finishing write tid 3 to smithi16030472-157 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: finishing write tid 4 to smithi16030472-157 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: oid 157 updating version 0 to 593 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: oid 157 updating version 593 to 635 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: oid 157 version 635 is already newer than 634 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 635 (ObjNum 844 snap 0 seq_num 844) dirty exists 2024-09-21T18:08:30.595 INFO:tasks.rados.rados.0.smithi160.stdout:1555: left oid 157 (ObjNum 844 snap 0 seq_num 844) 2024-09-21T18:08:30.597 INFO:tasks.rados.rados.0.smithi160.stdout:1569: expect (ObjNum 774 snap 0 seq_num 774) 2024-09-21T18:08:30.598 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:30.604 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:30.601+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:236 3.1 e45/31 3:96516620:::smithi16030472-58:head v 45'30, mlcod=45'24)", 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 75.962295936999993, 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.202914602, 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.648 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684350+0000", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684354+0000", 2024-09-21T18:08:30.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2959999999999999e-06 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684626+0000", 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00027191299999999998 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684653+0000", 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7195999999999998e-05 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.650 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.684667+0000", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3767e-05 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787798+0000", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.103131111 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.787831+0000", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3229000000000003e-05 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887202+0000", 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099371371 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:14.887265+0000", 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2718999999999994e-05 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:08:30.652 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 71.808101825999998, 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.654 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.396459531999994, 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:30.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:08:30.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:08:30.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.393628660000005, 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.389655751999996, 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:08:30.666 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:08:30.667 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:08:30.668 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.386538801, 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:08:30.669 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.670 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.671 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.672 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:08:30.673 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.384607797000001, 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.674 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.675 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.676 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.677 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.381501037999996, 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.678 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.679 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:08:30.680 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.681 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.378343353999995, 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.682 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.683 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.684 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.685 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.376110303000004, 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.686 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.687 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:08:30.688 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:08:30.689 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.373858630000001, 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.690 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:08:30.691 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.692 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:08:30.693 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.281281042000003, 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.694 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:08:30.695 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:08:30.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:08:30.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 64.504611666000002, 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:08:30.698 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:08:30.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.702 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.705 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:08:30.705 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:30.705 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 58.495104081999997, 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:30.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:08:30.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:08:30.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 58.388274103000001, 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:08:30.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:08:30.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:08:30.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 51.054902550999998, 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:08:30.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 46.282911081999998, 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.718 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:08:30.722 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 36.400762626999999, 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:30.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:08:30.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:08:30.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.726 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2913 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'440, mlcod=60'435)", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 36.398396601999998, 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20445745200000001, 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248250+0000", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248253+0000", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0139999999999999e-06 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248265+0000", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2289999999999999e-05 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248286+0000", 2024-09-21T18:08:30.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1285e-05 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.248300+0000", 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.375e-05 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451311+0000", 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.203011145 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.729 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451347+0000", 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5355999999999998e-05 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.452613+0000", 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001266971 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.452707+0000", 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3640999999999993e-05 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 11.860638179, 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.734 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.735 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.735 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:30.735 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:30.735 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2387224 2024-09-21T18:08:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 856763 2024-09-21T18:08:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:1561: finishing write tid 1 to smithi16030472-449 2024-09-21T18:08:30.736 INFO:tasks.rados.rados.0.smithi160.stdout:1567: expect (ObjNum 221 snap 0 seq_num 221) 2024-09-21T18:08:30.737 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:30.738 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:30.731+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:30.803 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1052784 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1565: finishing write tid 1 to smithi16030472-148 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1565: finishing write tid 2 to smithi16030472-148 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1562: finishing write tid 1 to smithi16030472-235 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1551: finishing copy_from racing read to smithi16030472-153 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1561: finishing write tid 2 to smithi16030472-449 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1561: finishing write tid 3 to smithi16030472-449 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1561: finishing write tid 4 to smithi16030472-449 2024-09-21T18:08:30.834 INFO:tasks.rados.rados.0.smithi160.stdout:1561: oid 449 updating version 0 to 576 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1561: oid 449 updating version 576 to 578 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1561: oid 449 version 578 is already newer than 577 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 578 (ObjNum 848 snap 0 seq_num 848) dirty exists 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1561: left oid 449 (ObjNum 848 snap 0 seq_num 848) 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1571: got (0) Success 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: finishing write tid 3 to smithi16030472-82 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: finishing write tid 4 to smithi16030472-82 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: oid 82 updating version 0 to 534 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: oid 82 updating version 534 to 536 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: oid 82 version 536 is already newer than 535 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 536 (ObjNum 850 snap 0 seq_num 850) dirty exists 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1563: left oid 82 (ObjNum 850 snap 0 seq_num 850) 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1565: finishing write tid 3 to smithi16030472-148 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1551: finishing copy_from to smithi16030472-153 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 153 v 465 (ObjNum 624 snap 0 seq_num 624) dirty exists 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1562: finishing write tid 2 to smithi16030472-235 2024-09-21T18:08:30.835 INFO:tasks.rados.rados.0.smithi160.stdout:1554: finishing copy_from racing read to smithi16030472-260 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:1565: finishing write tid 4 to smithi16030472-148 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:1565: oid 148 updating version 0 to 506 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:1565: oid 148 updating version 506 to 507 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:1565: oid 148 version 507 is already newer than 505 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 148 v 507 (ObjNum 851 snap 0 seq_num 851) dirty exists 2024-09-21T18:08:30.836 INFO:tasks.rados.rados.0.smithi160.stdout:1565: left oid 148 (ObjNum 851 snap 0 seq_num 851) 2024-09-21T18:08:30.837 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:30.829+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:30.860 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:30.860 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:30.860 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:08:30.860 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:30.874 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:30.878 INFO:tasks.rados.rados.0.smithi160.stdout:1562: finishing write tid 3 to smithi16030472-235 2024-09-21T18:08:30.878 INFO:tasks.rados.rados.0.smithi160.stdout:1562: finishing write tid 4 to smithi16030472-235 2024-09-21T18:08:30.878 INFO:tasks.rados.rados.0.smithi160.stdout:1562: oid 235 updating version 0 to 430 2024-09-21T18:08:30.879 INFO:tasks.rados.rados.0.smithi160.stdout:1562: oid 235 updating version 430 to 631 2024-09-21T18:08:30.879 INFO:tasks.rados.rados.0.smithi160.stdout:1562: oid 235 version 631 is already newer than 630 2024-09-21T18:08:30.879 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 235 v 631 (ObjNum 849 snap 0 seq_num 849) dirty exists 2024-09-21T18:08:30.879 INFO:tasks.rados.rados.0.smithi160.stdout:1562: left oid 235 (ObjNum 849 snap 0 seq_num 849) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1550: done (13 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1551: done (12 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1555: done (11 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1561: done (10 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1562: done (9 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1563: done (8 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1565: done (7 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1567: done (6 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1569: done (5 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1570: done (4 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1571: done (3 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1572: done (2 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1573: done (1 left) 2024-09-21T18:08:30.891 INFO:tasks.rados.rados.0.smithi160.stdout:1574: copy_from oid 409 from oid 43 current snap is 0 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1554: finishing copy_from to smithi16030472-260 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 260 v 442 (ObjNum 580 snap 0 seq_num 580) dirty exists 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1554: done (1 left) 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1575: cache_evict oid 483 snap -1 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1576: read oid 425 snap -1 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1576: expect deleted 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1577: write (excl) oid 78 current snap is 0 2024-09-21T18:08:30.892 INFO:tasks.rados.rados.0.smithi160.stdout:1577: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-21T18:08:30.900 INFO:tasks.rados.rados.0.smithi160.stdout:1577: writing smithi16030472-78 from 736371 to 1155304 tid 1 2024-09-21T18:08:30.908 INFO:tasks.rados.rados.0.smithi160.stdout:1577: writing smithi16030472-78 from 1798905 to 2233202 tid 2 2024-09-21T18:08:30.909 INFO:tasks.rados.rados.0.smithi160.stdout:1578: write oid 47 current snap is 0 2024-09-21T18:08:30.909 INFO:tasks.rados.rados.0.smithi160.stdout:1578: seq_num 854 ranges {753178=494182,1857387=495791} 2024-09-21T18:08:30.919 INFO:tasks.rados.rados.0.smithi160.stdout:1578: writing smithi16030472-47 from 753178 to 1247360 tid 1 2024-09-21T18:08:30.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1578: writing smithi16030472-47 from 1857387 to 2353178 tid 2 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1579: cache_try_flush oid 175 snap -1 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1580: cache_flush oid 48 snap -1 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1581: copy_from oid 346 from oid 106 current snap is 0 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1582: read oid 252 snap -1 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1582: expect (ObjNum 251 snap 0 seq_num 251) 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1583: write oid 54 current snap is 0 2024-09-21T18:08:30.928 INFO:tasks.rados.rados.0.smithi160.stdout:1583: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-21T18:08:30.936 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:08:30.937 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:30.937+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:30.939 INFO:tasks.rados.rados.0.smithi160.stdout:1583: writing smithi16030472-54 from 769985 to 1339416 tid 1 2024-09-21T18:08:30.947 INFO:tasks.rados.rados.0.smithi160.stdout:1583: writing smithi16030472-54 from 1915869 to 2353835 tid 2 2024-09-21T18:08:30.947 INFO:tasks.rados.rados.0.smithi160.stdout:1584: write (excl) oid 206 current snap is 0 2024-09-21T18:08:30.947 INFO:tasks.rados.rados.0.smithi160.stdout:1584: seq_num 856 ranges {786792=644680,1857998=528794} 2024-09-21T18:08:30.957 INFO:tasks.rados.rados.0.smithi160.stdout:1584: writing smithi16030472-206 from 786792 to 1431472 tid 1 2024-09-21T18:08:30.963 INFO:tasks.rados.rados.0.smithi160.stdout:1584: writing smithi16030472-206 from 1857998 to 2386792 tid 2 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1585: cache_try_flush oid 153 snap -1 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1586: copy_from oid 490 from oid 75 current snap is 0 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1578: finishing write tid 1 to smithi16030472-47 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1587: cache_try_flush oid 440 snap -1 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1588: cache_evict oid 333 snap -1 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout:1589: copy_from oid 439 from oid 324 current snap is 0 2024-09-21T18:08:30.964 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:30.967 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:30.963+0000 7ff409995640 -1 osd.4 78 handle_osd_map injecting map crc failure 2024-09-21T18:08:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:1575: got (2) No such file or directory 2024-09-21T18:08:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:1585: got (0) Success 2024-09-21T18:08:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 153 v 0 (ObjNum 624 snap 0 seq_num 624) dirty exists 2024-09-21T18:08:30.994 INFO:tasks.rados.rados.0.smithi160.stdout:1575: done (15 left) 2024-09-21T18:08:30.994 INFO:tasks.rados.rados.0.smithi160.stdout:1585: done (14 left) 2024-09-21T18:08:30.994 INFO:tasks.rados.rados.0.smithi160.stdout:1590: read oid 447 snap -1 2024-09-21T18:08:30.994 INFO:tasks.rados.rados.0.smithi160.stdout:1590: expect deleted 2024-09-21T18:08:30.994 INFO:tasks.rados.rados.0.smithi160.stdout:1591: delete oid 214 current snap is 0 2024-09-21T18:08:30.995 INFO:tasks.rados.rados.0.smithi160.stdout:1587: got (2) No such file or directory 2024-09-21T18:08:30.997 INFO:tasks.rados.rados.0.smithi160.stdout:1588: got (16) Device or resource busy 2024-09-21T18:08:31.003 INFO:tasks.rados.rados.0.smithi160.stdout:1584: finishing write tid 1 to smithi16030472-206 2024-09-21T18:08:31.003 INFO:tasks.rados.rados.0.smithi160.stdout:1584: finishing write tid 2 to smithi16030472-206 2024-09-21T18:08:31.003 INFO:tasks.rados.rados.0.smithi160.stdout:1584: finishing write tid 3 to smithi16030472-206 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:1584: finishing write tid 4 to smithi16030472-206 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:1584: oid 206 updating version 0 to 538 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:1584: oid 206 version 538 is already newer than 537 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:1584: oid 206 updating version 538 to 539 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 206 v 539 (ObjNum 856 snap 0 seq_num 856) dirty exists 2024-09-21T18:08:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:1584: left oid 206 (ObjNum 856 snap 0 seq_num 856) 2024-09-21T18:08:31.006 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:31.006 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:31.006 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:31.006 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:31.006 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:31.007 INFO:tasks.rados.rados.0.smithi160.stdout:1577: finishing write tid 1 to smithi16030472-78 2024-09-21T18:08:31.008 INFO:tasks.rados.rados.0.smithi160.stdout:1574: finishing copy_from racing read to smithi16030472-409 2024-09-21T18:08:31.012 INFO:tasks.rados.rados.0.smithi160.stdout:1574: finishing copy_from to smithi16030472-409 2024-09-21T18:08:31.013 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 40 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:08:31.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:31.038 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:31.037+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:31.067 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:31.068 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:31.088 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:31.088 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:31.108 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:31.128 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:31.138 INFO:tasks.rados.rados.0.smithi160.stdout:1579: got (0) Success 2024-09-21T18:08:31.138 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 0 (ObjNum 719 snap 0 seq_num 719) dirty exists 2024-09-21T18:08:31.138 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:31.137+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:31.145 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:31.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:31.165 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:31.166 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:31.186 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:31.186 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:31.187 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:31.193 INFO:tasks.rados.rados.0.smithi160.stdout:1574: done (15 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1579: done (14 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1584: done (13 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1587: done (12 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1588: done (11 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1591: done (10 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1592: cache_try_flush oid 105 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1593: cache_flush oid 95 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1576: done (11 left) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1594: read oid 315 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1594: expect (ObjNum 737 snap 0 seq_num 737) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1595: cache_evict oid 209 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1596: read oid 3 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1596: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1597: cache_evict oid 290 snap -1 2024-09-21T18:08:31.194 INFO:tasks.rados.rados.0.smithi160.stdout:1598: cache_flush oid 241 snap -1 2024-09-21T18:08:31.195 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.195 INFO:tasks.rados.rados.0.smithi160.stdout:1595: got (16) Device or resource busy 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:1595: done (15 left) 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:1599: cache_try_flush oid 341 snap -1 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:1599: got (0) Success 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 0 (ObjNum 705 snap 0 seq_num 705) dirty exists 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:1599: done (15 left) 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout:1600: copy_from oid 371 from oid 199 current snap is 0 2024-09-21T18:08:31.196 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.196 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.196 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.208 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:31.208 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:31.213 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:31.215 INFO:tasks.rados.rados.0.smithi160.stdout:1586: finishing copy_from racing read to smithi16030472-490 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: finishing write tid 2 to smithi16030472-47 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: finishing write tid 3 to smithi16030472-47 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: finishing write tid 4 to smithi16030472-47 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: oid 47 updating version 0 to 632 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: oid 47 updating version 632 to 634 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: oid 47 version 634 is already newer than 633 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 634 (ObjNum 854 snap 0 seq_num 854) dirty exists 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: left oid 47 (ObjNum 854 snap 0 seq_num 854) 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1578: done (15 left) 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout:1601: cache_flush oid 127 snap -1 2024-09-21T18:08:31.224 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.225 INFO:tasks.rados.rados.0.smithi160.stdout:1582: expect (ObjNum 251 snap 0 seq_num 251) 2024-09-21T18:08:31.227 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1852761 2024-09-21T18:08:31.231 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:31.239 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:31.236+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:31.251 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:31.251 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:31.287 INFO:tasks.rados.rados.0.smithi160.stdout:1582: done (15 left) 2024-09-21T18:08:31.288 INFO:tasks.rados.rados.0.smithi160.stdout:1602: read oid 93 snap -1 2024-09-21T18:08:31.288 INFO:tasks.rados.rados.0.smithi160.stdout:1602: expect deleted 2024-09-21T18:08:31.288 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1592: got (0) Success 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 105 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1590: done (15 left) 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1592: done (14 left) 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1603: read oid 293 snap -1 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1603: expect deleted 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout:1604: cache_flush oid 152 snap -1 2024-09-21T18:08:31.295 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.297 INFO:tasks.rados.rados.0.smithi160.stdout:1594: expect (ObjNum 737 snap 0 seq_num 737) 2024-09-21T18:08:31.298 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.321 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:31.321 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:31.332 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:31.325+0000 7f6a3cca9640 -1 osd.3 78 handle_osd_map injecting map crc failure 2024-09-21T18:08:31.332 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:31.329+0000 7f3ce267c640 -1 osd.6 78 handle_osd_map injecting map crc failure 2024-09-21T18:08:31.333 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.339 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.339 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.339 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:31.337+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:31.347 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:31.348 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:31.354 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:31.396 INFO:tasks.rados.rados.0.smithi160.stdout:1596: expect (ObjNum 2 snap 0 seq_num 2) 2024-09-21T18:08:31.410 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 59499 2024-09-21T18:08:31.440 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:31.436+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:31.445 INFO:tasks.rados.rados.0.smithi160.stdout:1577: finishing write tid 2 to smithi16030472-78 2024-09-21T18:08:31.445 INFO:tasks.rados.rados.0.smithi160.stdout:1577: finishing write tid 3 to smithi16030472-78 2024-09-21T18:08:31.445 INFO:tasks.rados.rados.0.smithi160.stdout:1577: finishing write tid 4 to smithi16030472-78 2024-09-21T18:08:31.445 INFO:tasks.rados.rados.0.smithi160.stdout:1577: oid 78 updating version 0 to 637 2024-09-21T18:08:31.445 INFO:tasks.rados.rados.0.smithi160.stdout:1577: oid 78 version 637 is already newer than 636 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1577: oid 78 updating version 637 to 638 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 638 (ObjNum 853 snap 0 seq_num 853) dirty exists 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1577: left oid 78 (ObjNum 853 snap 0 seq_num 853) 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1583: finishing write tid 1 to smithi16030472-54 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1583: finishing write tid 2 to smithi16030472-54 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1577: done (15 left) 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1594: done (14 left) 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1596: done (13 left) 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1602: done (12 left) 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1605: cache_try_flush oid 314 snap -1 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1593: got (0) Success 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 95 v 0 (ObjNum 94 snap 0 seq_num 94) dirty exists 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1583: finishing write tid 3 to smithi16030472-54 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:1580: got (0) Success 2024-09-21T18:08:31.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 48 v 0 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1586: finishing copy_from to smithi16030472-490 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 490 v 49 (ObjNum 74 snap 0 seq_num 74) dirty exists 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1598: got (0) Success 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 0 (ObjNum 240 snap 0 seq_num 240) dirty exists 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1580: done (12 left) 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1586: done (11 left) 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1593: done (10 left) 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1598: done (9 left) 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1606: read oid 341 snap -1 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1606: expect (ObjNum 705 snap 0 seq_num 705) 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1607: copy_from oid 406 from oid 334 current snap is 0 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1597: got (2) No such file or directory 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1601: got (2) No such file or directory 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1583: finishing write tid 4 to smithi16030472-54 2024-09-21T18:08:31.447 INFO:tasks.rados.rados.0.smithi160.stdout:1583: oid 54 updating version 0 to 636 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1583: oid 54 updating version 636 to 637 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1583: oid 54 version 637 is already newer than 635 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 637 (ObjNum 855 snap 0 seq_num 855) dirty exists 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1583: left oid 54 (ObjNum 855 snap 0 seq_num 855) 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1583: done (10 left) 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1597: done (9 left) 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1601: done (8 left) 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1608: write oid 39 current snap is 0 2024-09-21T18:08:31.448 INFO:tasks.rados.rados.0.smithi160.stdout:1608: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-21T18:08:31.451 INFO:tasks.rados.rados.0.smithi160.stdout:1608: writing smithi16030472-39 from 420406 to 1215584 tid 1 2024-09-21T18:08:31.458 INFO:tasks.rados.rados.0.smithi160.stdout:1608: writing smithi16030472-39 from 1974962 to 2420406 tid 2 2024-09-21T18:08:31.458 INFO:tasks.rados.rados.0.smithi160.stdout:1609: write oid 433 current snap is 0 2024-09-21T18:08:31.458 INFO:tasks.rados.rados.0.smithi160.stdout:1605: got (2) No such file or directory 2024-09-21T18:08:31.458 INFO:tasks.rados.rados.0.smithi160.stdout:1609: seq_num 859 ranges {437213=470427,1633444=728951} 2024-09-21T18:08:31.463 INFO:tasks.rados.rados.0.smithi160.stdout:1609: writing smithi16030472-433 from 437213 to 907640 tid 1 2024-09-21T18:08:31.471 INFO:tasks.rados.rados.0.smithi160.stdout:1609: writing smithi16030472-433 from 1633444 to 2362395 tid 2 2024-09-21T18:08:31.472 INFO:tasks.rados.rados.0.smithi160.stdout:1605: done (9 left) 2024-09-21T18:08:31.472 INFO:tasks.rados.rados.0.smithi160.stdout:1610: cache_try_flush oid 329 snap -1 2024-09-21T18:08:31.472 INFO:tasks.rados.rados.0.smithi160.stdout:1606: expect (ObjNum 705 snap 0 seq_num 705) 2024-09-21T18:08:31.516 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.516 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.530 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:31.532 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.540 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:31.537+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:31.564 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:31.564 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:31.597 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.603 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:31.603 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:31.642 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:31.636+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1581: finishing copy_from to smithi16030472-346 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1581: got expected ENOENT (src dne) 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1581: finishing copy_from racing read to smithi16030472-346 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1589: finishing copy_from racing read to smithi16030472-439 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1604: got (2) No such file or directory 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1581: done (9 left) 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1604: done (8 left) 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1606: done (7 left) 2024-09-21T18:08:31.659 INFO:tasks.rados.rados.0.smithi160.stdout:1611: read oid 347 snap -1 2024-09-21T18:08:31.660 INFO:tasks.rados.rados.0.smithi160.stdout:1611: expect (ObjNum 773 snap 0 seq_num 773) 2024-09-21T18:08:31.660 INFO:tasks.rados.rados.0.smithi160.stdout:1612: write oid 239 current snap is 0 2024-09-21T18:08:31.660 INFO:tasks.rados.rados.0.smithi160.stdout:1612: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-21T18:08:31.662 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.662 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.665 INFO:tasks.rados.rados.0.smithi160.stdout:1612: writing smithi16030472-239 from 454020 to 1116049 tid 1 2024-09-21T18:08:31.672 INFO:tasks.rados.rados.0.smithi160.stdout:1612: writing smithi16030472-239 from 1691926 to 2280888 tid 2 2024-09-21T18:08:31.672 INFO:tasks.rados.rados.0.smithi160.stdout:1613: write (excl) oid 283 current snap is 0 2024-09-21T18:08:31.672 INFO:tasks.rados.rados.0.smithi160.stdout:1613: seq_num 861 ranges {470827=737278,1750408=720419} 2024-09-21T18:08:31.677 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:31.678 INFO:tasks.rados.rados.0.smithi160.stdout:1613: writing smithi16030472-283 from 470827 to 1208105 tid 1 2024-09-21T18:08:31.682 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:31.685 INFO:tasks.rados.rados.0.smithi160.stdout:1613: writing smithi16030472-283 from 1750408 to 2470827 tid 2 2024-09-21T18:08:31.685 INFO:tasks.rados.rados.0.smithi160.stdout:1614: cache_evict oid 351 snap -1 2024-09-21T18:08:31.685 INFO:tasks.rados.rados.0.smithi160.stdout:1615: write (excl) oid 467 current snap is 0 2024-09-21T18:08:31.685 INFO:tasks.rados.rados.0.smithi160.stdout:1615: seq_num 862 ranges {487634=412527,1408890=592631} 2024-09-21T18:08:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:31.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:31.691 INFO:tasks.rados.rados.0.smithi160.stdout:1615: writing smithi16030472-467 from 487634 to 900161 tid 1 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1615: writing smithi16030472-467 from 1408890 to 2001521 tid 2 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1616: cache_try_flush oid 231 snap -1 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1617: copy_from oid 127 from oid 327 current snap is 0 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1618: cache_evict oid 392 snap -1 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1619: write (excl) oid 392 current snap is 0 2024-09-21T18:08:31.700 INFO:tasks.rados.rados.0.smithi160.stdout:1619: seq_num 863 ranges {504441=487776,1467372=736289} 2024-09-21T18:08:31.707 INFO:tasks.rados.rados.0.smithi160.stdout:1619: writing smithi16030472-392 from 504441 to 992217 tid 1 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:1619: writing smithi16030472-392 from 1467372 to 2203661 tid 2 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:1589: finishing copy_from to smithi16030472-439 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 439 v 205 (ObjNum 323 snap 0 seq_num 323) dirty exists 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:1589: done (15 left) 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:1620: read oid 448 snap -1 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout:1620: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:08:31.715 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.742 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:31.740+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:31.769 INFO:tasks.rados.rados.0.smithi160.stdout:1620: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:08:31.769 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.804 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.805 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.811 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:31.822 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:31.843 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:31.836+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: finishing write tid 1 to smithi16030472-433 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: finishing write tid 2 to smithi16030472-433 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1613: finishing write tid 1 to smithi16030472-283 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: finishing write tid 3 to smithi16030472-433 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: finishing write tid 4 to smithi16030472-433 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: oid 433 updating version 0 to 542 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: oid 433 version 542 is already newer than 541 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:1609: oid 433 updating version 542 to 543 2024-09-21T18:08:31.893 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 543 (ObjNum 859 snap 0 seq_num 859) dirty exists 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1609: left oid 433 (ObjNum 859 snap 0 seq_num 859) 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1608: finishing write tid 1 to smithi16030472-39 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1608: finishing write tid 2 to smithi16030472-39 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1613: finishing write tid 2 to smithi16030472-283 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1603: done (15 left) 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1609: done (14 left) 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1620: done (13 left) 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1621: write (excl) oid 453 current snap is 0 2024-09-21T18:08:31.894 INFO:tasks.rados.rados.0.smithi160.stdout:1621: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-21T18:08:31.898 INFO:tasks.rados.rados.0.smithi160.stdout:1621: writing smithi16030472-453 from 521248 to 1084273 tid 1 2024-09-21T18:08:31.907 INFO:tasks.rados.rados.0.smithi160.stdout:1621: writing smithi16030472-453 from 1809501 to 2405801 tid 2 2024-09-21T18:08:31.907 INFO:tasks.rados.rados.0.smithi160.stdout:1622: copy_from oid 91 from oid 2 current snap is 0 2024-09-21T18:08:31.907 INFO:tasks.rados.rados.0.smithi160.stdout:1615: finishing write tid 1 to smithi16030472-467 2024-09-21T18:08:31.907 INFO:tasks.rados.rados.0.smithi160.stdout:1623: copy_from oid 123 from oid 373 current snap is 0 2024-09-21T18:08:31.907 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:31.943 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:31.939+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:31.952 INFO:tasks.rados.rados.0.smithi160.stdout:1607: finishing copy_from racing read to smithi16030472-406 2024-09-21T18:08:31.992 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:31.992 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: finishing write tid 3 to smithi16030472-283 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: finishing write tid 4 to smithi16030472-283 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: oid 283 updating version 0 to 641 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: oid 283 version 641 is already newer than 640 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: oid 283 version 641 is already newer than 639 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 283 v 641 (ObjNum 861 snap 0 seq_num 861) dirty exists 2024-09-21T18:08:31.993 INFO:tasks.rados.rados.0.smithi160.stdout:1613: left oid 283 (ObjNum 861 snap 0 seq_num 861) 2024-09-21T18:08:31.994 INFO:tasks.rados.rados.0.smithi160.stdout:1613: done (15 left) 2024-09-21T18:08:31.994 INFO:tasks.rados.rados.0.smithi160.stdout:1624: read oid 243 snap -1 2024-09-21T18:08:31.994 INFO:tasks.rados.rados.0.smithi160.stdout:1624: expect (ObjNum 762 snap 0 seq_num 762) 2024-09-21T18:08:31.994 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:32.007 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:32.019 INFO:tasks.rados.rados.0.smithi160.stdout:1615: finishing write tid 2 to smithi16030472-467 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: finishing write tid 3 to smithi16030472-467 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: finishing write tid 4 to smithi16030472-467 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: oid 467 updating version 0 to 581 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: oid 467 version 581 is already newer than 580 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: oid 467 updating version 581 to 582 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 467 v 582 (ObjNum 862 snap 0 seq_num 862) dirty exists 2024-09-21T18:08:32.020 INFO:tasks.rados.rados.0.smithi160.stdout:1615: left oid 467 (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:08:32.022 INFO:tasks.rados.rados.0.smithi160.stdout:1615: done (15 left) 2024-09-21T18:08:32.022 INFO:tasks.rados.rados.0.smithi160.stdout:1625: read oid 84 snap -1 2024-09-21T18:08:32.022 INFO:tasks.rados.rados.0.smithi160.stdout:1625: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-21T18:08:32.022 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:32.031 INFO:tasks.rados.rados.0.smithi160.stdout:1625: expect (ObjNum 83 snap 0 seq_num 83) 2024-09-21T18:08:32.045 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:32.039+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:32.071 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1621: finishing write tid 1 to smithi16030472-453 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1621: finishing write tid 2 to smithi16030472-453 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1617: finishing copy_from racing read to smithi16030472-127 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1621: finishing write tid 3 to smithi16030472-453 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1617: finishing copy_from to smithi16030472-127 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 127 v 229 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1622: finishing copy_from racing read to smithi16030472-91 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1616: got (0) Success 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 231 v 0 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1616: done (15 left) 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1617: done (14 left) 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1625: done (13 left) 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1626: write (excl) oid 403 current snap is 0 2024-09-21T18:08:32.136 INFO:tasks.rados.rados.0.smithi160.stdout:1626: seq_num 865 ranges {538055=638274,1867983=670072} 2024-09-21T18:08:32.142 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.143 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.145 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:32.140+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:32.150 INFO:tasks.rados.rados.0.smithi160.stdout:1626: writing smithi16030472-403 from 538055 to 1176329 tid 1 2024-09-21T18:08:32.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout:1626: writing smithi16030472-403 from 1867983 to 2538055 tid 2 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout:1627: cache_evict oid 143 snap -1 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout:1628: cache_evict oid 182 snap -1 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout:1608: finishing write tid 3 to smithi16030472-39 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:32.169 INFO:tasks.rados.rados.0.smithi160.stdout:1608: finishing write tid 4 to smithi16030472-39 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1608: oid 39 updating version 0 to 640 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1608: oid 39 version 640 is already newer than 638 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1608: oid 39 version 640 is already newer than 639 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 640 (ObjNum 858 snap 0 seq_num 858) dirty exists 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1608: left oid 39 (ObjNum 858 snap 0 seq_num 858) 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1607: finishing copy_from to smithi16030472-406 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 406 v 107 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1607: done (15 left) 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1608: done (14 left) 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1629: read oid 7 snap -1 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1629: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout:1630: cache_evict oid 10 snap -1 2024-09-21T18:08:32.170 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:32.246 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:32.239+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1621: finishing write tid 4 to smithi16030472-453 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1621: oid 453 updating version 0 to 544 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1621: oid 453 updating version 544 to 545 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1621: oid 453 updating version 545 to 546 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 453 v 546 (ObjNum 864 snap 0 seq_num 864) dirty exists 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1621: left oid 453 (ObjNum 864 snap 0 seq_num 864) 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:1622: finishing copy_from to smithi16030472-91 2024-09-21T18:08:32.271 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1627: got (2) No such file or directory 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1621: done (15 left) 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1622: done (14 left) 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1627: done (13 left) 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1631: cache_evict oid 284 snap -1 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1630: got (16) Device or resource busy 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1630: done (13 left) 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1632: copy_from oid 456 from oid 115 current snap is 0 2024-09-21T18:08:32.272 INFO:tasks.rados.rados.0.smithi160.stdout:1633: delete oid 307 current snap is 0 2024-09-21T18:08:32.273 INFO:tasks.rados.rados.0.smithi160.stdout:1631: got (16) Device or resource busy 2024-09-21T18:08:32.275 INFO:tasks.rados.rados.0.smithi160.stdout:1611: expect (ObjNum 773 snap 0 seq_num 773) 2024-09-21T18:08:32.275 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:32.293 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.293 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:32.347 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:32.344+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:32.413 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:32.448 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:32.444+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:32.457 INFO:tasks.rados.rados.0.smithi160.stdout:1614: got (2) No such file or directory 2024-09-21T18:08:32.458 INFO:tasks.rados.rados.0.smithi160.stdout:1626: finishing write tid 1 to smithi16030472-403 2024-09-21T18:08:32.458 INFO:tasks.rados.rados.0.smithi160.stdout:1624: expect (ObjNum 762 snap 0 seq_num 762) 2024-09-21T18:08:32.475 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.476 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.482 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1889625 2024-09-21T18:08:32.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:32.506 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:32.549 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:32.541+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:32.585 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1654376 2024-09-21T18:08:32.610 INFO:tasks.rados.rados.0.smithi160.stdout:1600: finishing copy_from to smithi16030472-371 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 107 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1618: got (0) Success 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1632: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1632: finishing copy_from to smithi16030472-456 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 520 (ObjNum 825 snap 0 seq_num 825) dirty exists 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1626: finishing write tid 2 to smithi16030472-403 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1612: finishing write tid 1 to smithi16030472-239 2024-09-21T18:08:32.611 INFO:tasks.rados.rados.0.smithi160.stdout:1629: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-21T18:08:32.617 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.617 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.621 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1918328 2024-09-21T18:08:32.623 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:32.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:32.669 INFO:tasks.rados.rados.0.smithi160.stdout:1628: got (0) Success 2024-09-21T18:08:32.669 INFO:tasks.rados.rados.0.smithi160.stdout:1626: finishing write tid 3 to smithi16030472-403 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1626: finishing write tid 4 to smithi16030472-403 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1626: oid 403 updating version 0 to 547 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1626: oid 403 updating version 547 to 548 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1626: oid 403 version 548 is already newer than 511 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 403 v 548 (ObjNum 865 snap 0 seq_num 865) dirty exists 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1626: left oid 403 (ObjNum 865 snap 0 seq_num 865) 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1600: finishing copy_from racing read to smithi16030472-371 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1619: finishing write tid 1 to smithi16030472-392 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1623: finishing copy_from to smithi16030472-123 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 265 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1610: got (0) Success 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 329 v 0 (ObjNum 747 snap 0 seq_num 747) dirty exists 2024-09-21T18:08:32.670 INFO:tasks.rados.rados.0.smithi160.stdout:1612: finishing write tid 2 to smithi16030472-239 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1612: finishing write tid 3 to smithi16030472-239 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1600: done (14 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1610: done (13 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1611: done (12 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1614: done (11 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1618: done (10 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1624: done (9 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1626: done (8 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1628: done (7 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1629: done (6 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1631: done (5 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1632: done (4 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1633: done (3 left) 2024-09-21T18:08:32.671 INFO:tasks.rados.rados.0.smithi160.stdout:1634: cache_flush oid 496 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1635: cache_try_flush oid 285 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1636: cache_try_flush oid 243 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1637: read oid 287 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1637: expect deleted 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1638: read oid 377 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1638: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1639: cache_try_flush oid 225 snap -1 2024-09-21T18:08:32.672 INFO:tasks.rados.rados.0.smithi160.stdout:1640: delete oid 155 current snap is 0 2024-09-21T18:08:32.677 INFO:tasks.rados.rados.0.smithi160.stdout:1638: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:08:32.679 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:32.675+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:32.700 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 178924 2024-09-21T18:08:32.751 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:32.748+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:32.760 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.760 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.770 INFO:tasks.rados.rados.0.smithi160.stdout:1623: finishing copy_from racing read to smithi16030472-123 2024-09-21T18:08:32.770 INFO:tasks.rados.rados.0.smithi160.stdout:1619: finishing write tid 2 to smithi16030472-392 2024-09-21T18:08:32.770 INFO:tasks.rados.rados.0.smithi160.stdout:1634: got (0) Success 2024-09-21T18:08:32.770 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 0 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:08:32.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:32.792 INFO:tasks.rados.rados.0.smithi160.stdout:1639: got (0) Success 2024-09-21T18:08:32.792 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 0 (ObjNum 776 snap 0 seq_num 776) dirty exists 2024-09-21T18:08:32.825 INFO:tasks.rados.rados.0.smithi160.stdout:1619: finishing write tid 3 to smithi16030472-392 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:1619: finishing write tid 4 to smithi16030472-392 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:1619: oid 392 updating version 0 to 644 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:1619: oid 392 version 644 is already newer than 643 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:1619: oid 392 version 644 is already newer than 432 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 392 v 644 (ObjNum 863 snap 0 seq_num 863) dirty exists 2024-09-21T18:08:32.826 INFO:tasks.rados.rados.0.smithi160.stdout:1619: left oid 392 (ObjNum 863 snap 0 seq_num 863) 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:1612: finishing write tid 4 to smithi16030472-239 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:1612: oid 239 updating version 0 to 642 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:1612: oid 239 version 642 is already newer than 431 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:1612: oid 239 version 642 is already newer than 641 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 239 v 642 (ObjNum 860 snap 0 seq_num 860) dirty exists 2024-09-21T18:08:32.827 INFO:tasks.rados.rados.0.smithi160.stdout:1612: left oid 239 (ObjNum 860 snap 0 seq_num 860) 2024-09-21T18:08:32.852 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:32.848+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:32.926 INFO:tasks.rados.rados.0.smithi160.stdout:1612: done (9 left) 2024-09-21T18:08:32.926 INFO:tasks.rados.rados.0.smithi160.stdout:1619: done (8 left) 2024-09-21T18:08:32.926 INFO:tasks.rados.rados.0.smithi160.stdout:1623: done (7 left) 2024-09-21T18:08:32.926 INFO:tasks.rados.rados.0.smithi160.stdout:1634: done (6 left) 2024-09-21T18:08:32.926 INFO:tasks.rados.rados.0.smithi160.stdout:1637: done (5 left) 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1638: done (4 left) 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1639: done (3 left) 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1640: done (2 left) 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1641: cache_flush oid 216 snap -1 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1642: copy_from oid 347 from oid 222 current snap is 0 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1643: write (excl) oid 15 current snap is 0 2024-09-21T18:08:32.927 INFO:tasks.rados.rados.0.smithi160.stdout:1643: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-21T18:08:32.945 INFO:tasks.rados.rados.0.smithi160.stdout:1643: writing smithi16030472-15 from 588476 to 1168850 tid 1 2024-09-21T18:08:32.946 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:32.946 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:32.955 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:32.948+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:32.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:32.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:08:32.963 INFO:tasks.rados.rados.0.smithi160.stdout:1643: writing smithi16030472-15 from 1643429 to 2247067 tid 2 2024-09-21T18:08:32.963 INFO:tasks.rados.rados.0.smithi160.stdout:1644: write (excl) oid 116 current snap is 0 2024-09-21T18:08:32.963 INFO:tasks.rados.rados.0.smithi160.stdout:1644: seq_num 869 ranges {605283=655623,1701911=747296} 2024-09-21T18:08:32.981 INFO:tasks.rados.rados.0.smithi160.stdout:1644: writing smithi16030472-116 from 605283 to 1260906 tid 1 2024-09-21T18:08:32.995 INFO:tasks.rados.rados.0.smithi160.stdout:1644: writing smithi16030472-116 from 1701911 to 2449207 tid 2 2024-09-21T18:08:32.995 INFO:tasks.rados.rados.0.smithi160.stdout:1645: read oid 299 snap -1 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:1645: expect (ObjNum 743 snap 0 seq_num 743) 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:1646: delete oid 366 current snap is 0 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:1641: got (2) No such file or directory 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:1642: finishing copy_from racing read to smithi16030472-347 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:1642: finishing copy_from to smithi16030472-347 2024-09-21T18:08:32.996 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 187 (ObjNum 221 snap 0 seq_num 221) dirty exists 2024-09-21T18:08:33.008 INFO:tasks.rados.rados.0.smithi160.stdout:1645: expect (ObjNum 743 snap 0 seq_num 743) 2024-09-21T18:08:33.009 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:33.053 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:33.047+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:33.084 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:33.093 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.093 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:33.128 INFO:tasks.rados.rados.0.smithi160.stdout:1636: got (0) Success 2024-09-21T18:08:33.128 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 0 (ObjNum 762 snap 0 seq_num 762) dirty exists 2024-09-21T18:08:33.128 INFO:tasks.rados.rados.0.smithi160.stdout:1635: got (0) Success 2024-09-21T18:08:33.128 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 285 v 0 (ObjNum 284 snap 0 seq_num 284) dirty exists 2024-09-21T18:08:33.128 INFO:tasks.rados.rados.0.smithi160.stdout:1644: finishing write tid 1 to smithi16030472-116 2024-09-21T18:08:33.171 INFO:tasks.rados.rados.0.smithi160.stdout:1643: finishing write tid 1 to smithi16030472-15 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1635: done (7 left) 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1636: done (6 left) 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1641: done (5 left) 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1642: done (4 left) 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1645: done (3 left) 2024-09-21T18:08:33.183 INFO:tasks.rados.rados.0.smithi160.stdout:1646: done (2 left) 2024-09-21T18:08:33.184 INFO:tasks.rados.rados.0.smithi160.stdout:1647: cache_flush oid 286 snap -1 2024-09-21T18:08:33.184 INFO:tasks.rados.rados.0.smithi160.stdout:1648: read oid 431 snap -1 2024-09-21T18:08:33.184 INFO:tasks.rados.rados.0.smithi160.stdout:1648: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-21T18:08:33.184 INFO:tasks.rados.rados.0.smithi160.stdout:1649: copy_from oid 34 from oid 239 current snap is 0 2024-09-21T18:08:33.184 INFO:tasks.rados.rados.0.smithi160.stdout:1650: delete oid 75 current snap is 0 2024-09-21T18:08:33.195 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:33.191+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:33.254 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:33.248+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:33.275 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.275 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:33.375 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:33.371+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:33.384 INFO:tasks.rados.rados.0.smithi160.stdout:1644: finishing write tid 2 to smithi16030472-116 2024-09-21T18:08:33.384 INFO:tasks.rados.rados.0.smithi160.stdout:1643: finishing write tid 2 to smithi16030472-15 2024-09-21T18:08:33.384 INFO:tasks.rados.rados.0.smithi160.stdout:1643: finishing write tid 3 to smithi16030472-15 2024-09-21T18:08:33.384 INFO:tasks.rados.rados.0.smithi160.stdout:1643: finishing write tid 4 to smithi16030472-15 2024-09-21T18:08:33.384 INFO:tasks.rados.rados.0.smithi160.stdout:1643: oid 15 updating version 0 to 585 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1643: oid 15 version 585 is already newer than 583 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1643: oid 15 version 585 is already newer than 584 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 585 (ObjNum 868 snap 0 seq_num 868) dirty exists 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1643: left oid 15 (ObjNum 868 snap 0 seq_num 868) 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: finishing write tid 3 to smithi16030472-116 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1647: got (0) Success 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 286 v 0 (ObjNum 595 snap 0 seq_num 595) dirty exists 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: finishing write tid 4 to smithi16030472-116 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: oid 116 updating version 0 to 550 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: oid 116 updating version 550 to 551 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: oid 116 version 551 is already newer than 549 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 551 (ObjNum 869 snap 0 seq_num 869) dirty exists 2024-09-21T18:08:33.385 INFO:tasks.rados.rados.0.smithi160.stdout:1644: left oid 116 (ObjNum 869 snap 0 seq_num 869) 2024-09-21T18:08:33.389 INFO:tasks.rados.rados.0.smithi160.stdout:1648: expect (ObjNum 64 snap 0 seq_num 64) 2024-09-21T18:08:33.456 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.456 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.456 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:33.449+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:33.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:33.513 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2307188 2024-09-21T18:08:33.557 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:33.553+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:33.594 INFO:tasks.rados.rados.0.smithi160.stdout:1649: finishing copy_from to smithi16030472-34 2024-09-21T18:08:33.594 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 642 (ObjNum 860 snap 0 seq_num 860) dirty exists 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1643: done (5 left) 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1644: done (4 left) 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1647: done (3 left) 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1648: done (2 left) 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1650: done (1 left) 2024-09-21T18:08:33.603 INFO:tasks.rados.rados.0.smithi160.stdout:1651: cache_flush oid 228 snap -1 2024-09-21T18:08:33.604 INFO:tasks.rados.rados.0.smithi160.stdout:1652: write oid 288 current snap is 0 2024-09-21T18:08:33.604 INFO:tasks.rados.rados.0.smithi160.stdout:1652: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-21T18:08:33.610 INFO:tasks.rados.rados.0.smithi160.stdout:1652: writing smithi16030472-288 from 655704 to 1137074 tid 1 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1652: writing smithi16030472-288 from 1761004 to 2255627 tid 2 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1653: read oid 496 snap -1 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1653: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1654: delete oid 41 current snap is 0 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1649: finishing copy_from racing read to smithi16030472-34 2024-09-21T18:08:33.624 INFO:tasks.rados.rados.0.smithi160.stdout:1652: finishing write tid 1 to smithi16030472-288 2024-09-21T18:08:33.637 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.637 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:33.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:33.699 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:33.695+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:33.705 INFO:tasks.rados.rados.0.smithi160.stdout:1649: done (4 left) 2024-09-21T18:08:33.705 INFO:tasks.rados.rados.0.smithi160.stdout:1654: done (3 left) 2024-09-21T18:08:33.705 INFO:tasks.rados.rados.0.smithi160.stdout:1655: copy_from oid 283 from oid 437 current snap is 0 2024-09-21T18:08:33.706 INFO:tasks.rados.rados.0.smithi160.stdout:1656: cache_flush oid 60 snap -1 2024-09-21T18:08:33.706 INFO:tasks.rados.rados.0.smithi160.stdout:1657: read oid 247 snap -1 2024-09-21T18:08:33.706 INFO:tasks.rados.rados.0.smithi160.stdout:1657: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-21T18:08:33.706 INFO:tasks.rados.rados.0.smithi160.stdout:1658: write oid 9 current snap is 0 2024-09-21T18:08:33.706 INFO:tasks.rados.rados.0.smithi160.stdout:1658: seq_num 874 ranges {689318=631868,1877968=498292} 2024-09-21T18:08:33.724 INFO:tasks.rados.rados.0.smithi160.stdout:1658: writing smithi16030472-9 from 689318 to 1321186 tid 1 2024-09-21T18:08:33.740 INFO:tasks.rados.rados.0.smithi160.stdout:1658: writing smithi16030472-9 from 1877968 to 2376260 tid 2 2024-09-21T18:08:33.740 INFO:tasks.rados.rados.0.smithi160.stdout:1659: read oid 245 snap -1 2024-09-21T18:08:33.741 INFO:tasks.rados.rados.0.smithi160.stdout:1659: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-21T18:08:33.741 INFO:tasks.rados.rados.0.smithi160.stdout:1660: read oid 8 snap -1 2024-09-21T18:08:33.741 INFO:tasks.rados.rados.0.smithi160.stdout:1660: expect (ObjNum 552 snap 0 seq_num 552) 2024-09-21T18:08:33.741 INFO:tasks.rados.rados.0.smithi160.stdout:1661: write oid 398 current snap is 0 2024-09-21T18:08:33.741 INFO:tasks.rados.rados.0.smithi160.stdout:1661: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-21T18:08:33.756 INFO:tasks.rados.rados.0.smithi160.stdout:1661: writing smithi16030472-398 from 706125 to 1129595 tid 1 2024-09-21T18:08:33.757 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:33.751+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:33.772 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.772 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.773 INFO:tasks.rados.rados.0.smithi160.stdout:1661: writing smithi16030472-398 from 1652803 to 2411106 tid 2 2024-09-21T18:08:33.773 INFO:tasks.rados.rados.0.smithi160.stdout:1662: write oid 87 current snap is 0 2024-09-21T18:08:33.773 INFO:tasks.rados.rados.0.smithi160.stdout:1662: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-21T18:08:33.785 INFO:tasks.rados.rados.0.smithi160.stdout:1662: writing smithi16030472-87 from 722932 to 1221651 tid 1 2024-09-21T18:08:33.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:33.796 INFO:tasks.rados.rados.0.smithi160.stdout:1662: writing smithi16030472-87 from 1994932 to 2496893 tid 2 2024-09-21T18:08:33.796 INFO:tasks.rados.rados.0.smithi160.stdout:1663: write (excl) oid 23 current snap is 0 2024-09-21T18:08:33.797 INFO:tasks.rados.rados.0.smithi160.stdout:1663: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-21T18:08:33.806 INFO:tasks.rados.rados.0.smithi160.stdout:1663: writing smithi16030472-23 from 739739 to 1313707 tid 1 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1663: writing smithi16030472-23 from 2053414 to 2739739 tid 2 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1664: delete oid 150 current snap is 0 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: finishing write tid 2 to smithi16030472-288 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: finishing write tid 3 to smithi16030472-288 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: finishing write tid 4 to smithi16030472-288 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: oid 288 updating version 0 to 586 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: oid 288 updating version 586 to 588 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:1652: oid 288 version 588 is already newer than 587 2024-09-21T18:08:33.817 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 288 v 588 (ObjNum 872 snap 0 seq_num 872) dirty exists 2024-09-21T18:08:33.818 INFO:tasks.rados.rados.0.smithi160.stdout:1652: left oid 288 (ObjNum 872 snap 0 seq_num 872) 2024-09-21T18:08:33.818 INFO:tasks.rados.rados.0.smithi160.stdout:1653: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:33.818 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:33.852 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:33.859 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:33.853+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:33.873 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:33.873 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:33.893 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:33.905 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:33.915 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:33.915 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:33.932 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:08:33.959 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:33.959 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:33.959 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:33.955+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:33.972 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:33.972 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:33.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:33.990 INFO:tasks.rados.rados.0.smithi160.stdout:1659: expect (ObjNum 395 snap 0 seq_num 395) 2024-09-21T18:08:33.994 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1521053 2024-09-21T18:08:34.010 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:34.060 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:34.053+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:34.068 INFO:tasks.rados.rados.0.smithi160.stdout:1660: expect (ObjNum 552 snap 0 seq_num 552) 2024-09-21T18:08:34.068 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:34.081 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:34.098 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:34.098 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:34.123 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:34.146 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:34.147 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:34.147 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:34.161 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:34.161 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:34.155+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:34.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:34.170 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:34.170 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:34.192 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1661: finishing write tid 1 to smithi16030472-398 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1656: got (0) Success 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 0 (ObjNum 59 snap 0 seq_num 59) dirty exists 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1661: finishing write tid 2 to smithi16030472-398 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1651: got (0) Success 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 228 v 0 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1655: finishing copy_from to smithi16030472-283 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1655: got expected ENOENT (src dne) 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1655: finishing copy_from racing read to smithi16030472-283 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1663: finishing write tid 1 to smithi16030472-23 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1663: finishing write tid 2 to smithi16030472-23 2024-09-21T18:08:34.210 INFO:tasks.rados.rados.0.smithi160.stdout:1663: finishing write tid 3 to smithi16030472-23 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1663: finishing write tid 4 to smithi16030472-23 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1663: oid 23 updating version 0 to 647 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1663: oid 23 updating version 647 to 648 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1663: oid 23 version 648 is already newer than 646 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 23 v 648 (ObjNum 877 snap 0 seq_num 877) dirty exists 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1663: left oid 23 (ObjNum 877 snap 0 seq_num 877) 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: finishing write tid 3 to smithi16030472-398 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: finishing write tid 4 to smithi16030472-398 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: oid 398 updating version 0 to 582 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: oid 398 version 582 is already newer than 581 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: oid 398 updating version 582 to 589 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 398 v 589 (ObjNum 875 snap 0 seq_num 875) dirty exists 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1661: left oid 398 (ObjNum 875 snap 0 seq_num 875) 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1651: done (12 left) 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1652: done (11 left) 2024-09-21T18:08:34.211 INFO:tasks.rados.rados.0.smithi160.stdout:1653: done (10 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1655: done (9 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1656: done (8 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1659: done (7 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1660: done (6 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1661: done (5 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1663: done (4 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1664: done (3 left) 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1665: cache_evict oid 407 snap -1 2024-09-21T18:08:34.212 INFO:tasks.rados.rados.0.smithi160.stdout:1666: delete oid 77 current snap is 0 2024-09-21T18:08:34.214 INFO:tasks.rados.rados.0.smithi160.stdout:1665: got (16) Device or resource busy 2024-09-21T18:08:34.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:34.215 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:34.222 INFO:tasks.rados.rados.0.smithi160.stdout:1658: finishing write tid 1 to smithi16030472-9 2024-09-21T18:08:34.222 INFO:tasks.rados.rados.0.smithi160.stdout:1658: finishing write tid 2 to smithi16030472-9 2024-09-21T18:08:34.222 INFO:tasks.rados.rados.0.smithi160.stdout:1657: expect (ObjNum 246 snap 0 seq_num 246) 2024-09-21T18:08:34.240 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:34.240 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:34.243 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:34.263 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:34.263 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:34.256+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:34.263 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:34.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:34.268 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:34.297 INFO:tasks.rados.rados.0.smithi160.stdout:1658: finishing write tid 3 to smithi16030472-9 2024-09-21T18:08:34.297 INFO:tasks.rados.rados.0.smithi160.stdout:1658: finishing write tid 4 to smithi16030472-9 2024-09-21T18:08:34.297 INFO:tasks.rados.rados.0.smithi160.stdout:1658: oid 9 updating version 0 to 645 2024-09-21T18:08:34.297 INFO:tasks.rados.rados.0.smithi160.stdout:1658: oid 9 version 645 is already newer than 644 2024-09-21T18:08:34.298 INFO:tasks.rados.rados.0.smithi160.stdout:1658: oid 9 version 645 is already newer than 643 2024-09-21T18:08:34.298 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 9 v 645 (ObjNum 874 snap 0 seq_num 874) dirty exists 2024-09-21T18:08:34.298 INFO:tasks.rados.rados.0.smithi160.stdout:1658: left oid 9 (ObjNum 874 snap 0 seq_num 874) 2024-09-21T18:08:34.302 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:34.302 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:34.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.302 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(client.4876.0:4952 3.0 e79/31)", 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:34.231623+0000", 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.069922619000000005, 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.069954919000000004, 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "no flag points reached", 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231623+0000", 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231623+0000", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231625+0000", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.719e-06 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231631+0000", 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0170000000000002e-06 2024-09-21T18:08:34.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231645+0000", 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.383e-05 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:4952 3.0 3:2cc490fa:::smithi16030472-77:head [delete] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e80)", 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:34.229571+0000", 2024-09-21T18:08:34.305 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.071974426999999994, 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.072105156000000004, 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229571+0000", 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229571+0000", 2024-09-21T18:08:34.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229575+0000", 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7349999999999998e-06 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229592+0000", 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6929999999999999e-05 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229615+0000", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3056000000000001e-05 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229629+0000", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4139e-05 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229717+0000", 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7526000000000002e-05 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229848+0000", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013096200000000001 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.229937+0000", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9130999999999998e-05 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:34.231943+0000", 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0020062220000000002 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.309 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.310 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.310 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:08:34.310 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 2 2024-09-21T18:08:34.310 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:34.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:34.329 INFO:tasks.rados.rados.0.smithi160.stdout:1662: finishing write tid 1 to smithi16030472-87 2024-09-21T18:08:34.329 INFO:tasks.rados.rados.0.smithi160.stdout:1662: finishing write tid 2 to smithi16030472-87 2024-09-21T18:08:34.329 INFO:tasks.rados.rados.0.smithi160.stdout:1662: finishing write tid 3 to smithi16030472-87 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:1662: finishing write tid 4 to smithi16030472-87 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:1662: oid 87 updating version 0 to 646 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:1662: oid 87 updating version 646 to 647 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:1662: oid 87 updating version 647 to 648 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 87 v 648 (ObjNum 876 snap 0 seq_num 876) dirty exists 2024-09-21T18:08:34.330 INFO:tasks.rados.rados.0.smithi160.stdout:1662: left oid 87 (ObjNum 876 snap 0 seq_num 876) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1657: done (4 left) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1658: done (3 left) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1662: done (2 left) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1665: done (1 left) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1666: done (0 left) 2024-09-21T18:08:34.333 INFO:tasks.rados.rados.0.smithi160.stdout:1667: delete oid 170 current snap is 0 2024-09-21T18:08:34.363 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:34.360+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:34.438 INFO:tasks.rados.rados.0.smithi160.stdout:1667: done (0 left) 2024-09-21T18:08:34.438 INFO:tasks.rados.rados.0.smithi160.stdout:1668: read oid 242 snap -1 2024-09-21T18:08:34.439 INFO:tasks.rados.rados.0.smithi160.stdout:1668: expect deleted 2024-09-21T18:08:34.439 INFO:tasks.rados.rados.0.smithi160.stdout:1669: copy_from oid 31 from oid 86 current snap is 0 2024-09-21T18:08:34.439 INFO:tasks.rados.rados.0.smithi160.stdout:1670: write (excl) oid 26 current snap is 0 2024-09-21T18:08:34.439 INFO:tasks.rados.rados.0.smithi160.stdout:1670: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-21T18:08:34.451 INFO:tasks.rados.rados.0.smithi160.stdout:1670: writing smithi16030472-26 from 406967 to 881931 tid 1 2024-09-21T18:08:34.463 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:34.458+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:34.468 INFO:tasks.rados.rados.0.smithi160.stdout:1670: writing smithi16030472-26 from 1370989 to 2140299 tid 2 2024-09-21T18:08:34.478 INFO:tasks.rados.rados.0.smithi160.stdout:1670: writing smithi16030472-26 from 2677186 to 2806967 tid 3 2024-09-21T18:08:34.478 INFO:tasks.rados.rados.0.smithi160.stdout:1671: copy_from oid 282 from oid 28 current snap is 0 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1672: read oid 146 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1672: expect (ObjNum 725 snap 0 seq_num 725) 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1673: cache_flush oid 270 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1674: read oid 360 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1674: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1675: cache_evict oid 487 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1676: cache_flush oid 346 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1677: cache_try_flush oid 99 snap -1 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1678: copy_from oid 459 from oid 355 current snap is 0 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1679: write oid 21 current snap is 0 2024-09-21T18:08:34.479 INFO:tasks.rados.rados.0.smithi160.stdout:1679: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-21T18:08:34.488 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:34.488 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:34.488 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:34.488 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:34.488 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:34.493 INFO:tasks.rados.rados.0.smithi160.stdout:1679: writing smithi16030472-21 from 423774 to 1090340 tid 1 2024-09-21T18:08:34.503 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:34.503 INFO:tasks.rados.rados.0.smithi160.stdout:1679: writing smithi16030472-21 from 1545824 to 2058792 tid 2 2024-09-21T18:08:34.510 INFO:tasks.rados.rados.0.smithi160.stdout:1679: writing smithi16030472-21 from 2704609 to 2823774 tid 3 2024-09-21T18:08:34.511 INFO:tasks.rados.rados.0.smithi160.stdout:1680: delete oid 444 current snap is 0 2024-09-21T18:08:34.511 INFO:tasks.rados.rados.0.smithi160.stdout:1676: got (0) Success 2024-09-21T18:08:34.511 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 346 v 0 (ObjNum 748 snap 0 seq_num 748) dirty exists 2024-09-21T18:08:34.564 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:34.557+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:34.602 INFO:tasks.rados.rados.0.smithi160.stdout:1674: expect (ObjNum 359 snap 0 seq_num 359) 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.595202145000002, 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:08:34.638 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:08:34.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:08:34.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594962520999999, 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:08:34.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:08:34.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:34.649 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594927114000001, 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:08:34.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594893941000002, 2024-09-21T18:08:34.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.658 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594864381000001, 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594834724999998, 2024-09-21T18:08:34.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594348469000003, 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:08:34.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.594314875999999, 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:08:34.673 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.677 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.494097183999997, 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.493498279999997, 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.683 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:08:34.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:08:34.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:08:34.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.492783662999997, 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:08:34.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:08:34.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:08:34.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:08:34.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.483530930999997, 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.693 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:08:34.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:34.698 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.483266229999998, 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:08:34.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.702 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.483171540999997, 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:08:34.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:08:34.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.483130250999999, 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.707 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:08:34.711 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.471860348, 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.462744115, 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.717 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:34.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.722 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.461680747999999, 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:08:34.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:08:34.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.726 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:34.727 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.461640438000003, 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:34.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:08:34.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:08:34.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.699002596, 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:08:34.732 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:08:34.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:08:34.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:34.736 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:34.738 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 186934 2024-09-21T18:08:34.738 INFO:tasks.rados.rados.0.smithi160.stdout:1672: expect (ObjNum 725 snap 0 seq_num 725) 2024-09-21T18:08:34.738 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:34.738 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:34.739 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:34.730+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:34.755 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 398143 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1675: got (16) Device or resource busy 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1671: finishing copy_from to smithi16030472-282 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 282 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1671: finishing copy_from racing read to smithi16030472-282 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1679: finishing write tid 1 to smithi16030472-21 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1669: finishing copy_from to smithi16030472-31 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 434 (ObjNum 571 snap 0 seq_num 571) dirty exists 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1677: got (0) Success 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 0 (ObjNum 829 snap 0 seq_num 829) dirty exists 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:1673: got (0) Success 2024-09-21T18:08:34.776 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 270 v 0 (ObjNum 682 snap 0 seq_num 682) dirty exists 2024-09-21T18:08:34.777 INFO:tasks.rados.rados.0.smithi160.stdout:1669: finishing copy_from racing read to smithi16030472-31 2024-09-21T18:08:34.817 INFO:tasks.rados.rados.0.smithi160.stdout:1679: finishing write tid 2 to smithi16030472-21 2024-09-21T18:08:34.817 INFO:tasks.rados.rados.0.smithi160.stdout:1679: finishing write tid 3 to smithi16030472-21 2024-09-21T18:08:34.839 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:34.834+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:34.840 INFO:tasks.rados.rados.0.smithi160.stdout:1670: finishing write tid 1 to smithi16030472-26 2024-09-21T18:08:34.840 INFO:tasks.rados.rados.0.smithi160.stdout:1670: finishing write tid 2 to smithi16030472-26 2024-09-21T18:08:34.841 INFO:tasks.rados.rados.0.smithi160.stdout:1670: finishing write tid 3 to smithi16030472-26 2024-09-21T18:08:34.873 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:34.874 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:34.874 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:34.889 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:34.920 INFO:tasks.rados.rados.0.smithi160.stdout:1679: finishing write tid 4 to smithi16030472-21 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: finishing write tid 5 to smithi16030472-21 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: oid 21 updating version 0 to 591 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: oid 21 updating version 591 to 592 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: oid 21 updating version 592 to 593 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: oid 21 version 593 is already newer than 590 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 593 (ObjNum 882 snap 0 seq_num 882) dirty exists 2024-09-21T18:08:34.921 INFO:tasks.rados.rados.0.smithi160.stdout:1679: left oid 21 (ObjNum 882 snap 0 seq_num 882) 2024-09-21T18:08:34.940 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:34.936+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:34.941 INFO:tasks.rados.rados.0.smithi160.stdout:1670: finishing write tid 4 to smithi16030472-26 2024-09-21T18:08:35.016 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.016 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:35.016 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:35.016 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:35.016 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:35.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:1670: finishing write tid 5 to smithi16030472-26 2024-09-21T18:08:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:1670: oid 26 updating version 0 to 649 2024-09-21T18:08:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:1670: oid 26 updating version 649 to 651 2024-09-21T18:08:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:1670: oid 26 updating version 651 to 652 2024-09-21T18:08:35.043 INFO:tasks.rados.rados.0.smithi160.stdout:1670: oid 26 version 652 is already newer than 650 2024-09-21T18:08:35.043 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 652 (ObjNum 881 snap 0 seq_num 881) dirty exists 2024-09-21T18:08:35.043 INFO:tasks.rados.rados.0.smithi160.stdout:1670: left oid 26 (ObjNum 881 snap 0 seq_num 881) 2024-09-21T18:08:35.043 INFO:tasks.rados.rados.0.smithi160.stdout:1678: finishing copy_from to smithi16030472-459 2024-09-21T18:08:35.043 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 252 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-09-21T18:08:35.052 INFO:tasks.rados.rados.0.smithi160.stdout:1678: finishing copy_from racing read to smithi16030472-459 2024-09-21T18:08:35.075 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:35.072+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:35.125 INFO:tasks.rados.rados.0.smithi160.stdout:1668: done (12 left) 2024-09-21T18:08:35.125 INFO:tasks.rados.rados.0.smithi160.stdout:1669: done (11 left) 2024-09-21T18:08:35.125 INFO:tasks.rados.rados.0.smithi160.stdout:1670: done (10 left) 2024-09-21T18:08:35.125 INFO:tasks.rados.rados.0.smithi160.stdout:1671: done (9 left) 2024-09-21T18:08:35.125 INFO:tasks.rados.rados.0.smithi160.stdout:1672: done (8 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1673: done (7 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1674: done (6 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1675: done (5 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1676: done (4 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1677: done (3 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1678: done (2 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1679: done (1 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1680: done (0 left) 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1681: cache_try_flush oid 7 snap -1 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1682: write (excl) oid 123 current snap is 0 2024-09-21T18:08:35.126 INFO:tasks.rados.rados.0.smithi160.stdout:1682: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-09-21T18:08:35.132 INFO:tasks.rados.rados.0.smithi160.stdout:1682: writing smithi16030472-123 from 457388 to 874452 tid 1 2024-09-21T18:08:35.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:35.136+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:35.143 INFO:tasks.rados.rados.0.smithi160.stdout:1682: writing smithi16030472-123 from 1546435 to 2063072 tid 2 2024-09-21T18:08:35.149 INFO:tasks.rados.rados.0.smithi160.stdout:1682: writing smithi16030472-123 from 2643102 to 2857388 tid 3 2024-09-21T18:08:35.150 INFO:tasks.rados.rados.0.smithi160.stdout:1683: write oid 80 current snap is 0 2024-09-21T18:08:35.150 INFO:tasks.rados.rados.0.smithi160.stdout:1683: seq_num 885 ranges {474195=492313,1604917=660295,2670525=203670} 2024-09-21T18:08:35.158 INFO:tasks.rados.rados.0.smithi160.stdout:1683: writing smithi16030472-80 from 474195 to 966508 tid 1 2024-09-21T18:08:35.160 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.160 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 110.858088827, 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:08:35.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 110.856874768, 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:08:35.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.168 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:08:35.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 110.855523848, 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.170 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:08:35.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:08:35.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 104.567943211, 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:08:35.175 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:08:35.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 82.469350581, 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:08:35.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:08:35.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 82.468873371000001, 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.186 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:08:35.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.190 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 82.468438418000005, 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.168703970999999, 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:35.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:08:35.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:08:35.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:08:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.068134579000002, 2024-09-21T18:08:35.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:08:35.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.067776511999995, 2024-09-21T18:08:35.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:08:35.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.067739678999999, 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:08:35.213 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:08:35.218 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.067279947000003, 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:08:35.222 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.066848419999999, 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.065911804999999, 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:08:35.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:08:35.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.065873121999999, 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:08:35.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:08:35.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.238 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.064586469999995, 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:08:35.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.243 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.962442006000003, 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:08:35.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:08:35.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.961718212999997, 2024-09-21T18:08:35.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.252 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 55.836007973999997, 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.253 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 31.196213487000001, 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:08:35.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.262 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:35.264 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:35.265 INFO:tasks.rados.rados.0.smithi160.stdout:1683: writing smithi16030472-80 from 1604917 to 2265212 tid 2 2024-09-21T18:08:35.265 INFO:tasks.rados.rados.0.smithi160.stdout:1683: writing smithi16030472-80 from 2670525 to 2874195 tid 3 2024-09-21T18:08:35.265 INFO:tasks.rados.rados.0.smithi160.stdout:1684: cache_flush oid 477 snap -1 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1685: read oid 105 snap -1 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1685: expect deleted 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1686: write oid 371 current snap is 0 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1686: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1686: writing smithi16030472-371 from 491002 to 1058564 tid 1 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1686: writing smithi16030472-371 from 1663399 to 2183705 tid 2 2024-09-21T18:08:35.266 INFO:tasks.rados.rados.0.smithi160.stdout:1686: writing smithi16030472-371 from 2814301 to 2891002 tid 3 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1687: write oid 62 current snap is 0 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1687: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1687: writing smithi16030472-62 from 507809 to 1150620 tid 1 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1687: writing smithi16030472-62 from 1721881 to 2385845 tid 2 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1688: read oid 201 snap -1 2024-09-21T18:08:35.267 INFO:tasks.rados.rados.0.smithi160.stdout:1688: expect (ObjNum 794 snap 0 seq_num 794) 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1689: copy_from oid 289 from oid 499 current snap is 0 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1690: read oid 180 snap -1 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1690: expect (ObjNum 591 snap 0 seq_num 591) 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1691: read oid 385 snap -1 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1691: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:35.268 INFO:tasks.rados.rados.0.smithi160.stdout:1692: delete oid 392 current snap is 0 2024-09-21T18:08:35.268 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:35.257+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:35.364 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:35.358+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:35.372 INFO:tasks.rados.rados.0.smithi160.stdout:1684: got (0) Success 2024-09-21T18:08:35.372 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 477 v 0 (ObjNum 3 snap 0 seq_num 304) clean dne 2024-09-21T18:08:35.440 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:35.441 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:35.441 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:35.441 INFO:tasks.rados.rados.0.smithi160.stdout:1682: finishing write tid 1 to smithi16030472-123 2024-09-21T18:08:35.442 INFO:tasks.rados.rados.0.smithi160.stdout:1681: got (0) Success 2024-09-21T18:08:35.442 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-21T18:08:35.442 INFO:tasks.rados.rados.0.smithi160.stdout:1682: finishing write tid 2 to smithi16030472-123 2024-09-21T18:08:35.444 INFO:tasks.rados.rados.0.smithi160.stdout:1682: finishing write tid 3 to smithi16030472-123 2024-09-21T18:08:35.457 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:35.485 INFO:tasks.rados.rados.0.smithi160.stdout:1691: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:35.485 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:35.503 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:35.500+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:35.566 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:35.560+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:35.585 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:35.585 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:35.585 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:35.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:35.602 INFO:tasks.rados.rados.0.smithi160.stdout:1683: finishing write tid 1 to smithi16030472-80 2024-09-21T18:08:35.602 INFO:tasks.rados.rados.0.smithi160.stdout:1686: finishing write tid 1 to smithi16030472-371 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1686: finishing write tid 2 to smithi16030472-371 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1686: finishing write tid 3 to smithi16030472-371 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: finishing write tid 4 to smithi16030472-123 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1687: finishing write tid 1 to smithi16030472-62 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: finishing write tid 5 to smithi16030472-123 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: oid 123 updating version 0 to 651 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: oid 123 updating version 651 to 652 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: oid 123 version 652 is already newer than 650 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: oid 123 updating version 652 to 653 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 653 (ObjNum 884 snap 0 seq_num 884) dirty exists 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1682: left oid 123 (ObjNum 884 snap 0 seq_num 884) 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1686: finishing write tid 4 to smithi16030472-371 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1686: finishing write tid 5 to smithi16030472-371 2024-09-21T18:08:35.603 INFO:tasks.rados.rados.0.smithi160.stdout:1686: oid 371 updating version 0 to 596 2024-09-21T18:08:35.604 INFO:tasks.rados.rados.0.smithi160.stdout:1686: oid 371 version 596 is already newer than 594 2024-09-21T18:08:35.604 INFO:tasks.rados.rados.0.smithi160.stdout:1686: oid 371 version 596 is already newer than 595 2024-09-21T18:08:35.604 INFO:tasks.rados.rados.0.smithi160.stdout:1686: oid 371 updating version 596 to 597 2024-09-21T18:08:35.604 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 597 (ObjNum 886 snap 0 seq_num 886) dirty exists 2024-09-21T18:08:35.604 INFO:tasks.rados.rados.0.smithi160.stdout:1686: left oid 371 (ObjNum 886 snap 0 seq_num 886) 2024-09-21T18:08:35.653 INFO:tasks.rados.rados.0.smithi160.stdout:1687: finishing write tid 2 to smithi16030472-62 2024-09-21T18:08:35.666 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:35.661+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:35.667 INFO:tasks.rados.rados.0.smithi160.stdout:1683: finishing write tid 2 to smithi16030472-80 2024-09-21T18:08:35.763 INFO:tasks.rados.rados.0.smithi160.stdout:1687: finishing write tid 3 to smithi16030472-62 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:1687: finishing write tid 4 to smithi16030472-62 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:1687: oid 62 updating version 0 to 655 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:1687: oid 62 version 655 is already newer than 654 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:1687: oid 62 updating version 655 to 656 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 62 v 656 (ObjNum 887 snap 0 seq_num 887) dirty exists 2024-09-21T18:08:35.764 INFO:tasks.rados.rados.0.smithi160.stdout:1687: left oid 62 (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:08:35.767 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:35.764+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:35.768 INFO:tasks.rados.rados.0.smithi160.stdout:1683: finishing write tid 3 to smithi16030472-80 2024-09-21T18:08:35.768 INFO:tasks.rados.rados.0.smithi160.stdout:1683: finishing write tid 4 to smithi16030472-80 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.771 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.737349375999997, 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:08:35.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:08:35.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.733738825000003, 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:08:35.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:08:35.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:08:35.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.730932285999998, 2024-09-21T18:08:35.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:08:35.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:08:35.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.727184085999994, 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:08:35.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.788 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.106333281999994, 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.005479210000004, 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:08:35.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:08:35.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:08:35.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.00472293, 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:08:35.797 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:08:35.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.004331372999999, 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:08:35.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:08:35.805 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.004030248999996, 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 64.003306398999996, 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.810 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.814 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.902409560000002, 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.815 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:08:35.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:08:35.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.902315277, 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:08:35.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:08:35.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:08:35.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.901402081999997, 2024-09-21T18:08:35.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:08:35.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.827 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.900832393999998, 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.828 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.829 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.831 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.900104910000003, 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:08:35.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:08:35.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 63.833923001000002, 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:08:35.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:08:35.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:08:35.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.577877196000003, 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:08:35.840 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:08:35.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:08:35.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.577594398000002, 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:08:35.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:08:35.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:08:35.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.577515413, 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:08:35.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:08:35.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 62.574897082, 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:08:35.853 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:35.857 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:35.858 INFO:tasks.rados.rados.0.smithi160.stdout:1688: expect (ObjNum 794 snap 0 seq_num 794) 2024-09-21T18:08:35.858 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3308301 2024-09-21T18:08:35.859 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:35.859 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:35.868 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:35.866+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 371606 2024-09-21T18:08:35.916 INFO:tasks.rados.rados.0.smithi160.stdout:1690: expect (ObjNum 591 snap 0 seq_num 591) 2024-09-21T18:08:35.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:08:35.989 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:35.990 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:35.990 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:36.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:36.007 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:36.002+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:36.056 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2084799 2024-09-21T18:08:36.070 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:36.064+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1689: finishing copy_from to smithi16030472-289 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1689: got expected ENOENT (src dne) 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: finishing write tid 5 to smithi16030472-80 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: oid 80 updating version 0 to 554 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: oid 80 updating version 554 to 555 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: oid 80 updating version 555 to 556 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: oid 80 version 556 is already newer than 553 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 556 (ObjNum 885 snap 0 seq_num 885) dirty exists 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1683: left oid 80 (ObjNum 885 snap 0 seq_num 885) 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1689: finishing copy_from racing read to smithi16030472-289 2024-09-21T18:08:36.085 INFO:tasks.rados.rados.0.smithi160.stdout:1681: done (11 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1682: done (10 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1683: done (9 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1684: done (8 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1685: done (7 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1686: done (6 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1687: done (5 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1688: done (4 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1689: done (3 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1690: done (2 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1691: done (1 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1692: done (0 left) 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1693: cache_flush oid 484 snap -1 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1694: write (excl) oid 221 current snap is 0 2024-09-21T18:08:36.086 INFO:tasks.rados.rados.0.smithi160.stdout:1694: seq_num 889 ranges {541423=793309,2122492=667633} 2024-09-21T18:08:36.092 INFO:tasks.rados.rados.0.smithi160.stdout:1694: writing smithi16030472-221 from 541423 to 1334732 tid 1 2024-09-21T18:08:36.100 INFO:tasks.rados.rados.0.smithi160.stdout:1694: writing smithi16030472-221 from 2122492 to 2790125 tid 2 2024-09-21T18:08:36.100 INFO:tasks.rados.rados.0.smithi160.stdout:1695: delete oid 352 current snap is 0 2024-09-21T18:08:36.131 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:36.131 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:36.132 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:36.132 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:36.146 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:36.171 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:36.165+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:36.186 INFO:tasks.rados.rados.0.smithi160.stdout:1693: got (0) Success 2024-09-21T18:08:36.186 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 0 (ObjNum 129 snap 0 seq_num 3128926872) clean dne 2024-09-21T18:08:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:1693: done (2 left) 2024-09-21T18:08:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:1695: done (1 left) 2024-09-21T18:08:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:1696: cache_try_flush oid 42 snap -1 2024-09-21T18:08:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:1697: write (excl) oid 87 current snap is 0 2024-09-21T18:08:36.204 INFO:tasks.rados.rados.0.smithi160.stdout:1697: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-21T18:08:36.215 INFO:tasks.rados.rados.0.smithi160.stdout:1697: writing smithi16030472-87 from 575037 to 1235197 tid 1 2024-09-21T18:08:36.227 INFO:tasks.rados.rados.0.smithi160.stdout:1697: writing smithi16030472-87 from 1955809 to 2627111 tid 2 2024-09-21T18:08:36.227 INFO:tasks.rados.rados.0.smithi160.stdout:1698: cache_flush oid 114 snap -1 2024-09-21T18:08:36.227 INFO:tasks.rados.rados.0.smithi160.stdout:1699: write (excl) oid 388 current snap is 0 2024-09-21T18:08:36.227 INFO:tasks.rados.rados.0.smithi160.stdout:1699: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-21T18:08:36.238 INFO:tasks.rados.rados.0.smithi160.stdout:1699: writing smithi16030472-388 from 591844 to 1327253 tid 1 2024-09-21T18:08:36.247 INFO:tasks.rados.rados.0.smithi160.stdout:1699: writing smithi16030472-388 from 1897938 to 2429251 tid 2 2024-09-21T18:08:36.252 INFO:tasks.rados.rados.0.smithi160.stdout:1699: writing smithi16030472-388 from 2862486 to 2991844 tid 3 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:1700: read oid 121 snap -1 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:1700: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:1701: write oid 252 current snap is 0 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:1698: got (0) Success 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 0 (ObjNum 389 snap 0 seq_num 389) dirty exists 2024-09-21T18:08:36.253 INFO:tasks.rados.rados.0.smithi160.stdout:1701: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2024-09-21T18:08:36.258 INFO:tasks.rados.rados.0.smithi160.stdout:1701: writing smithi16030472-252 from 608651 to 1019309 tid 1 2024-09-21T18:08:36.265 INFO:tasks.rados.rados.0.smithi160.stdout:1701: writing smithi16030472-252 from 1556420 to 2231391 tid 2 2024-09-21T18:08:36.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1701: writing smithi16030472-252 from 2773556 to 3008651 tid 3 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1698: done (6 left) 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1702: cache_try_flush oid 421 snap -1 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1701: finishing write tid 1 to smithi16030472-252 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1703: cache_try_flush oid 105 snap -1 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1704: delete oid 283 current snap is 0 2024-09-21T18:08:36.271 INFO:tasks.rados.rados.0.smithi160.stdout:1702: got (0) Success 2024-09-21T18:08:36.272 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 421 v 0 (ObjNum 420 snap 0 seq_num 420) dirty exists 2024-09-21T18:08:36.272 INFO:tasks.rados.rados.0.smithi160.stdout:1703: got (0) Success 2024-09-21T18:08:36.272 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 105 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:08:36.273 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:36.265+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:36.314 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:36.315 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:08:36.315 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:36.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:36.374 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:36.366+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:36.422 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:36.443 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:36.465 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:36.465 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:36.465 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:36.465 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:36.475 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:36.472+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:36.480 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:36.486 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:36.542 INFO:tasks.rados.rados.0.smithi160.stdout:1694: finishing write tid 1 to smithi16030472-221 2024-09-21T18:08:36.542 INFO:tasks.rados.rados.0.smithi160.stdout:1694: finishing write tid 2 to smithi16030472-221 2024-09-21T18:08:36.542 INFO:tasks.rados.rados.0.smithi160.stdout:1694: finishing write tid 3 to smithi16030472-221 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:1694: finishing write tid 4 to smithi16030472-221 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:1694: oid 221 updating version 0 to 646 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:1694: oid 221 version 646 is already newer than 644 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:1694: oid 221 version 646 is already newer than 645 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 221 v 646 (ObjNum 889 snap 0 seq_num 889) dirty exists 2024-09-21T18:08:36.543 INFO:tasks.rados.rados.0.smithi160.stdout:1694: left oid 221 (ObjNum 889 snap 0 seq_num 889) 2024-09-21T18:08:36.562 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:36.570 INFO:tasks.rados.rados.0.smithi160.stdout:1701: finishing write tid 2 to smithi16030472-252 2024-09-21T18:08:36.571 INFO:tasks.rados.rados.0.smithi160.stdout:1701: finishing write tid 3 to smithi16030472-252 2024-09-21T18:08:36.571 INFO:tasks.rados.rados.0.smithi160.stdout:1701: finishing write tid 4 to smithi16030472-252 2024-09-21T18:08:36.571 INFO:tasks.rados.rados.0.smithi160.stdout:1701: finishing write tid 5 to smithi16030472-252 2024-09-21T18:08:36.571 INFO:tasks.rados.rados.0.smithi160.stdout:1701: oid 252 updating version 0 to 659 2024-09-21T18:08:36.572 INFO:tasks.rados.rados.0.smithi160.stdout:1701: oid 252 updating version 659 to 661 2024-09-21T18:08:36.572 INFO:tasks.rados.rados.0.smithi160.stdout:1701: oid 252 version 661 is already newer than 658 2024-09-21T18:08:36.572 INFO:tasks.rados.rados.0.smithi160.stdout:1701: oid 252 version 661 is already newer than 660 2024-09-21T18:08:36.572 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 252 v 661 (ObjNum 893 snap 0 seq_num 893) dirty exists 2024-09-21T18:08:36.572 INFO:tasks.rados.rados.0.smithi160.stdout:1701: left oid 252 (ObjNum 893 snap 0 seq_num 893) 2024-09-21T18:08:36.575 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:36.569+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:36.587 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:36.588 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:36.615 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:36.615 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:36.616 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:36.642 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:36.642 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:36.650 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:36.650 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:36.664 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:36.668 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:36.668 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:36.677 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:36.669+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:36.695 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:36.695 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:36.721 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:36.722 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:36.724 INFO:tasks.rados.rados.0.smithi160.stdout:1696: got (0) Success 2024-09-21T18:08:36.724 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 0 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:08:36.724 INFO:tasks.rados.rados.0.smithi160.stdout:1697: finishing write tid 1 to smithi16030472-87 2024-09-21T18:08:36.725 INFO:tasks.rados.rados.0.smithi160.stdout:1697: finishing write tid 2 to smithi16030472-87 2024-09-21T18:08:36.725 INFO:tasks.rados.rados.0.smithi160.stdout:1697: finishing write tid 3 to smithi16030472-87 2024-09-21T18:08:36.725 INFO:tasks.rados.rados.0.smithi160.stdout:1697: finishing write tid 4 to smithi16030472-87 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:1697: oid 87 updating version 0 to 654 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:1697: oid 87 version 654 is already newer than 653 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:1697: oid 87 updating version 654 to 655 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 87 v 655 (ObjNum 891 snap 0 seq_num 891) dirty exists 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:1697: left oid 87 (ObjNum 891 snap 0 seq_num 891) 2024-09-21T18:08:36.726 INFO:tasks.rados.rados.0.smithi160.stdout:1699: finishing write tid 1 to smithi16030472-388 2024-09-21T18:08:36.746 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:36.746 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:36.772 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:36.777 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:36.776+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:36.788 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:36.788 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:36.795 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:36.796 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:36.803 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:36.821 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:36.825 INFO:tasks.rados.rados.0.smithi160.stdout:1699: finishing write tid 2 to smithi16030472-388 2024-09-21T18:08:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:36.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:36.879 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:36.873+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:36.925 INFO:tasks.rados.rados.0.smithi160.stdout:1699: finishing write tid 3 to smithi16030472-388 2024-09-21T18:08:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:08:36.973 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:36.973 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:36.980 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:36.974+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:36.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:37.033 INFO:tasks.rados.rados.0.smithi160.stdout:1700: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:37.043 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 605049 2024-09-21T18:08:37.080 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:37.073+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:37.110 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.111 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:37.130 INFO:tasks.rados.rados.0.smithi160.stdout:1699: finishing write tid 4 to smithi16030472-388 2024-09-21T18:08:37.130 INFO:tasks.rados.rados.0.smithi160.stdout:1699: finishing write tid 5 to smithi16030472-388 2024-09-21T18:08:37.130 INFO:tasks.rados.rados.0.smithi160.stdout:1699: oid 388 updating version 0 to 657 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1699: oid 388 updating version 657 to 659 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1699: oid 388 version 659 is already newer than 656 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1699: oid 388 version 659 is already newer than 658 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 388 v 659 (ObjNum 892 snap 0 seq_num 892) dirty exists 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1699: left oid 388 (ObjNum 892 snap 0 seq_num 892) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1694: done (8 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1696: done (7 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1697: done (6 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1699: done (5 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1700: done (4 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1701: done (3 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1702: done (2 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1703: done (1 left) 2024-09-21T18:08:37.131 INFO:tasks.rados.rados.0.smithi160.stdout:1704: done (0 left) 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1705: cache_flush oid 418 snap -1 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1706: copy_from oid 235 from oid 484 current snap is 0 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1707: cache_evict oid 446 snap -1 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1708: copy_from oid 456 from oid 372 current snap is 0 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1709: read oid 248 snap -1 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1709: expect (ObjNum 699 snap 0 seq_num 699) 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1710: write (excl) oid 115 current snap is 0 2024-09-21T18:08:37.132 INFO:tasks.rados.rados.0.smithi160.stdout:1710: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-09-21T18:08:37.132 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:37.125+0000 7fb260c0e640 -1 osd.11 80 handle_osd_map injecting map crc failure 2024-09-21T18:08:37.140 INFO:tasks.rados.rados.0.smithi160.stdout:1710: writing smithi16030472-115 from 642265 to 1203421 tid 1 2024-09-21T18:08:37.150 INFO:tasks.rados.rados.0.smithi160.stdout:1710: writing smithi16030472-115 from 1673384 to 2352024 tid 2 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1710: writing smithi16030472-115 from 2828402 to 3042265 tid 3 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1711: cache_evict oid 410 snap -1 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1705: got (2) No such file or directory 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1705: done (6 left) 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1712: delete oid 385 current snap is 0 2024-09-21T18:08:37.156 INFO:tasks.rados.rados.0.smithi160.stdout:1707: got (16) Device or resource busy 2024-09-21T18:08:37.181 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:37.173+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:37.247 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.247 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:37.282 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:37.277+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:37.345 INFO:tasks.rados.rados.0.smithi160.stdout:1709: expect (ObjNum 699 snap 0 seq_num 699) 2024-09-21T18:08:37.382 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:37.377+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:37.386 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.386 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.400 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:37.407 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2271777 2024-09-21T18:08:37.484 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:37.478+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:37.505 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 297840 2024-09-21T18:08:37.527 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.527 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:1710: finishing write tid 1 to smithi16030472-115 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:1708: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:1708: finishing copy_from to smithi16030472-456 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 615 (ObjNum 802 snap 0 seq_num 802) dirty exists 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:1706: finishing copy_from to smithi16030472-235 2024-09-21T18:08:37.563 INFO:tasks.rados.rados.0.smithi160.stdout:1706: got expected ENOENT (src dne) 2024-09-21T18:08:37.564 INFO:tasks.rados.rados.0.smithi160.stdout:1706: finishing copy_from racing read to smithi16030472-235 2024-09-21T18:08:37.564 INFO:tasks.rados.rados.0.smithi160.stdout:1710: finishing write tid 2 to smithi16030472-115 2024-09-21T18:08:37.584 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:37.577+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:37.657 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.658 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.662 INFO:tasks.rados.rados.0.smithi160.stdout:1711: got (2) No such file or directory 2024-09-21T18:08:37.662 INFO:tasks.rados.rados.0.smithi160.stdout:1710: finishing write tid 3 to smithi16030472-115 2024-09-21T18:08:37.662 INFO:tasks.rados.rados.0.smithi160.stdout:1710: finishing write tid 4 to smithi16030472-115 2024-09-21T18:08:37.662 INFO:tasks.rados.rados.0.smithi160.stdout:1710: finishing write tid 5 to smithi16030472-115 2024-09-21T18:08:37.662 INFO:tasks.rados.rados.0.smithi160.stdout:1710: oid 115 updating version 0 to 559 2024-09-21T18:08:37.663 INFO:tasks.rados.rados.0.smithi160.stdout:1710: oid 115 version 559 is already newer than 557 2024-09-21T18:08:37.663 INFO:tasks.rados.rados.0.smithi160.stdout:1710: oid 115 updating version 559 to 560 2024-09-21T18:08:37.663 INFO:tasks.rados.rados.0.smithi160.stdout:1710: oid 115 version 560 is already newer than 558 2024-09-21T18:08:37.663 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 115 v 560 (ObjNum 895 snap 0 seq_num 895) dirty exists 2024-09-21T18:08:37.663 INFO:tasks.rados.rados.0.smithi160.stdout:1710: left oid 115 (ObjNum 895 snap 0 seq_num 895) 2024-09-21T18:08:37.670 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:37.687 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:37.680+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:37.760 INFO:tasks.rados.rados.0.smithi160.stdout:1706: done (6 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1707: done (5 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1708: done (4 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1709: done (3 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1710: done (2 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1711: done (1 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1712: done (0 left) 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1713: write (excl) oid 118 current snap is 0 2024-09-21T18:08:37.761 INFO:tasks.rados.rados.0.smithi160.stdout:1713: seq_num 897 ranges {675879=711654,2073995=682309} 2024-09-21T18:08:37.774 INFO:tasks.rados.rados.0.smithi160.stdout:1713: writing smithi16030472-118 from 675879 to 1387533 tid 1 2024-09-21T18:08:37.786 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:37.781+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:37.788 INFO:tasks.rados.rados.0.smithi160.stdout:1713: writing smithi16030472-118 from 2073995 to 2756304 tid 2 2024-09-21T18:08:37.788 INFO:tasks.rados.rados.0.smithi160.stdout:1714: write (excl) oid 124 current snap is 0 2024-09-21T18:08:37.788 INFO:tasks.rados.rados.0.smithi160.stdout:1714: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-21T18:08:37.792 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.793 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.800 INFO:tasks.rados.rados.0.smithi160.stdout:1714: writing smithi16030472-124 from 692686 to 1195942 tid 1 2024-09-21T18:08:37.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:37.810 INFO:tasks.rados.rados.0.smithi160.stdout:1714: writing smithi16030472-124 from 1848830 to 2274797 tid 2 2024-09-21T18:08:37.818 INFO:tasks.rados.rados.0.smithi160.stdout:1714: writing smithi16030472-124 from 2910671 to 3092686 tid 3 2024-09-21T18:08:37.818 INFO:tasks.rados.rados.0.smithi160.stdout:1715: delete oid 299 current snap is 0 2024-09-21T18:08:37.887 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:37.882+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:37.912 INFO:tasks.rados.rados.0.smithi160.stdout:1714: finishing write tid 1 to smithi16030472-124 2024-09-21T18:08:37.918 INFO:tasks.rados.rados.0.smithi160.stdout:1714: finishing write tid 2 to smithi16030472-124 2024-09-21T18:08:37.919 INFO:tasks.rados.rados.0.smithi160.stdout:1714: finishing write tid 3 to smithi16030472-124 2024-09-21T18:08:37.921 INFO:tasks.rados.rados.0.smithi160.stdout:1715: done (2 left) 2024-09-21T18:08:37.922 INFO:tasks.rados.rados.0.smithi160.stdout:1716: read oid 368 snap -1 2024-09-21T18:08:37.922 INFO:tasks.rados.rados.0.smithi160.stdout:1716: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-21T18:08:37.922 INFO:tasks.rados.rados.0.smithi160.stdout:1717: cache_try_flush oid 280 snap -1 2024-09-21T18:08:37.922 INFO:tasks.rados.rados.0.smithi160.stdout:1718: write oid 49 current snap is 0 2024-09-21T18:08:37.922 INFO:tasks.rados.rados.0.smithi160.stdout:1718: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-09-21T18:08:37.933 INFO:tasks.rados.rados.0.smithi160.stdout:1718: writing smithi16030472-49 from 726300 to 1380054 tid 1 2024-09-21T18:08:37.934 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:37.934 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:37.942 INFO:tasks.rados.rados.0.smithi160.stdout:1718: writing smithi16030472-49 from 1965794 to 2395430 tid 2 2024-09-21T18:08:37.949 INFO:tasks.rados.rados.0.smithi160.stdout:1718: writing smithi16030472-49 from 2965517 to 3126300 tid 3 2024-09-21T18:08:37.949 INFO:tasks.rados.rados.0.smithi160.stdout:1719: cache_flush oid 182 snap -1 2024-09-21T18:08:37.949 INFO:tasks.rados.rados.0.smithi160.stdout:1720: cache_try_flush oid 487 snap -1 2024-09-21T18:08:37.949 INFO:tasks.rados.rados.0.smithi160.stdout:1721: copy_from oid 26 from oid 352 current snap is 0 2024-09-21T18:08:37.949 INFO:tasks.rados.rados.0.smithi160.stdout:1722: delete oid 393 current snap is 0 2024-09-21T18:08:37.949 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:37.987 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:37.981+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:1714: finishing write tid 4 to smithi16030472-124 2024-09-21T18:08:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:1714: finishing write tid 5 to smithi16030472-124 2024-09-21T18:08:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:1714: oid 124 updating version 0 to 663 2024-09-21T18:08:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:1714: oid 124 version 663 is already newer than 662 2024-09-21T18:08:38.020 INFO:tasks.rados.rados.0.smithi160.stdout:1714: oid 124 updating version 663 to 664 2024-09-21T18:08:38.021 INFO:tasks.rados.rados.0.smithi160.stdout:1714: oid 124 updating version 664 to 665 2024-09-21T18:08:38.021 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 124 v 665 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:08:38.021 INFO:tasks.rados.rados.0.smithi160.stdout:1714: left oid 124 (ObjNum 898 snap 0 seq_num 898) 2024-09-21T18:08:38.021 INFO:tasks.rados.rados.0.smithi160.stdout:1717: got (2) No such file or directory 2024-09-21T18:08:38.032 INFO:tasks.rados.rados.0.smithi160.stdout:1718: finishing write tid 1 to smithi16030472-49 2024-09-21T18:08:38.032 INFO:tasks.rados.rados.0.smithi160.stdout:1718: finishing write tid 2 to smithi16030472-49 2024-09-21T18:08:38.053 INFO:tasks.rados.rados.0.smithi160.stdout:1721: finishing copy_from to smithi16030472-26 2024-09-21T18:08:38.053 INFO:tasks.rados.rados.0.smithi160.stdout:1721: got expected ENOENT (src dne) 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1721: finishing copy_from racing read to smithi16030472-26 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1714: done (8 left) 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1717: done (7 left) 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1721: done (6 left) 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1722: done (5 left) 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1723: cache_evict oid 469 snap -1 2024-09-21T18:08:38.054 INFO:tasks.rados.rados.0.smithi160.stdout:1724: read oid 410 snap -1 2024-09-21T18:08:38.055 INFO:tasks.rados.rados.0.smithi160.stdout:1724: expect deleted 2024-09-21T18:08:38.055 INFO:tasks.rados.rados.0.smithi160.stdout:1725: write oid 415 current snap is 0 2024-09-21T18:08:38.055 INFO:tasks.rados.rados.0.smithi160.stdout:1725: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-09-21T18:08:38.066 INFO:tasks.rados.rados.0.smithi160.stdout:1725: writing smithi16030472-415 from 759914 to 1164166 tid 1 2024-09-21T18:08:38.073 INFO:tasks.rados.rados.0.smithi160.stdout:1725: writing smithi16030472-415 from 1566405 to 1999710 tid 2 2024-09-21T18:08:38.076 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.076 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1725: writing smithi16030472-415 from 2504010 to 3159914 tid 3 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1726: cache_flush oid 97 snap -1 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1727: read oid 214 snap -1 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1727: expect deleted 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1728: read oid 149 snap -1 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1728: expect (ObjNum 789 snap 0 seq_num 789) 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1729: delete oid 185 current snap is 0 2024-09-21T18:08:38.084 INFO:tasks.rados.rados.0.smithi160.stdout:1723: got (0) Success 2024-09-21T18:08:38.088 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:38.084+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:38.092 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:38.096 INFO:tasks.rados.rados.0.smithi160.stdout:1728: expect (ObjNum 789 snap 0 seq_num 789) 2024-09-21T18:08:38.168 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:38.189 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:38.184+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:38.222 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.222 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.224 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:38.218+0000 7f094d1de640 -1 osd.0 80 handle_osd_map injecting map crc failure 2024-09-21T18:08:38.236 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1542179 2024-09-21T18:08:38.237 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1719: got (0) Success 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 182 v 0 (ObjNum 676 snap 0 seq_num 676) dirty exists 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: finishing write tid 3 to smithi16030472-49 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: finishing write tid 4 to smithi16030472-49 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: finishing write tid 5 to smithi16030472-49 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: oid 49 updating version 0 to 667 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: oid 49 updating version 667 to 668 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: oid 49 version 668 is already newer than 666 2024-09-21T18:08:38.280 INFO:tasks.rados.rados.0.smithi160.stdout:1718: oid 49 updating version 668 to 669 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 669 (ObjNum 900 snap 0 seq_num 900) dirty exists 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:1718: left oid 49 (ObjNum 900 snap 0 seq_num 900) 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:1720: got (0) Success 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 487 v 0 (ObjNum 570 snap 0 seq_num 570) dirty exists 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:1713: finishing write tid 1 to smithi16030472-118 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:1713: finishing write tid 2 to smithi16030472-118 2024-09-21T18:08:38.281 INFO:tasks.rados.rados.0.smithi160.stdout:1725: finishing write tid 1 to smithi16030472-415 2024-09-21T18:08:38.288 INFO:tasks.rados.rados.0.smithi160.stdout:1713: finishing write tid 3 to smithi16030472-118 2024-09-21T18:08:38.288 INFO:tasks.rados.rados.0.smithi160.stdout:1713: finishing write tid 4 to smithi16030472-118 2024-09-21T18:08:38.288 INFO:tasks.rados.rados.0.smithi160.stdout:1713: oid 118 updating version 0 to 648 2024-09-21T18:08:38.288 INFO:tasks.rados.rados.0.smithi160.stdout:1713: oid 118 updating version 648 to 649 2024-09-21T18:08:38.288 INFO:tasks.rados.rados.0.smithi160.stdout:1713: oid 118 version 649 is already newer than 647 2024-09-21T18:08:38.289 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 118 v 649 (ObjNum 897 snap 0 seq_num 897) dirty exists 2024-09-21T18:08:38.289 INFO:tasks.rados.rados.0.smithi160.stdout:1713: left oid 118 (ObjNum 897 snap 0 seq_num 897) 2024-09-21T18:08:38.289 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:38.282+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:38.304 INFO:tasks.rados.rados.0.smithi160.stdout:1726: got (0) Success 2024-09-21T18:08:38.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 97 v 0 (ObjNum 185 snap 0 seq_num 185) dirty exists 2024-09-21T18:08:38.390 INFO:tasks.rados.rados.0.smithi160.stdout:1716: expect (ObjNum 367 snap 0 seq_num 367) 2024-09-21T18:08:38.390 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 425656 2024-09-21T18:08:38.391 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:38.388+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:38.397 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.397 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:38.418 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:38.458 INFO:tasks.rados.rados.0.smithi160.stdout:1713: done (11 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1716: done (10 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1718: done (9 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1719: done (8 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1720: done (7 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1723: done (6 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1724: done (5 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1726: done (4 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1727: done (3 left) 2024-09-21T18:08:38.459 INFO:tasks.rados.rados.0.smithi160.stdout:1728: done (2 left) 2024-09-21T18:08:38.460 INFO:tasks.rados.rados.0.smithi160.stdout:1729: done (1 left) 2024-09-21T18:08:38.460 INFO:tasks.rados.rados.0.smithi160.stdout:1730: write (excl) oid 60 current snap is 0 2024-09-21T18:08:38.460 INFO:tasks.rados.rados.0.smithi160.stdout:1730: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-09-21T18:08:38.470 INFO:tasks.rados.rados.0.smithi160.stdout:1730: writing smithi16030472-60 from 793528 to 1348278 tid 1 2024-09-21T18:08:38.479 INFO:tasks.rados.rados.0.smithi160.stdout:1730: writing smithi16030472-60 from 2083369 to 2520343 tid 2 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1730: writing smithi16030472-60 from 2958856 to 3193528 tid 3 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1731: cache_try_flush oid 249 snap -1 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1732: copy_from oid 483 from oid 395 current snap is 0 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1733: copy_from oid 91 from oid 244 current snap is 0 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1734: copy_from oid 64 from oid 409 current snap is 0 2024-09-21T18:08:38.484 INFO:tasks.rados.rados.0.smithi160.stdout:1735: cache_evict oid 470 snap -1 2024-09-21T18:08:38.485 INFO:tasks.rados.rados.0.smithi160.stdout:1736: read oid 40 snap -1 2024-09-21T18:08:38.485 INFO:tasks.rados.rados.0.smithi160.stdout:1736: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:08:38.485 INFO:tasks.rados.rados.0.smithi160.stdout:1737: delete oid 434 current snap is 0 2024-09-21T18:08:38.485 INFO:tasks.rados.rados.0.smithi160.stdout:1735: got (16) Device or resource busy 2024-09-21T18:08:38.491 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:38.486+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:38.502 INFO:tasks.rados.rados.0.smithi160.stdout:1733: finishing copy_from racing read to smithi16030472-91 2024-09-21T18:08:38.546 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.546 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:38.587 INFO:tasks.rados.rados.0.smithi160.stdout:1736: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:08:38.630 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:38.625+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:38.648 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:38.693 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:38.688+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:38.712 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.712 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.730 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1733: finishing copy_from to smithi16030472-91 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 212 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: finishing write tid 2 to smithi16030472-415 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: finishing write tid 3 to smithi16030472-415 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: finishing write tid 4 to smithi16030472-415 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: finishing write tid 5 to smithi16030472-415 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: oid 415 updating version 0 to 614 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: oid 415 updating version 614 to 671 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: oid 415 version 671 is already newer than 670 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: oid 415 updating version 671 to 672 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 415 v 672 (ObjNum 902 snap 0 seq_num 902) dirty exists 2024-09-21T18:08:38.732 INFO:tasks.rados.rados.0.smithi160.stdout:1725: left oid 415 (ObjNum 902 snap 0 seq_num 902) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1725: done (8 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1733: done (7 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1735: done (6 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1736: done (5 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1737: done (4 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1738: copy_from oid 364 from oid 362 current snap is 0 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1730: finishing write tid 1 to smithi16030472-60 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1730: finishing write tid 2 to smithi16030472-60 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1730: finishing write tid 3 to smithi16030472-60 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1734: finishing copy_from racing read to smithi16030472-64 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1734: finishing copy_from to smithi16030472-64 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 64 v 40 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1732: finishing copy_from racing read to smithi16030472-483 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1734: done (4 left) 2024-09-21T18:08:38.733 INFO:tasks.rados.rados.0.smithi160.stdout:1739: delete oid 203 current snap is 0 2024-09-21T18:08:38.734 INFO:tasks.rados.rados.0.smithi160.stdout:1731: got (0) Success 2024-09-21T18:08:38.734 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 249 v 0 (ObjNum 590 snap 0 seq_num 590) dirty exists 2024-09-21T18:08:38.794 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:38.790+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:38.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:38.849 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.849 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.863 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:38.895 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:38.892+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: finishing write tid 4 to smithi16030472-60 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: finishing write tid 5 to smithi16030472-60 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: oid 60 updating version 0 to 674 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: oid 60 version 674 is already newer than 673 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: oid 60 updating version 674 to 676 2024-09-21T18:08:38.924 INFO:tasks.rados.rados.0.smithi160.stdout:1730: oid 60 version 676 is already newer than 675 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 676 (ObjNum 904 snap 0 seq_num 904) dirty exists 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1730: left oid 60 (ObjNum 904 snap 0 seq_num 904) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1732: finishing copy_from to smithi16030472-483 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 483 v 325 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1738: finishing copy_from to smithi16030472-364 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1738: got expected ENOENT (src dne) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1730: done (4 left) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1731: done (3 left) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1732: done (2 left) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1739: done (1 left) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1740: read oid 202 snap -1 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1740: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1741: write (excl) oid 263 current snap is 0 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1738: finishing copy_from racing read to smithi16030472-364 2024-09-21T18:08:38.925 INFO:tasks.rados.rados.0.smithi160.stdout:1741: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-21T18:08:38.939 INFO:tasks.rados.rados.0.smithi160.stdout:1741: writing smithi16030472-263 from 443949 to 940799 tid 1 2024-09-21T18:08:38.957 INFO:tasks.rados.rados.0.smithi160.stdout:1741: writing smithi16030472-263 from 1458815 to 2043116 tid 2 2024-09-21T18:08:38.976 INFO:tasks.rados.rados.0.smithi160.stdout:1741: writing smithi16030472-263 from 2641125 to 3243949 tid 3 2024-09-21T18:08:38.976 INFO:tasks.rados.rados.0.smithi160.stdout:1738: done (2 left) 2024-09-21T18:08:38.976 INFO:tasks.rados.rados.0.smithi160.stdout:1742: cache_try_flush oid 367 snap -1 2024-09-21T18:08:38.976 INFO:tasks.rados.rados.0.smithi160.stdout:1743: write oid 27 current snap is 0 2024-09-21T18:08:38.976 INFO:tasks.rados.rados.0.smithi160.stdout:1743: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-21T18:08:38.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:08:38.988 INFO:tasks.rados.rados.0.smithi160.stdout:1743: writing smithi16030472-27 from 460756 to 1032855 tid 1 2024-09-21T18:08:38.988 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:38.988 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:38.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:38.996 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:38.992+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:38.997 INFO:tasks.rados.rados.0.smithi160.stdout:1743: writing smithi16030472-27 from 1517297 to 1961609 tid 2 2024-09-21T18:08:39.004 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:39.008 INFO:tasks.rados.rados.0.smithi160.stdout:1743: writing smithi16030472-27 from 2668548 to 3141418 tid 3 2024-09-21T18:08:39.008 INFO:tasks.rados.rados.0.smithi160.stdout:1744: cache_flush oid 385 snap -1 2024-09-21T18:08:39.008 INFO:tasks.rados.rados.0.smithi160.stdout:1740: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-21T18:08:39.011 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:39.011 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:39.031 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:39.032 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:39.050 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:39.050 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:39.097 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:39.092+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: finishing write tid 1 to smithi16030472-263 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1740: done (4 left) 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1745: cache_flush oid 124 snap -1 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1742: got (2) No such file or directory 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: finishing write tid 2 to smithi16030472-263 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: finishing write tid 3 to smithi16030472-263 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: finishing write tid 4 to smithi16030472-263 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: finishing write tid 5 to smithi16030472-263 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: oid 263 updating version 0 to 677 2024-09-21T18:08:39.108 INFO:tasks.rados.rados.0.smithi160.stdout:1741: oid 263 updating version 677 to 678 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1741: oid 263 updating version 678 to 679 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1741: oid 263 updating version 679 to 680 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 263 v 680 (ObjNum 907 snap 0 seq_num 907) dirty exists 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1741: left oid 263 (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1741: done (4 left) 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1742: done (3 left) 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1746: read oid 292 snap -1 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1746: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1747: write (excl) oid 173 current snap is 0 2024-09-21T18:08:39.109 INFO:tasks.rados.rados.0.smithi160.stdout:1747: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-21T18:08:39.115 INFO:tasks.rados.rados.0.smithi160.stdout:1747: writing smithi16030472-173 from 477563 to 1124911 tid 1 2024-09-21T18:08:39.124 INFO:tasks.rados.rados.0.smithi160.stdout:1747: writing smithi16030472-173 from 1859426 to 2447396 tid 2 2024-09-21T18:08:39.130 INFO:tasks.rados.rados.0.smithi160.stdout:1747: writing smithi16030472-173 from 2979618 to 3277563 tid 3 2024-09-21T18:08:39.130 INFO:tasks.rados.rados.0.smithi160.stdout:1748: read oid 100 snap -1 2024-09-21T18:08:39.130 INFO:tasks.rados.rados.0.smithi160.stdout:1748: expect (ObjNum 99 snap 0 seq_num 99) 2024-09-21T18:08:39.130 INFO:tasks.rados.rados.0.smithi160.stdout:1749: write (excl) oid 338 current snap is 0 2024-09-21T18:08:39.130 INFO:tasks.rados.rados.0.smithi160.stdout:1749: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-21T18:08:39.130 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:39.138 INFO:tasks.rados.rados.0.smithi160.stdout:1749: writing smithi16030472-338 from 494370 to 1216967 tid 1 2024-09-21T18:08:39.139 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:39.139 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:39.146 INFO:tasks.rados.rados.0.smithi160.stdout:1749: writing smithi16030472-338 from 1917908 to 2365889 tid 2 2024-09-21T18:08:39.151 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:39.151 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:39.152 INFO:tasks.rados.rados.0.smithi160.stdout:1749: writing smithi16030472-338 from 3007041 to 3294370 tid 3 2024-09-21T18:08:39.153 INFO:tasks.rados.rados.0.smithi160.stdout:1750: cache_evict oid 107 snap -1 2024-09-21T18:08:39.153 INFO:tasks.rados.rados.0.smithi160.stdout:1743: finishing write tid 1 to smithi16030472-27 2024-09-21T18:08:39.153 INFO:tasks.rados.rados.0.smithi160.stdout:1751: write oid 25 current snap is 0 2024-09-21T18:08:39.153 INFO:tasks.rados.rados.0.smithi160.stdout:1743: finishing write tid 2 to smithi16030472-27 2024-09-21T18:08:39.153 INFO:tasks.rados.rados.0.smithi160.stdout:1751: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-21T18:08:39.154 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:39.160 INFO:tasks.rados.rados.0.smithi160.stdout:1751: writing smithi16030472-25 from 511177 to 1309023 tid 1 2024-09-21T18:08:39.168 INFO:tasks.rados.rados.0.smithi160.stdout:1751: writing smithi16030472-25 from 1976390 to 2568029 tid 2 2024-09-21T18:08:39.171 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:39.172 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:39.172 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:39.172 INFO:tasks.rados.rados.0.smithi160.stdout:1751: writing smithi16030472-25 from 3034464 to 3311177 tid 3 2024-09-21T18:08:39.172 INFO:tasks.rados.rados.0.smithi160.stdout:1752: delete oid 259 current snap is 0 2024-09-21T18:08:39.172 INFO:tasks.rados.rados.0.smithi160.stdout:1743: finishing write tid 3 to smithi16030472-27 2024-09-21T18:08:39.172 INFO:tasks.rados.rados.0.smithi160.stdout:1747: finishing write tid 1 to smithi16030472-173 2024-09-21T18:08:39.172 INFO:tasks.rados.rados.0.smithi160.stdout:1750: got (0) Success 2024-09-21T18:08:39.188 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:39.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:39.198 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:39.193+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:39.209 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:39.209 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:39.216 INFO:tasks.rados.rados.0.smithi160.stdout:1746: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:39.216 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:39.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:39.232 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:39.252 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:39.252 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:39.269 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:39.269 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:39.285 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:39.285 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:39.290 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:39.291 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:39.299 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:39.300 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:39.296+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:39.310 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:39.310 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:39.331 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:39.331 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:39.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:39.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:39.338 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2033482 2024-09-21T18:08:39.355 INFO:tasks.rados.rados.0.smithi160.stdout:1743: finishing write tid 4 to smithi16030472-27 2024-09-21T18:08:39.355 INFO:tasks.rados.rados.0.smithi160.stdout:1743: finishing write tid 5 to smithi16030472-27 2024-09-21T18:08:39.355 INFO:tasks.rados.rados.0.smithi160.stdout:1743: oid 27 updating version 0 to 665 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1743: oid 27 version 665 is already newer than 663 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1743: oid 27 version 665 is already newer than 664 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1743: oid 27 version 665 is already newer than 662 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 665 (ObjNum 908 snap 0 seq_num 908) dirty exists 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1743: left oid 27 (ObjNum 908 snap 0 seq_num 908) 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1744: got (0) Success 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 0 (ObjNum 129 snap 0 seq_num 3136606112) clean dne 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1747: finishing write tid 2 to smithi16030472-173 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1747: finishing write tid 3 to smithi16030472-173 2024-09-21T18:08:39.356 INFO:tasks.rados.rados.0.smithi160.stdout:1748: expect (ObjNum 99 snap 0 seq_num 99) 2024-09-21T18:08:39.358 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1264994 2024-09-21T18:08:39.400 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:39.396+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:39.434 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3486202 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1749: finishing write tid 1 to smithi16030472-338 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1751: finishing write tid 1 to smithi16030472-25 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: finishing write tid 4 to smithi16030472-173 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: finishing write tid 5 to smithi16030472-173 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: oid 173 updating version 0 to 590 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: oid 173 version 590 is already newer than 588 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: oid 173 updating version 590 to 591 2024-09-21T18:08:39.441 INFO:tasks.rados.rados.0.smithi160.stdout:1747: oid 173 version 591 is already newer than 589 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 591 (ObjNum 909 snap 0 seq_num 909) dirty exists 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:1747: left oid 173 (ObjNum 909 snap 0 seq_num 909) 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:1745: got (0) Success 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 124 v 0 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:1749: finishing write tid 2 to smithi16030472-338 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:1749: finishing write tid 3 to smithi16030472-338 2024-09-21T18:08:39.442 INFO:tasks.rados.rados.0.smithi160.stdout:1749: finishing write tid 4 to smithi16030472-338 2024-09-21T18:08:39.456 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:39.456 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:39.456 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.779208335999996, 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:08:39.457 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:08:39.458 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.459 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.460 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.779137973999994, 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.461 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:08:39.462 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.463 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.464 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.767336538999999, 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.465 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:08:39.466 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.467 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.468 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.759470156000006, 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.469 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:08:39.470 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.471 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.472 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:39.473 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.759084512000001, 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.474 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:08:39.475 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.476 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 80.759039063000003, 2024-09-21T18:08:39.477 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.478 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.479 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.480 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:08:39.481 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.482 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 79.711141146000003, 2024-09-21T18:08:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:08:39.483 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.484 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:08:39.485 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.486 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.487 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 79.711033504, 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.488 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.489 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.490 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.491 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 74.135250889000005, 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.492 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:08:39.493 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.494 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:08:39.495 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:08:39.496 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.497 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 74.135198954000003, 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.498 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:08:39.499 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.500 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:08:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:08:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.501 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.502 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:08:39.503 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.504 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.368494565999995, 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.505 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.510 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.139391308, 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.511 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.513 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:08:39.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.138963317999995, 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.517 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:08:39.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:08:39.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.138916749000003, 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.523 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:08:39.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 68.638751639000006, 2024-09-21T18:08:39.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:08:39.532 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.533 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.534 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 68.638247688999996, 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:39.535 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.536 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.537 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.538 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:08:39.539 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 61.630715893000001, 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.540 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.541 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.542 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:08:39.543 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:08:39.544 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:08:39.545 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 61.630641572999998, 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.546 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:08:39.547 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:08:39.548 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.549 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.550 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.551 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 61.228838328000002, 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.552 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:08:39.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.555 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 61.126488440000003, 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:39.556 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:39.557 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:08:39.558 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:08:39.559 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:08:39.560 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:39.561 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:39.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: finishing write tid 5 to smithi16030472-338 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: oid 338 updating version 0 to 564 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: oid 338 updating version 564 to 566 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: oid 338 version 566 is already newer than 563 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: oid 338 version 566 is already newer than 565 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 338 v 566 (ObjNum 910 snap 0 seq_num 910) dirty exists 2024-09-21T18:08:39.564 INFO:tasks.rados.rados.0.smithi160.stdout:1749: left oid 338 (ObjNum 910 snap 0 seq_num 910) 2024-09-21T18:08:39.565 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:39.560+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:39.647 INFO:tasks.rados.rados.0.smithi160.stdout:1751: finishing write tid 2 to smithi16030472-25 2024-09-21T18:08:39.647 INFO:tasks.rados.rados.0.smithi160.stdout:1751: finishing write tid 3 to smithi16030472-25 2024-09-21T18:08:39.664 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:39.658+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:39.698 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:39.699 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:39.699 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:39.699 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:39.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:39.764 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:39.760+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1743: done (9 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1744: done (8 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1745: done (7 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1746: done (6 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1747: done (5 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1748: done (4 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1749: done (3 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1750: done (2 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1752: done (1 left) 2024-09-21T18:08:39.768 INFO:tasks.rados.rados.0.smithi160.stdout:1753: write oid 225 current snap is 0 2024-09-21T18:08:39.769 INFO:tasks.rados.rados.0.smithi160.stdout:1753: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-21T18:08:39.775 INFO:tasks.rados.rados.0.smithi160.stdout:1753: writing smithi16030472-225 from 544791 to 1209488 tid 1 2024-09-21T18:08:39.781 INFO:tasks.rados.rados.0.smithi160.stdout:1753: writing smithi16030472-225 from 1693354 to 2288662 tid 2 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1753: writing smithi16030472-225 from 2689310 to 3134893 tid 3 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1754: read oid 234 snap -1 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1754: expect (ObjNum 781 snap 0 seq_num 781) 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1755: read oid 188 snap -1 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1755: expect (ObjNum 187 snap 0 seq_num 187) 2024-09-21T18:08:39.787 INFO:tasks.rados.rados.0.smithi160.stdout:1756: cache_try_flush oid 290 snap -1 2024-09-21T18:08:39.788 INFO:tasks.rados.rados.0.smithi160.stdout:1757: read oid 215 snap -1 2024-09-21T18:08:39.788 INFO:tasks.rados.rados.0.smithi160.stdout:1757: expect (ObjNum 607 snap 0 seq_num 607) 2024-09-21T18:08:39.788 INFO:tasks.rados.rados.0.smithi160.stdout:1758: read oid 161 snap -1 2024-09-21T18:08:39.788 INFO:tasks.rados.rados.0.smithi160.stdout:1758: expect (ObjNum 160 snap 0 seq_num 160) 2024-09-21T18:08:39.788 INFO:tasks.rados.rados.0.smithi160.stdout:1759: delete oid 52 current snap is 0 2024-09-21T18:08:39.865 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:39.860+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:39.880 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:39.880 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:39.880 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:39.880 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:39.880 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:39.887 INFO:tasks.rados.rados.0.smithi160.stdout:1751: finishing write tid 4 to smithi16030472-25 2024-09-21T18:08:39.887 INFO:tasks.rados.rados.0.smithi160.stdout:1751: finishing write tid 5 to smithi16030472-25 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1751: oid 25 updating version 0 to 654 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1751: oid 25 version 654 is already newer than 651 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1751: oid 25 version 654 is already newer than 652 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1751: oid 25 version 654 is already newer than 653 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 25 v 654 (ObjNum 911 snap 0 seq_num 911) dirty exists 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1751: left oid 25 (ObjNum 911 snap 0 seq_num 911) 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1753: finishing write tid 1 to smithi16030472-225 2024-09-21T18:08:39.888 INFO:tasks.rados.rados.0.smithi160.stdout:1756: got (2) No such file or directory 2024-09-21T18:08:39.889 INFO:tasks.rados.rados.0.smithi160.stdout:1754: expect (ObjNum 781 snap 0 seq_num 781) 2024-09-21T18:08:39.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:39.966 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:39.964+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:08:39.977 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:39.969+0000 7fb260c0e640 -1 osd.11 81 handle_osd_map injecting map crc failure 2024-09-21T18:08:40.019 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.019 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:40.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:40.019 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:40.020 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.020 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2139891 2024-09-21T18:08:40.030 INFO:tasks.rados.rados.0.smithi160.stdout:1755: expect (ObjNum 187 snap 0 seq_num 187) 2024-09-21T18:08:40.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:40.075 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:40.070+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:40.106 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 944283 2024-09-21T18:08:40.129 INFO:tasks.rados.rados.0.smithi160.stdout:1753: finishing write tid 2 to smithi16030472-225 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: finishing write tid 3 to smithi16030472-225 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: finishing write tid 4 to smithi16030472-225 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: finishing write tid 5 to smithi16030472-225 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: oid 225 updating version 0 to 657 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: oid 225 version 657 is already newer than 656 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: oid 225 updating version 657 to 658 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: oid 225 version 658 is already newer than 655 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 658 (ObjNum 913 snap 0 seq_num 913) dirty exists 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1753: left oid 225 (ObjNum 913 snap 0 seq_num 913) 2024-09-21T18:08:40.130 INFO:tasks.rados.rados.0.smithi160.stdout:1758: expect (ObjNum 160 snap 0 seq_num 160) 2024-09-21T18:08:40.168 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:40.164+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:40.188 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:40.201 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.202 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:40.202 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:40.202 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:40.217 INFO:tasks.rados.rados.0.smithi160.stdout:1757: expect (ObjNum 607 snap 0 seq_num 607) 2024-09-21T18:08:40.268 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:40.268 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:40.262+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1751: done (7 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1753: done (6 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1754: done (5 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1755: done (4 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1756: done (3 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1757: done (2 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1758: done (1 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1759: done (0 left) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1760: read oid 31 snap -1 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1760: expect (ObjNum 571 snap 0 seq_num 571) 2024-09-21T18:08:40.299 INFO:tasks.rados.rados.0.smithi160.stdout:1761: write oid 360 current snap is 0 2024-09-21T18:08:40.300 INFO:tasks.rados.rados.0.smithi160.stdout:1761: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-21T18:08:40.300 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:40.294+0000 7ff40b999640 -1 osd.4 81 handle_osd_map injecting map crc failure 2024-09-21T18:08:40.314 INFO:tasks.rados.rados.0.smithi160.stdout:1761: writing smithi16030472-360 from 578405 to 993600 tid 1 2024-09-21T18:08:40.330 INFO:tasks.rados.rados.0.smithi160.stdout:1761: writing smithi16030472-360 from 1410318 to 2009295 tid 2 2024-09-21T18:08:40.347 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.347 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:40.347 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:40.347 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:40.347 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.350 INFO:tasks.rados.rados.0.smithi160.stdout:1761: writing smithi16030472-360 from 2744156 to 3328636 tid 3 2024-09-21T18:08:40.350 INFO:tasks.rados.rados.0.smithi160.stdout:1762: delete oid 108 current snap is 0 2024-09-21T18:08:40.350 INFO:tasks.rados.rados.0.smithi160.stdout:1761: finishing write tid 1 to smithi16030472-360 2024-09-21T18:08:40.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:40.403 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:40.398+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:40.407 INFO:tasks.rados.rados.0.smithi160.stdout:1760: expect (ObjNum 571 snap 0 seq_num 571) 2024-09-21T18:08:40.407 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 120043 2024-09-21T18:08:40.470 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:40.465+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:40.470 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1429575 2024-09-21T18:08:40.535 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 87.32081882, 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.536 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:08:40.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:08:40.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:40.542 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.776881270999993, 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:08:40.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:08:40.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:08:40.547 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.615957807000001, 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:08:40.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.615283211999994, 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.552 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:08:40.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:08:40.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.556 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.615256673999994, 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:40.560 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.605422931999996, 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:08:40.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:08:40.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.564 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.593011144000002, 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:08:40.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:08:40.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:08:40.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.568 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.584121073999995, 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.571 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:08:40.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:08:40.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:08:40.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.583309056999994, 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.583 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.583200844999993, 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:08:40.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:08:40.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.482929698999996, 2024-09-21T18:08:40.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:08:40.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:08:40.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.482033439999995, 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.481551827000004, 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.598 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.601 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.481099401999998, 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.602 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.603 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:08:40.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:08:40.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.481049076000005, 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.610 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.355316295999998, 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.611 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:08:40.612 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.613 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:08:40.614 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.615 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.616 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.351038239000005, 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.617 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:08:40.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.619 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.620 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:08:40.621 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.622 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:08:40.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.624 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.625 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.350667998999995, 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:40.626 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.627 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:40.628 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.629 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.630 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:08:40.631 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.632 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.633 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:08:40.634 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:08:40.634 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.634 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.634 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.634 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:08:40.635 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:08:40.636 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.637 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.638 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.639 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.350598269000002, 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.640 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.641 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.642 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.643 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:08:40.644 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.645 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.646 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.647 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:08:40.648 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.649 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.650 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 68.759021278999995, 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:40.651 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.652 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.653 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.655 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:40.656 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1760: done (2 left) 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1762: done (1 left) 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1763: cache_evict oid 290 snap -1 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1764: read oid 294 snap -1 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1764: expect (ObjNum 795 snap 0 seq_num 795) 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1765: copy_from oid 332 from oid 412 current snap is 0 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1766: read oid 153 snap -1 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1766: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1767: write oid 73 current snap is 0 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1767: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-21T18:08:40.660 INFO:tasks.rados.rados.0.smithi160.stdout:1767: writing smithi16030472-73 from 612019 to 1177712 tid 1 2024-09-21T18:08:40.661 INFO:tasks.rados.rados.0.smithi160.stdout:1767: writing smithi16030472-73 from 1810929 to 2413575 tid 2 2024-09-21T18:08:40.661 INFO:tasks.rados.rados.0.smithi160.stdout:1767: writing smithi16030472-73 from 3082649 to 3412019 tid 3 2024-09-21T18:08:40.661 INFO:tasks.rados.rados.0.smithi160.stdout:1768: cache_try_flush oid 396 snap -1 2024-09-21T18:08:40.661 INFO:tasks.rados.rados.0.smithi160.stdout:1766: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:08:40.661 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:40.661 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:40.656+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:40.726 INFO:tasks.rados.rados.0.smithi160.stdout:1763: got (2) No such file or directory 2024-09-21T18:08:40.726 INFO:tasks.rados.rados.0.smithi160.stdout:1768: got (2) No such file or directory 2024-09-21T18:08:40.726 INFO:tasks.rados.rados.0.smithi160.stdout:1763: done (6 left) 2024-09-21T18:08:40.726 INFO:tasks.rados.rados.0.smithi160.stdout:1766: done (5 left) 2024-09-21T18:08:40.726 INFO:tasks.rados.rados.0.smithi160.stdout:1768: done (4 left) 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1769: read oid 470 snap -1 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1769: expect (ObjNum 643 snap 0 seq_num 643) 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1770: read oid 138 snap -1 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1770: expect (ObjNum 137 snap 0 seq_num 137) 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1771: cache_flush oid 78 snap -1 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1772: write (excl) oid 465 current snap is 0 2024-09-21T18:08:40.727 INFO:tasks.rados.rados.0.smithi160.stdout:1772: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-21T18:08:40.731 INFO:tasks.rados.rados.0.smithi160.stdout:1772: writing smithi16030472-465 from 628826 to 1269768 tid 1 2024-09-21T18:08:40.737 INFO:tasks.rados.rados.0.smithi160.stdout:1772: writing smithi16030472-465 from 1869411 to 2332068 tid 2 2024-09-21T18:08:40.743 INFO:tasks.rados.rados.0.smithi160.stdout:1772: writing smithi16030472-465 from 2826425 to 3361074 tid 3 2024-09-21T18:08:40.743 INFO:tasks.rados.rados.0.smithi160.stdout:1773: cache_flush oid 231 snap -1 2024-09-21T18:08:40.743 INFO:tasks.rados.rados.0.smithi160.stdout:1761: finishing write tid 2 to smithi16030472-360 2024-09-21T18:08:40.743 INFO:tasks.rados.rados.0.smithi160.stdout:1774: delete oid 56 current snap is 0 2024-09-21T18:08:40.743 INFO:tasks.rados.rados.0.smithi160.stdout:1767: finishing write tid 1 to smithi16030472-73 2024-09-21T18:08:40.759 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:40.756+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:40.796 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.796 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:40.796 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:40.796 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: finishing write tid 3 to smithi16030472-360 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: finishing write tid 4 to smithi16030472-360 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: finishing write tid 5 to smithi16030472-360 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: oid 360 updating version 0 to 657 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: oid 360 version 657 is already newer than 655 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: oid 360 updating version 657 to 658 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: oid 360 version 658 is already newer than 656 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 658 (ObjNum 915 snap 0 seq_num 915) dirty exists 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: left oid 360 (ObjNum 915 snap 0 seq_num 915) 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1761: done (9 left) 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1774: done (8 left) 2024-09-21T18:08:40.841 INFO:tasks.rados.rados.0.smithi160.stdout:1775: cache_flush oid 91 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1776: cache_evict oid 398 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1777: cache_try_flush oid 231 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1778: read oid 117 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1778: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1779: read oid 488 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1779: expect (ObjNum 487 snap 0 seq_num 487) 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1780: cache_flush oid 342 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1781: cache_flush oid 236 snap -1 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1782: write (excl) oid 314 current snap is 0 2024-09-21T18:08:40.842 INFO:tasks.rados.rados.0.smithi160.stdout:1782: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-21T18:08:40.855 INFO:tasks.rados.rados.0.smithi160.stdout:1782: writing smithi16030472-314 from 662440 to 1170233 tid 1 2024-09-21T18:08:40.860 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:40.853+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:40.870 INFO:tasks.rados.rados.0.smithi160.stdout:1782: writing smithi16030472-314 from 1702728 to 2452701 tid 2 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1782: writing smithi16030472-314 from 2881271 to 3438464 tid 3 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1776: got (16) Device or resource busy 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1776: done (15 left) 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1783: cache_evict oid 268 snap -1 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1765: finishing copy_from racing read to smithi16030472-332 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1767: finishing write tid 2 to smithi16030472-73 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1767: finishing write tid 3 to smithi16030472-73 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1767: finishing write tid 4 to smithi16030472-73 2024-09-21T18:08:40.884 INFO:tasks.rados.rados.0.smithi160.stdout:1767: finishing write tid 5 to smithi16030472-73 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1767: oid 73 updating version 0 to 666 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1767: oid 73 updating version 666 to 669 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1767: oid 73 version 669 is already newer than 668 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1767: oid 73 version 669 is already newer than 667 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 73 v 669 (ObjNum 917 snap 0 seq_num 917) dirty exists 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1767: left oid 73 (ObjNum 917 snap 0 seq_num 917) 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1765: finishing copy_from to smithi16030472-332 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 622 (ObjNum 837 snap 0 seq_num 837) dirty exists 2024-09-21T18:08:40.885 INFO:tasks.rados.rados.0.smithi160.stdout:1764: expect (ObjNum 795 snap 0 seq_num 795) 2024-09-21T18:08:40.932 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2259086 2024-09-21T18:08:40.938 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:40.938 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:40.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:40.961 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:40.956+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:40.985 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1978029 2024-09-21T18:08:41.051 INFO:tasks.rados.rados.0.smithi160.stdout:1779: expect (ObjNum 487 snap 0 seq_num 487) 2024-09-21T18:08:41.062 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:41.056+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:41.086 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:41.086 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:41.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:41.086 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:41.086 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:41.093 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1641667 2024-09-21T18:08:41.098 INFO:tasks.rados.rados.0.smithi160.stdout:1778: expect (ObjNum 268 snap 0 seq_num 268) 2024-09-21T18:08:41.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:41.124 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:41.150 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:41.163 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:41.160+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:41.177 INFO:tasks.rados.rados.0.smithi160.stdout:1780: got (0) Success 2024-09-21T18:08:41.177 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 0 (ObjNum 465 snap 0 seq_num 465) dirty exists 2024-09-21T18:08:41.177 INFO:tasks.rados.rados.0.smithi160.stdout:1769: expect (ObjNum 643 snap 0 seq_num 643) 2024-09-21T18:08:41.223 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:41.223 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:41.223 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:41.224 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:41.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:41.263 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:41.257+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:41.299 INFO:tasks.rados.rados.0.smithi160.stdout:1773: got (0) Success 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 231 v 0 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1777: got (0) Success 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 231 v 0 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1781: got (2) No such file or directory 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1783: got (2) No such file or directory 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1772: finishing write tid 1 to smithi16030472-465 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1772: finishing write tid 2 to smithi16030472-465 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1772: finishing write tid 3 to smithi16030472-465 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1782: finishing write tid 1 to smithi16030472-314 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1764: done (15 left) 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1765: done (14 left) 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1767: done (13 left) 2024-09-21T18:08:41.300 INFO:tasks.rados.rados.0.smithi160.stdout:1769: done (12 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1773: done (11 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1777: done (10 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1778: done (9 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1779: done (8 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1780: done (7 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1781: done (6 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1783: done (5 left) 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1784: write oid 412 current snap is 0 2024-09-21T18:08:41.301 INFO:tasks.rados.rados.0.smithi160.stdout:1784: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-21T18:08:41.306 INFO:tasks.rados.rados.0.smithi160.stdout:1784: writing smithi16030472-412 from 679247 to 1262289 tid 1 2024-09-21T18:08:41.314 INFO:tasks.rados.rados.0.smithi160.stdout:1784: writing smithi16030472-412 from 2044857 to 2654841 tid 2 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1784: writing smithi16030472-412 from 3192341 to 3479247 tid 3 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1785: delete oid 342 current snap is 0 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1782: finishing write tid 2 to smithi16030472-314 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1782: finishing write tid 3 to smithi16030472-314 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1772: finishing write tid 4 to smithi16030472-465 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1772: finishing write tid 5 to smithi16030472-465 2024-09-21T18:08:41.319 INFO:tasks.rados.rados.0.smithi160.stdout:1772: oid 465 updating version 0 to 670 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:1772: oid 465 updating version 670 to 671 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:1772: oid 465 updating version 671 to 672 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:1772: oid 465 updating version 672 to 673 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 465 v 673 (ObjNum 918 snap 0 seq_num 918) dirty exists 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:1772: left oid 465 (ObjNum 918 snap 0 seq_num 918) 2024-09-21T18:08:41.320 INFO:tasks.rados.rados.0.smithi160.stdout:1770: expect (ObjNum 137 snap 0 seq_num 137) 2024-09-21T18:08:41.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:41.364 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1517632 2024-09-21T18:08:41.364 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:41.360+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:41.367 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:41.368 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:41.368 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:41.368 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:41.368 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:41.376 INFO:tasks.rados.rados.0.smithi160.stdout:1775: got (0) Success 2024-09-21T18:08:41.376 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 0 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-09-21T18:08:41.376 INFO:tasks.rados.rados.0.smithi160.stdout:1771: got (0) Success 2024-09-21T18:08:41.376 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 0 (ObjNum 853 snap 0 seq_num 853) dirty exists 2024-09-21T18:08:41.376 INFO:tasks.rados.rados.0.smithi160.stdout:1782: finishing write tid 4 to smithi16030472-314 2024-09-21T18:08:41.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: finishing write tid 5 to smithi16030472-314 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: oid 314 updating version 0 to 659 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: oid 314 updating version 659 to 661 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: oid 314 updating version 661 to 662 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: oid 314 version 662 is already newer than 660 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 314 v 662 (ObjNum 920 snap 0 seq_num 920) dirty exists 2024-09-21T18:08:41.386 INFO:tasks.rados.rados.0.smithi160.stdout:1782: left oid 314 (ObjNum 920 snap 0 seq_num 920) 2024-09-21T18:08:41.471 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:41.466+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:41.486 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:41.507 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:41.507 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:41.527 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:41.527 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:41.547 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:41.554 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 115.657500782, 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:41.555 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:08:41.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:41.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 114.57210839299999, 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:41.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:08:41.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:41.563 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:41.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:41.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:41.564 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:41.564 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:41.566 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:41.558+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:41.566 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:41.570 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:41.587 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:41.608 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:41.608 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:41.620 INFO:tasks.rados.rados.0.smithi160.stdout:1784: finishing write tid 1 to smithi16030472-412 2024-09-21T18:08:41.620 INFO:tasks.rados.rados.0.smithi160.stdout:1784: finishing write tid 2 to smithi16030472-412 2024-09-21T18:08:41.620 INFO:tasks.rados.rados.0.smithi160.stdout:1784: finishing write tid 3 to smithi16030472-412 2024-09-21T18:08:41.628 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:41.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:41.667 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:41.662+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:41.668 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:41.668 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:41.689 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:41.689 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:41.701 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:41.701 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:41.710 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:41.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:41.716 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:41.722 INFO:tasks.rados.rados.0.smithi160.stdout:1784: finishing write tid 4 to smithi16030472-412 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: finishing write tid 5 to smithi16030472-412 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: oid 412 updating version 0 to 686 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: oid 412 version 686 is already newer than 685 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: oid 412 version 686 is already newer than 684 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: oid 412 version 686 is already newer than 683 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 686 (ObjNum 921 snap 0 seq_num 921) dirty exists 2024-09-21T18:08:41.723 INFO:tasks.rados.rados.0.smithi160.stdout:1784: left oid 412 (ObjNum 921 snap 0 seq_num 921) 2024-09-21T18:08:41.732 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:41.732 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:41.755 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:41.768 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:41.762+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:41.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:41.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1770: done (6 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1771: done (5 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1772: done (4 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1775: done (3 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1782: done (2 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1784: done (1 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1785: done (0 left) 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1786: write oid 262 current snap is 0 2024-09-21T18:08:41.823 INFO:tasks.rados.rados.0.smithi160.stdout:1786: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-21T18:08:41.835 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:41.835 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:41.841 INFO:tasks.rados.rados.0.smithi160.stdout:1786: writing smithi16030472-262 from 712861 to 1446401 tid 1 2024-09-21T18:08:41.849 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:41.857 INFO:tasks.rados.rados.0.smithi160.stdout:1786: writing smithi16030472-262 from 2161821 to 2775474 tid 2 2024-09-21T18:08:41.866 INFO:tasks.rados.rados.0.smithi160.stdout:1786: writing smithi16030472-262 from 3247187 to 3512861 tid 3 2024-09-21T18:08:41.866 INFO:tasks.rados.rados.0.smithi160.stdout:1787: copy_from oid 44 from oid 460 current snap is 0 2024-09-21T18:08:41.866 INFO:tasks.rados.rados.0.smithi160.stdout:1788: write (excl) oid 140 current snap is 0 2024-09-21T18:08:41.866 INFO:tasks.rados.rados.0.smithi160.stdout:1788: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-21T18:08:41.868 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:41.862+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:41.879 INFO:tasks.rados.rados.0.smithi160.stdout:1788: writing smithi16030472-140 from 729668 to 1138457 tid 1 2024-09-21T18:08:41.892 INFO:tasks.rados.rados.0.smithi160.stdout:1788: writing smithi16030472-140 from 1820303 to 2577614 tid 2 2024-09-21T18:08:41.900 INFO:tasks.rados.rados.0.smithi160.stdout:1788: writing smithi16030472-140 from 3274610 to 3529668 tid 3 2024-09-21T18:08:41.900 INFO:tasks.rados.rados.0.smithi160.stdout:1789: delete oid 91 current snap is 0 2024-09-21T18:08:41.970 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:41.964+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:41.978 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:41.978 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:41.982 INFO:tasks.rados.rados.0.smithi160.stdout:1787: finishing copy_from racing read to smithi16030472-44 2024-09-21T18:08:41.982 INFO:tasks.rados.rados.0.smithi160.stdout:1787: finishing copy_from to smithi16030472-44 2024-09-21T18:08:41.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 44 v 312 (ObjNum 459 snap 0 seq_num 459) dirty exists 2024-09-21T18:08:41.982 INFO:tasks.rados.rados.0.smithi160.stdout:1786: finishing write tid 1 to smithi16030472-262 2024-09-21T18:08:41.993 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:42.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:08:42.005 INFO:tasks.rados.rados.0.smithi160.stdout:1787: done (3 left) 2024-09-21T18:08:42.005 INFO:tasks.rados.rados.0.smithi160.stdout:1789: done (2 left) 2024-09-21T18:08:42.005 INFO:tasks.rados.rados.0.smithi160.stdout:1790: read oid 451 snap -1 2024-09-21T18:08:42.006 INFO:tasks.rados.rados.0.smithi160.stdout:1790: expect (ObjNum 603 snap 0 seq_num 603) 2024-09-21T18:08:42.006 INFO:tasks.rados.rados.0.smithi160.stdout:1791: cache_flush oid 58 snap -1 2024-09-21T18:08:42.006 INFO:tasks.rados.rados.0.smithi160.stdout:1792: cache_try_flush oid 202 snap -1 2024-09-21T18:08:42.006 INFO:tasks.rados.rados.0.smithi160.stdout:1793: cache_evict oid 496 snap -1 2024-09-21T18:08:42.006 INFO:tasks.rados.rados.0.smithi160.stdout:1794: delete oid 338 current snap is 0 2024-09-21T18:08:42.071 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:42.068+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:42.106 INFO:tasks.rados.rados.0.smithi160.stdout:1791: got (2) No such file or directory 2024-09-21T18:08:42.106 INFO:tasks.rados.rados.0.smithi160.stdout:1792: got (2) No such file or directory 2024-09-21T18:08:42.108 INFO:tasks.rados.rados.0.smithi160.stdout:1793: got (0) Success 2024-09-21T18:08:42.110 INFO:tasks.rados.rados.0.smithi160.stdout:1791: done (6 left) 2024-09-21T18:08:42.110 INFO:tasks.rados.rados.0.smithi160.stdout:1792: done (5 left) 2024-09-21T18:08:42.110 INFO:tasks.rados.rados.0.smithi160.stdout:1793: done (4 left) 2024-09-21T18:08:42.110 INFO:tasks.rados.rados.0.smithi160.stdout:1794: done (3 left) 2024-09-21T18:08:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:1795: cache_evict oid 232 snap -1 2024-09-21T18:08:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:1796: cache_flush oid 20 snap -1 2024-09-21T18:08:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:1797: cache_evict oid 432 snap -1 2024-09-21T18:08:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:1798: write oid 220 current snap is 0 2024-09-21T18:08:42.111 INFO:tasks.rados.rados.0.smithi160.stdout:1798: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-09-21T18:08:42.124 INFO:tasks.rados.rados.0.smithi160.stdout:1798: writing smithi16030472-220 from 780089 to 1414625 tid 1 2024-09-21T18:08:42.135 INFO:tasks.rados.rados.0.smithi160.stdout:1798: writing smithi16030472-220 from 1879396 to 2500387 tid 2 2024-09-21T18:08:42.145 INFO:tasks.rados.rados.0.smithi160.stdout:1798: writing smithi16030472-220 from 3240526 to 3580089 tid 3 2024-09-21T18:08:42.146 INFO:tasks.rados.rados.0.smithi160.stdout:1799: cache_try_flush oid 67 snap -1 2024-09-21T18:08:42.146 INFO:tasks.rados.rados.0.smithi160.stdout:1800: copy_from oid 80 from oid 199 current snap is 0 2024-09-21T18:08:42.146 INFO:tasks.rados.rados.0.smithi160.stdout:1801: write (excl) oid 441 current snap is 0 2024-09-21T18:08:42.146 INFO:tasks.rados.rados.0.smithi160.stdout:1801: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2024-09-21T18:08:42.157 INFO:tasks.rados.rados.0.smithi160.stdout:1801: writing smithi16030472-441 from 796896 to 1223034 tid 1 2024-09-21T18:08:42.159 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.160 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:1801: writing smithi16030472-441 from 1654231 to 2418880 tid 2 2024-09-21T18:08:42.172 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:42.168+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:42.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:42.176 INFO:tasks.rados.rados.0.smithi160.stdout:1801: writing smithi16030472-441 from 2984302 to 3464377 tid 3 2024-09-21T18:08:42.177 INFO:tasks.rados.rados.0.smithi160.stdout:1802: delete oid 227 current snap is 0 2024-09-21T18:08:42.183 INFO:tasks.rados.rados.0.smithi160.stdout:1786: finishing write tid 2 to smithi16030472-262 2024-09-21T18:08:42.183 INFO:tasks.rados.rados.0.smithi160.stdout:1786: finishing write tid 3 to smithi16030472-262 2024-09-21T18:08:42.183 INFO:tasks.rados.rados.0.smithi160.stdout:1786: finishing write tid 4 to smithi16030472-262 2024-09-21T18:08:42.183 INFO:tasks.rados.rados.0.smithi160.stdout:1786: finishing write tid 5 to smithi16030472-262 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:1786: oid 262 updating version 0 to 675 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:1786: oid 262 version 675 is already newer than 639 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:1786: oid 262 updating version 675 to 676 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:1786: oid 262 version 676 is already newer than 674 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 262 v 676 (ObjNum 923 snap 0 seq_num 923) dirty exists 2024-09-21T18:08:42.184 INFO:tasks.rados.rados.0.smithi160.stdout:1786: left oid 262 (ObjNum 923 snap 0 seq_num 923) 2024-09-21T18:08:42.189 INFO:tasks.rados.rados.0.smithi160.stdout:1788: finishing write tid 1 to smithi16030472-140 2024-09-21T18:08:42.216 INFO:tasks.rados.rados.0.smithi160.stdout:1795: got (0) Success 2024-09-21T18:08:42.273 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:42.266+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:42.317 INFO:tasks.rados.rados.0.smithi160.stdout:1797: got (0) Success 2024-09-21T18:08:42.345 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.345 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.360 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:42.379 INFO:tasks.rados.rados.0.smithi160.stdout:1799: got (0) Success 2024-09-21T18:08:42.379 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 67 v 0 (ObjNum 66 snap 0 seq_num 66) dirty exists 2024-09-21T18:08:42.379 INFO:tasks.rados.rados.0.smithi160.stdout:1801: finishing write tid 1 to smithi16030472-441 2024-09-21T18:08:42.379 INFO:tasks.rados.rados.0.smithi160.stdout:1801: finishing write tid 2 to smithi16030472-441 2024-09-21T18:08:42.379 INFO:tasks.rados.rados.0.smithi160.stdout:1800: finishing copy_from racing read to smithi16030472-80 2024-09-21T18:08:42.381 INFO:tasks.rados.rados.0.smithi160.stdout:1800: finishing copy_from to smithi16030472-80 2024-09-21T18:08:42.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 107 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:08:42.407 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:42.404+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:42.437 INFO:tasks.rados.rados.0.smithi160.stdout:1790: expect (ObjNum 603 snap 0 seq_num 603) 2024-09-21T18:08:42.437 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:42.475 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:42.470+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:42.490 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.490 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.504 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:42.549 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:42.578 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:42.572+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:42.579 INFO:tasks.rados.rados.0.smithi160.stdout:1798: finishing write tid 1 to smithi16030472-220 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: finishing write tid 2 to smithi16030472-140 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: finishing write tid 3 to smithi16030472-140 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1798: finishing write tid 2 to smithi16030472-220 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1796: got (0) Success 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 20 v 0 (ObjNum 92 snap 0 seq_num 92) dirty exists 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: finishing write tid 4 to smithi16030472-140 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: finishing write tid 5 to smithi16030472-140 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: oid 140 updating version 0 to 679 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: oid 140 version 679 is already newer than 677 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: oid 140 version 679 is already newer than 640 2024-09-21T18:08:42.580 INFO:tasks.rados.rados.0.smithi160.stdout:1788: oid 140 version 679 is already newer than 678 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 140 v 679 (ObjNum 924 snap 0 seq_num 924) dirty exists 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1788: left oid 140 (ObjNum 924 snap 0 seq_num 924) 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: finishing write tid 3 to smithi16030472-441 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: finishing write tid 4 to smithi16030472-441 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: finishing write tid 5 to smithi16030472-441 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: oid 441 updating version 0 to 691 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: oid 441 version 691 is already newer than 689 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: oid 441 version 691 is already newer than 688 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: oid 441 version 691 is already newer than 690 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 691 (ObjNum 928 snap 0 seq_num 928) dirty exists 2024-09-21T18:08:42.581 INFO:tasks.rados.rados.0.smithi160.stdout:1801: left oid 441 (ObjNum 928 snap 0 seq_num 928) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1786: done (10 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1788: done (9 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1790: done (8 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1795: done (7 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1796: done (6 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1797: done (5 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1799: done (4 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1800: done (3 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1801: done (2 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1802: done (1 left) 2024-09-21T18:08:42.664 INFO:tasks.rados.rados.0.smithi160.stdout:1803: write (excl) oid 388 current snap is 0 2024-09-21T18:08:42.665 INFO:tasks.rados.rados.0.smithi160.stdout:1803: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-21T18:08:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:1803: writing smithi16030472-388 from 430510 to 1007146 tid 1 2024-09-21T18:08:42.673 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.673 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.676 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:42.672+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:42.681 INFO:tasks.rados.rados.0.smithi160.stdout:1803: writing smithi16030472-388 from 1654842 to 2423160 tid 2 2024-09-21T18:08:42.687 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:42.689 INFO:tasks.rados.rados.0.smithi160.stdout:1803: writing smithi16030472-388 from 2922795 to 3425414 tid 3 2024-09-21T18:08:42.689 INFO:tasks.rados.rados.0.smithi160.stdout:1804: read oid 188 snap -1 2024-09-21T18:08:42.689 INFO:tasks.rados.rados.0.smithi160.stdout:1804: expect (ObjNum 187 snap 0 seq_num 187) 2024-09-21T18:08:42.689 INFO:tasks.rados.rados.0.smithi160.stdout:1805: delete oid 209 current snap is 0 2024-09-21T18:08:42.689 INFO:tasks.rados.rados.0.smithi160.stdout:1803: finishing write tid 1 to smithi16030472-388 2024-09-21T18:08:42.697 INFO:tasks.rados.rados.0.smithi160.stdout:1804: expect (ObjNum 187 snap 0 seq_num 187) 2024-09-21T18:08:42.777 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1316514 2024-09-21T18:08:42.777 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:42.774+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:42.799 INFO:tasks.rados.rados.0.smithi160.stdout:1803: finishing write tid 2 to smithi16030472-388 2024-09-21T18:08:42.799 INFO:tasks.rados.rados.0.smithi160.stdout:1803: finishing write tid 3 to smithi16030472-388 2024-09-21T18:08:42.799 INFO:tasks.rados.rados.0.smithi160.stdout:1803: finishing write tid 4 to smithi16030472-388 2024-09-21T18:08:42.799 INFO:tasks.rados.rados.0.smithi160.stdout:1803: finishing write tid 5 to smithi16030472-388 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:1803: oid 388 updating version 0 to 680 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:1803: oid 388 updating version 680 to 682 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:1803: oid 388 updating version 682 to 683 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:1803: oid 388 version 683 is already newer than 681 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 388 v 683 (ObjNum 930 snap 0 seq_num 930) dirty exists 2024-09-21T18:08:42.800 INFO:tasks.rados.rados.0.smithi160.stdout:1803: left oid 388 (ObjNum 930 snap 0 seq_num 930) 2024-09-21T18:08:42.844 INFO:tasks.rados.rados.0.smithi160.stdout:1798: finishing write tid 3 to smithi16030472-220 2024-09-21T18:08:42.844 INFO:tasks.rados.rados.0.smithi160.stdout:1798: finishing write tid 4 to smithi16030472-220 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: finishing write tid 5 to smithi16030472-220 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: oid 220 updating version 0 to 665 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: oid 220 version 665 is already newer than 664 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: oid 220 version 665 is already newer than 663 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: oid 220 updating version 665 to 666 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 666 (ObjNum 927 snap 0 seq_num 927) dirty exists 2024-09-21T18:08:42.845 INFO:tasks.rados.rados.0.smithi160.stdout:1798: left oid 220 (ObjNum 927 snap 0 seq_num 927) 2024-09-21T18:08:42.852 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.853 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.867 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:42.878 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:42.870+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:42.898 INFO:tasks.rados.rados.0.smithi160.stdout:1798: done (3 left) 2024-09-21T18:08:42.898 INFO:tasks.rados.rados.0.smithi160.stdout:1803: done (2 left) 2024-09-21T18:08:42.898 INFO:tasks.rados.rados.0.smithi160.stdout:1804: done (1 left) 2024-09-21T18:08:42.898 INFO:tasks.rados.rados.0.smithi160.stdout:1805: done (0 left) 2024-09-21T18:08:42.898 INFO:tasks.rados.rados.0.smithi160.stdout:1806: delete oid 234 current snap is 0 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1806: done (0 left) 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1807: cache_try_flush oid 188 snap -1 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1808: cache_flush oid 242 snap -1 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1809: copy_from oid 193 from oid 473 current snap is 0 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1810: copy_from oid 363 from oid 495 current snap is 0 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1811: cache_try_flush oid 9 snap -1 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1812: read oid 66 snap -1 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1812: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-21T18:08:42.903 INFO:tasks.rados.rados.0.smithi160.stdout:1813: delete oid 201 current snap is 0 2024-09-21T18:08:42.904 INFO:tasks.rados.rados.0.smithi160.stdout:1808: got (0) Success 2024-09-21T18:08:42.904 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 242 v 0 (ObjNum 129 snap 0 seq_num 3126304760) clean dne 2024-09-21T18:08:42.978 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:42.972+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:42.987 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:42.988 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:08:42.992 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:42.988+0000 7f0b6c216640 -1 osd.9 82 handle_osd_map injecting map crc failure 2024-09-21T18:08:43.002 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:43.018 INFO:tasks.rados.rados.0.smithi160.stdout:1810: finishing copy_from racing read to smithi16030472-363 2024-09-21T18:08:43.079 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:43.074+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:43.095 INFO:tasks.rados.rados.0.smithi160.stdout:1807: got (0) Success 2024-09-21T18:08:43.095 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 188 v 0 (ObjNum 187 snap 0 seq_num 187) dirty exists 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1807: done (6 left) 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1808: done (5 left) 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1813: done (4 left) 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1814: read oid 229 snap -1 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1814: expect (ObjNum 228 snap 0 seq_num 228) 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1815: read oid 403 snap -1 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1815: expect (ObjNum 865 snap 0 seq_num 865) 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1816: read oid 187 snap -1 2024-09-21T18:08:43.097 INFO:tasks.rados.rados.0.smithi160.stdout:1816: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-21T18:08:43.098 INFO:tasks.rados.rados.0.smithi160.stdout:1817: write oid 368 current snap is 0 2024-09-21T18:08:43.098 INFO:tasks.rados.rados.0.smithi160.stdout:1817: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-21T18:08:43.111 INFO:tasks.rados.rados.0.smithi160.stdout:1817: writing smithi16030472-368 from 497738 to 975370 tid 1 2024-09-21T18:08:43.122 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:43.120+0000 7fca4e93e640 -1 osd.2 82 handle_osd_map injecting map crc failure 2024-09-21T18:08:43.133 INFO:tasks.rados.rados.0.smithi160.stdout:1817: writing smithi16030472-368 from 1772417 to 2548073 tid 2 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1817: writing smithi16030472-368 from 3316134 to 3697738 tid 3 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1818: read oid 219 snap -1 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1818: expect (ObjNum 581 snap 0 seq_num 581) 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1819: cache_flush oid 346 snap -1 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1820: write oid 491 current snap is 0 2024-09-21T18:08:43.149 INFO:tasks.rados.rados.0.smithi160.stdout:1820: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-21T18:08:43.160 INFO:tasks.rados.rados.0.smithi160.stdout:1820: writing smithi16030472-491 from 514545 to 1067426 tid 1 2024-09-21T18:08:43.172 INFO:tasks.rados.rados.0.smithi160.stdout:1820: writing smithi16030472-491 from 1830899 to 2350213 tid 2 2024-09-21T18:08:43.175 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.176 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.180 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:43.174+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:43.181 INFO:tasks.rados.rados.0.smithi160.stdout:1820: writing smithi16030472-491 from 2943557 to 3418889 tid 3 2024-09-21T18:08:43.181 INFO:tasks.rados.rados.0.smithi160.stdout:1821: cache_evict oid 463 snap -1 2024-09-21T18:08:43.181 INFO:tasks.rados.rados.0.smithi160.stdout:1822: write oid 258 current snap is 0 2024-09-21T18:08:43.181 INFO:tasks.rados.rados.0.smithi160.stdout:1822: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-21T18:08:43.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:43.191 INFO:tasks.rados.rados.0.smithi160.stdout:1822: writing smithi16030472-258 from 531352 to 1275835 tid 1 2024-09-21T18:08:43.199 INFO:tasks.rados.rados.0.smithi160.stdout:1822: writing smithi16030472-258 from 2005734 to 2785059 tid 2 2024-09-21T18:08:43.205 INFO:tasks.rados.rados.0.smithi160.stdout:1822: writing smithi16030472-258 from 3487333 to 3731352 tid 3 2024-09-21T18:08:43.205 INFO:tasks.rados.rados.0.smithi160.stdout:1823: delete oid 478 current snap is 0 2024-09-21T18:08:43.281 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:43.276+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:43.300 INFO:tasks.rados.rados.0.smithi160.stdout:1814: expect (ObjNum 228 snap 0 seq_num 228) 2024-09-21T18:08:43.300 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:43.318 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.318 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.333 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:43.382 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:43.380+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:43.439 INFO:tasks.rados.rados.0.smithi160.stdout:1810: finishing copy_from to smithi16030472-363 2024-09-21T18:08:43.439 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 530 (ObjNum 838 snap 0 seq_num 838) dirty exists 2024-09-21T18:08:43.439 INFO:tasks.rados.rados.0.smithi160.stdout:1816: expect (ObjNum 186 snap 0 seq_num 186) 2024-09-21T18:08:43.449 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3032706 2024-09-21T18:08:43.460 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.460 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:43.482 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:43.480+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:43.494 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1584138 2024-09-21T18:08:43.516 INFO:tasks.rados.rados.0.smithi160.stdout:1811: got (0) Success 2024-09-21T18:08:43.516 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 9 v 0 (ObjNum 874 snap 0 seq_num 874) dirty exists 2024-09-21T18:08:43.516 INFO:tasks.rados.rados.0.smithi160.stdout:1815: expect (ObjNum 865 snap 0 seq_num 865) 2024-09-21T18:08:43.526 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1535597 2024-09-21T18:08:43.573 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:43.583 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:43.580+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:43.606 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.607 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.608 INFO:tasks.rados.rados.0.smithi160.stdout:1812: expect (ObjNum 65 snap 0 seq_num 65) 2024-09-21T18:08:43.608 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:43.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:43.640 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:43.684 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:43.680+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1819: got (0) Success 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 346 v 0 (ObjNum 748 snap 0 seq_num 748) dirty exists 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1821: got (2) No such file or directory 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1809: finishing copy_from to smithi16030472-193 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1809: got expected ENOENT (src dne) 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1809: finishing copy_from racing read to smithi16030472-193 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1820: finishing write tid 1 to smithi16030472-491 2024-09-21T18:08:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:1822: finishing write tid 1 to smithi16030472-258 2024-09-21T18:08:43.696 INFO:tasks.rados.rados.0.smithi160.stdout:1822: finishing write tid 2 to smithi16030472-258 2024-09-21T18:08:43.718 INFO:tasks.rados.rados.0.smithi160.stdout:1822: finishing write tid 3 to smithi16030472-258 2024-09-21T18:08:43.718 INFO:tasks.rados.rados.0.smithi160.stdout:1822: finishing write tid 4 to smithi16030472-258 2024-09-21T18:08:43.729 INFO:tasks.rados.rados.0.smithi160.stdout:1817: finishing write tid 1 to smithi16030472-368 2024-09-21T18:08:43.729 INFO:tasks.rados.rados.0.smithi160.stdout:1820: finishing write tid 2 to smithi16030472-491 2024-09-21T18:08:43.729 INFO:tasks.rados.rados.0.smithi160.stdout:1817: finishing write tid 2 to smithi16030472-368 2024-09-21T18:08:43.730 INFO:tasks.rados.rados.0.smithi160.stdout:1817: finishing write tid 3 to smithi16030472-368 2024-09-21T18:08:43.730 INFO:tasks.rados.rados.0.smithi160.stdout:1817: finishing write tid 4 to smithi16030472-368 2024-09-21T18:08:43.730 INFO:tasks.rados.rados.0.smithi160.stdout:1817: finishing write tid 5 to smithi16030472-368 2024-09-21T18:08:43.730 INFO:tasks.rados.rados.0.smithi160.stdout:1817: oid 368 updating version 0 to 620 2024-09-21T18:08:43.730 INFO:tasks.rados.rados.0.smithi160.stdout:1817: oid 368 version 620 is already newer than 617 2024-09-21T18:08:43.731 INFO:tasks.rados.rados.0.smithi160.stdout:1817: oid 368 version 620 is already newer than 618 2024-09-21T18:08:43.731 INFO:tasks.rados.rados.0.smithi160.stdout:1817: oid 368 version 620 is already newer than 619 2024-09-21T18:08:43.731 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 368 v 620 (ObjNum 934 snap 0 seq_num 934) dirty exists 2024-09-21T18:08:43.731 INFO:tasks.rados.rados.0.smithi160.stdout:1817: left oid 368 (ObjNum 934 snap 0 seq_num 934) 2024-09-21T18:08:43.785 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:43.777+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:43.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:43.793 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.793 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:43.819 INFO:tasks.rados.rados.0.smithi160.stdout:1822: finishing write tid 5 to smithi16030472-258 2024-09-21T18:08:43.819 INFO:tasks.rados.rados.0.smithi160.stdout:1822: oid 258 updating version 0 to 667 2024-09-21T18:08:43.819 INFO:tasks.rados.rados.0.smithi160.stdout:1822: oid 258 updating version 667 to 669 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1822: oid 258 updating version 669 to 670 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1822: oid 258 version 670 is already newer than 668 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 670 (ObjNum 936 snap 0 seq_num 936) dirty exists 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1822: left oid 258 (ObjNum 936 snap 0 seq_num 936) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1809: done (13 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1810: done (12 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1811: done (11 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1812: done (10 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1814: done (9 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1815: done (8 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1816: done (7 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1817: done (6 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1819: done (5 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1821: done (4 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1822: done (3 left) 2024-09-21T18:08:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:1823: done (2 left) 2024-09-21T18:08:43.821 INFO:tasks.rados.rados.0.smithi160.stdout:1824: delete oid 114 current snap is 0 2024-09-21T18:08:43.821 INFO:tasks.rados.rados.0.smithi160.stdout:1820: finishing write tid 3 to smithi16030472-491 2024-09-21T18:08:43.821 INFO:tasks.rados.rados.0.smithi160.stdout:1820: finishing write tid 4 to smithi16030472-491 2024-09-21T18:08:43.886 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:43.881+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:43.926 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:43.926 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:43.932 INFO:tasks.rados.rados.0.smithi160.stdout:1818: expect (ObjNum 581 snap 0 seq_num 581) 2024-09-21T18:08:43.940 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:43.946 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:43.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:43.966 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2249555 2024-09-21T18:08:43.967 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:43.987 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:43.984+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:43.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:43.989 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:44.011 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:44.031 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:44.031 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:44.053 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:44.064 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:44.064 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:44.076 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:44.077 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:44.077 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:44.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:44.088 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:44.082+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:44.099 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:44.100 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:1820: finishing write tid 5 to smithi16030472-491 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:1820: oid 491 updating version 0 to 593 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:1820: oid 491 updating version 593 to 595 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:1820: oid 491 updating version 595 to 596 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:1820: oid 491 version 596 is already newer than 594 2024-09-21T18:08:44.123 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 596 (ObjNum 935 snap 0 seq_num 935) dirty exists 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1820: left oid 491 (ObjNum 935 snap 0 seq_num 935) 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1818: done (2 left) 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1820: done (1 left) 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1824: done (0 left) 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1825: cache_try_flush oid 307 snap -1 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1826: read oid 136 snap -1 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1826: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1827: copy_from oid 49 from oid 177 current snap is 0 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1828: write (excl) oid 427 current snap is 0 2024-09-21T18:08:44.124 INFO:tasks.rados.rados.0.smithi160.stdout:1828: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-21T18:08:44.125 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:44.135 INFO:tasks.rados.rados.0.smithi160.stdout:1828: writing smithi16030472-427 from 581773 to 1152003 tid 1 2024-09-21T18:08:44.144 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:44.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:44.147 INFO:tasks.rados.rados.0.smithi160.stdout:1828: writing smithi16030472-427 from 1664827 to 2191479 tid 2 2024-09-21T18:08:44.159 INFO:tasks.rados.rados.0.smithi160.stdout:1828: writing smithi16030472-427 from 2653249 to 3290022 tid 3 2024-09-21T18:08:44.159 INFO:tasks.rados.rados.0.smithi160.stdout:1829: delete oid 316 current snap is 0 2024-09-21T18:08:44.159 INFO:tasks.rados.rados.0.smithi160.stdout:1827: finishing copy_from racing read to smithi16030472-49 2024-09-21T18:08:44.159 INFO:tasks.rados.rados.0.smithi160.stdout:1827: finishing copy_from to smithi16030472-49 2024-09-21T18:08:44.159 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 417 (ObjNum 678 snap 0 seq_num 678) dirty exists 2024-09-21T18:08:44.168 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:44.168 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:44.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:44.188 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:44.184+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:44.202 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:44.202 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:44.202 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:44.203 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:44.203 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:44.208 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:44.208 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:44.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:44.231 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:44.231 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:44.254 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:44.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:44.258 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:44.289 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:44.284+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:44.335 INFO:tasks.rados.rados.0.smithi160.stdout:1826: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:08:44.335 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:44.336 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:44.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:44.336 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:08:44.336 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:44.350 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:44.390 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:44.385+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:44.421 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1825: got (0) Success 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 0 (ObjNum 129 snap 0 seq_num 3136084568) clean dne 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1825: done (4 left) 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1826: done (3 left) 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1827: done (2 left) 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1829: done (1 left) 2024-09-21T18:08:44.465 INFO:tasks.rados.rados.0.smithi160.stdout:1830: delete oid 347 current snap is 0 2024-09-21T18:08:44.472 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:44.473 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:44.473 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:44.473 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:44.487 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:44.533 INFO:tasks.rados.rados.0.smithi160.stdout:1828: finishing write tid 1 to smithi16030472-427 2024-09-21T18:08:44.534 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:44.532+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:44.570 INFO:tasks.rados.rados.0.smithi160.stdout:1830: done (1 left) 2024-09-21T18:08:44.570 INFO:tasks.rados.rados.0.smithi160.stdout:1831: cache_try_flush oid 26 snap -1 2024-09-21T18:08:44.570 INFO:tasks.rados.rados.0.smithi160.stdout:1832: write oid 78 current snap is 0 2024-09-21T18:08:44.570 INFO:tasks.rados.rados.0.smithi160.stdout:1832: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-21T18:08:44.591 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:44.586+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:44.593 INFO:tasks.rados.rados.0.smithi160.stdout:1832: writing smithi16030472-78 from 632194 to 1428171 tid 1 2024-09-21T18:08:44.609 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:44.609 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:44.610 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:44.610 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:44.617 INFO:tasks.rados.rados.0.smithi160.stdout:1832: writing smithi16030472-78 from 2123920 to 2914252 tid 2 2024-09-21T18:08:44.624 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:44.629 INFO:tasks.rados.rados.0.smithi160.stdout:1832: writing smithi16030472-78 from 3419165 to 3832194 tid 3 2024-09-21T18:08:44.629 INFO:tasks.rados.rados.0.smithi160.stdout:1833: read oid 263 snap -1 2024-09-21T18:08:44.629 INFO:tasks.rados.rados.0.smithi160.stdout:1833: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:08:44.629 INFO:tasks.rados.rados.0.smithi160.stdout:1834: write (excl) oid 241 current snap is 0 2024-09-21T18:08:44.629 INFO:tasks.rados.rados.0.smithi160.stdout:1834: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2024-09-21T18:08:44.642 INFO:tasks.rados.rados.0.smithi160.stdout:1834: writing smithi16030472-241 from 649001 to 1236580 tid 1 2024-09-21T18:08:44.653 INFO:tasks.rados.rados.0.smithi160.stdout:1834: writing smithi16030472-241 from 1898755 to 2432745 tid 2 2024-09-21T18:08:44.664 INFO:tasks.rados.rados.0.smithi160.stdout:1834: writing smithi16030472-241 from 3162941 to 3849001 tid 3 2024-09-21T18:08:44.664 INFO:tasks.rados.rados.0.smithi160.stdout:1835: delete oid 421 current snap is 0 2024-09-21T18:08:44.664 INFO:tasks.rados.rados.0.smithi160.stdout:1828: finishing write tid 2 to smithi16030472-427 2024-09-21T18:08:44.665 INFO:tasks.rados.rados.0.smithi160.stdout:1828: finishing write tid 3 to smithi16030472-427 2024-09-21T18:08:44.665 INFO:tasks.rados.rados.0.smithi160.stdout:1828: finishing write tid 4 to smithi16030472-427 2024-09-21T18:08:44.665 INFO:tasks.rados.rados.0.smithi160.stdout:1833: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:08:44.665 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:44.692 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:44.688+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:44.712 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:44.788 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2921447 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.791 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.192552750999994, 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:08:44.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.762582803999997, 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:08:44.796 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:08:44.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.665062429000002, 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:08:44.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.664378179000003, 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.805 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:08:44.809 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.663544823999999, 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.660826194000002, 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:08:44.816 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:08:44.817 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.565017787999999, 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.562713246999998, 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.822 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2900 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'245, mlcod=59'234)", 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:44.826 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 50.462432960000001, 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22363195199999999, 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325923+0000", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2929999999999999e-06 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325935+0000", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1763000000000001e-05 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325952+0000", 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6722999999999999e-05 2024-09-21T18:08:44.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325965+0000", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3743e-05 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545223+0000", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21925772700000001 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545279+0000", 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6085999999999998e-05 2024-09-21T18:08:44.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549487+0000", 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004207868 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549553+0000", 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5748999999999997e-05 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.830 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.585084627000001, 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:08:44.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:08:44.836 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.483928944999999, 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:08:44.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:08:44.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:08:44.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.483133283000001, 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.844 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.481997462999999, 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:44.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:08:44.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:08:44.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:08:44.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.461279869999998, 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:44.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:08:44.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:08:44.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.451927185999999, 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:08:44.853 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.446552537999999, 2024-09-21T18:08:44.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:08:44.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:08:44.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.861 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.444004263, 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:08:44.865 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 25.216303726, 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:08:44.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:08:44.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 25.215637433000001, 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:08:44.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:08:44.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 25.215545321, 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.878 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.879 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.879 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:44.879 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:44.879 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1834: finishing write tid 1 to smithi16030472-241 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1834: finishing write tid 2 to smithi16030472-241 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: finishing write tid 5 to smithi16030472-427 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: oid 427 updating version 0 to 667 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: oid 427 updating version 667 to 668 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: oid 427 updating version 668 to 669 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: oid 427 version 669 is already newer than 666 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 669 (ObjNum 939 snap 0 seq_num 939) dirty exists 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1828: left oid 427 (ObjNum 939 snap 0 seq_num 939) 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1831: got (0) Success 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 0 (ObjNum 881 snap 0 seq_num 881) dirty exists 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1832: finishing write tid 1 to smithi16030472-78 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1834: finishing write tid 3 to smithi16030472-241 2024-09-21T18:08:44.881 INFO:tasks.rados.rados.0.smithi160.stdout:1834: finishing write tid 4 to smithi16030472-241 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: finishing write tid 5 to smithi16030472-241 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: oid 241 updating version 0 to 626 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: oid 241 version 626 is already newer than 624 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: oid 241 version 626 is already newer than 623 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: oid 241 version 626 is already newer than 625 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 626 (ObjNum 943 snap 0 seq_num 943) dirty exists 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: left oid 241 (ObjNum 943 snap 0 seq_num 943) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1828: done (5 left) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1831: done (4 left) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1833: done (3 left) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1834: done (2 left) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1835: done (1 left) 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1836: write oid 69 current snap is 0 2024-09-21T18:08:44.882 INFO:tasks.rados.rados.0.smithi160.stdout:1836: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-21T18:08:44.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:44.883 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:44.874+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:44.886 INFO:tasks.rados.rados.0.smithi160.stdout:1836: writing smithi16030472-69 from 682615 to 1420692 tid 1 2024-09-21T18:08:44.894 INFO:tasks.rados.rados.0.smithi160.stdout:1836: writing smithi16030472-69 from 2015719 to 2553378 tid 2 2024-09-21T18:08:44.901 INFO:tasks.rados.rados.0.smithi160.stdout:1836: writing smithi16030472-69 from 3217787 to 3638545 tid 3 2024-09-21T18:08:44.901 INFO:tasks.rados.rados.0.smithi160.stdout:1837: write (excl) oid 162 current snap is 0 2024-09-21T18:08:44.902 INFO:tasks.rados.rados.0.smithi160.stdout:1837: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-21T18:08:44.909 INFO:tasks.rados.rados.0.smithi160.stdout:1837: writing smithi16030472-162 from 699422 to 1112748 tid 1 2024-09-21T18:08:44.917 INFO:tasks.rados.rados.0.smithi160.stdout:1837: writing smithi16030472-162 from 1557848 to 2239165 tid 2 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1837: writing smithi16030472-162 from 3012504 to 3760887 tid 3 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1838: read oid 113 snap -1 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1838: expect (ObjNum 707 snap 0 seq_num 707) 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1839: cache_flush oid 175 snap -1 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1840: cache_flush oid 10 snap -1 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1841: write (excl) oid 220 current snap is 0 2024-09-21T18:08:44.928 INFO:tasks.rados.rados.0.smithi160.stdout:1841: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-21T18:08:44.937 INFO:tasks.rados.rados.0.smithi160.stdout:1841: writing smithi16030472-220 from 716229 to 1204804 tid 1 2024-09-21T18:08:44.943 INFO:tasks.rados.rados.0.smithi160.stdout:1841: writing smithi16030472-220 from 1616330 to 2157658 tid 2 2024-09-21T18:08:44.950 INFO:tasks.rados.rados.0.smithi160.stdout:1841: writing smithi16030472-220 from 2756280 to 3315935 tid 3 2024-09-21T18:08:44.950 INFO:tasks.rados.rados.0.smithi160.stdout:1842: copy_from oid 276 from oid 438 current snap is 0 2024-09-21T18:08:44.950 INFO:tasks.rados.rados.0.smithi160.stdout:1843: read oid 392 snap -1 2024-09-21T18:08:44.950 INFO:tasks.rados.rados.0.smithi160.stdout:1843: expect deleted 2024-09-21T18:08:44.951 INFO:tasks.rados.rados.0.smithi160.stdout:1844: copy_from oid 8 from oid 338 current snap is 0 2024-09-21T18:08:44.951 INFO:tasks.rados.rados.0.smithi160.stdout:1832: finishing write tid 2 to smithi16030472-78 2024-09-21T18:08:44.951 INFO:tasks.rados.rados.0.smithi160.stdout:1838: expect (ObjNum 707 snap 0 seq_num 707) 2024-09-21T18:08:44.951 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:44.981 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:44.976+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:45.009 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:45.009 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:45.009 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:45.009 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:45.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:08:45.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:45.042 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 711199 2024-09-21T18:08:45.082 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:45.074+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:45.137 INFO:tasks.rados.rados.0.smithi160.stdout:1839: got (0) Success 2024-09-21T18:08:45.137 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 0 (ObjNum 719 snap 0 seq_num 719) dirty exists 2024-09-21T18:08:45.137 INFO:tasks.rados.rados.0.smithi160.stdout:1841: finishing write tid 1 to smithi16030472-220 2024-09-21T18:08:45.137 INFO:tasks.rados.rados.0.smithi160.stdout:1838: done (9 left) 2024-09-21T18:08:45.137 INFO:tasks.rados.rados.0.smithi160.stdout:1839: done (8 left) 2024-09-21T18:08:45.138 INFO:tasks.rados.rados.0.smithi160.stdout:1843: done (7 left) 2024-09-21T18:08:45.138 INFO:tasks.rados.rados.0.smithi160.stdout:1845: delete oid 352 current snap is 0 2024-09-21T18:08:45.183 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:45.180+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:45.190 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:45.190 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:45.190 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:45.204 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:45.233 INFO:tasks.rados.rados.0.smithi160.stdout:1832: finishing write tid 3 to smithi16030472-78 2024-09-21T18:08:45.237 INFO:tasks.rados.rados.0.smithi160.stdout:1841: finishing write tid 2 to smithi16030472-220 2024-09-21T18:08:45.238 INFO:tasks.rados.rados.0.smithi160.stdout:1841: finishing write tid 3 to smithi16030472-220 2024-09-21T18:08:45.238 INFO:tasks.rados.rados.0.smithi160.stdout:1841: finishing write tid 4 to smithi16030472-220 2024-09-21T18:08:45.239 INFO:tasks.rados.rados.0.smithi160.stdout:1841: finishing write tid 5 to smithi16030472-220 2024-09-21T18:08:45.239 INFO:tasks.rados.rados.0.smithi160.stdout:1841: oid 220 updating version 0 to 673 2024-09-21T18:08:45.239 INFO:tasks.rados.rados.0.smithi160.stdout:1841: oid 220 updating version 673 to 674 2024-09-21T18:08:45.239 INFO:tasks.rados.rados.0.smithi160.stdout:1841: oid 220 version 674 is already newer than 672 2024-09-21T18:08:45.240 INFO:tasks.rados.rados.0.smithi160.stdout:1841: oid 220 version 674 is already newer than 671 2024-09-21T18:08:45.240 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 674 (ObjNum 947 snap 0 seq_num 947) dirty exists 2024-09-21T18:08:45.240 INFO:tasks.rados.rados.0.smithi160.stdout:1841: left oid 220 (ObjNum 947 snap 0 seq_num 947) 2024-09-21T18:08:45.260 INFO:tasks.rados.rados.0.smithi160.stdout:1844: finishing copy_from to smithi16030472-8 2024-09-21T18:08:45.260 INFO:tasks.rados.rados.0.smithi160.stdout:1844: got expected ENOENT (src dne) 2024-09-21T18:08:45.260 INFO:tasks.rados.rados.0.smithi160.stdout:1840: got (0) Success 2024-09-21T18:08:45.260 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 10 v 0 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-21T18:08:45.260 INFO:tasks.rados.rados.0.smithi160.stdout:1844: finishing copy_from racing read to smithi16030472-8 2024-09-21T18:08:45.284 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:45.280+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:45.329 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:45.329 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 86.488232948000004, 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.076590654, 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:08:45.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:08:45.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.073759781999996, 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:08:45.338 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:08:45.342 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.069786874000002, 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:08:45.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.066669923000006, 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:45.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.064738919000007, 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:08:45.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.061632160000002, 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.359 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.058474476000001, 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:45.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:08:45.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:08:45.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:08:45.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.056241424999996, 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.053989752000007, 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:08:45.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:08:45.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 79.961412163999995, 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:08:45.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:08:45.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:08:45.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 79.184742787999994, 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.175235204000003, 2024-09-21T18:08:45.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:08:45.384 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.068405225000006, 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:08:45.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:08:45.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.388 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 65.735033673000004, 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 60.963042203999997, 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 51.080893748999998, 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.400 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 26.540769301000001, 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:45.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:08:45.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:08:45.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:08:45.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 9.2308108410000003, 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.9864085300000003, 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:45.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:45.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:45.413 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:45.414 INFO:tasks.rados.rados.0.smithi160.stdout:1842: finishing copy_from racing read to smithi16030472-276 2024-09-21T18:08:45.415 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:45.415 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:45.406+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:45.432 INFO:tasks.rados.rados.0.smithi160.stdout:1832: finishing write tid 4 to smithi16030472-78 2024-09-21T18:08:45.432 INFO:tasks.rados.rados.0.smithi160.stdout:1832: finishing write tid 5 to smithi16030472-78 2024-09-21T18:08:45.432 INFO:tasks.rados.rados.0.smithi160.stdout:1832: oid 78 updating version 0 to 684 2024-09-21T18:08:45.432 INFO:tasks.rados.rados.0.smithi160.stdout:1832: oid 78 updating version 684 to 687 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1832: oid 78 version 687 is already newer than 685 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1832: oid 78 version 687 is already newer than 686 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 687 (ObjNum 942 snap 0 seq_num 942) dirty exists 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1832: left oid 78 (ObjNum 942 snap 0 seq_num 942) 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1836: finishing write tid 1 to smithi16030472-69 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1836: finishing write tid 2 to smithi16030472-69 2024-09-21T18:08:45.433 INFO:tasks.rados.rados.0.smithi160.stdout:1836: finishing write tid 3 to smithi16030472-69 2024-09-21T18:08:45.515 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:45.514+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:45.533 INFO:tasks.rados.rados.0.smithi160.stdout:1836: finishing write tid 4 to smithi16030472-69 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: finishing write tid 5 to smithi16030472-69 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: oid 69 updating version 0 to 689 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: oid 69 updating version 689 to 690 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: oid 69 version 690 is already newer than 688 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: oid 69 updating version 690 to 691 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 691 (ObjNum 945 snap 0 seq_num 945) dirty exists 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1836: left oid 69 (ObjNum 945 snap 0 seq_num 945) 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1837: finishing write tid 1 to smithi16030472-162 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1837: finishing write tid 2 to smithi16030472-162 2024-09-21T18:08:45.534 INFO:tasks.rados.rados.0.smithi160.stdout:1837: finishing write tid 3 to smithi16030472-162 2024-09-21T18:08:45.546 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:45.546 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:45.546 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:08:45.546 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:45.560 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:45.616 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:45.614+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:45.634 INFO:tasks.rados.rados.0.smithi160.stdout:1837: finishing write tid 4 to smithi16030472-162 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: finishing write tid 5 to smithi16030472-162 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: oid 162 updating version 0 to 692 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: oid 162 updating version 692 to 693 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: oid 162 updating version 693 to 695 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: oid 162 version 695 is already newer than 694 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 162 v 695 (ObjNum 946 snap 0 seq_num 946) dirty exists 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: left oid 162 (ObjNum 946 snap 0 seq_num 946) 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1832: done (7 left) 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1836: done (6 left) 2024-09-21T18:08:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:1837: done (5 left) 2024-09-21T18:08:45.636 INFO:tasks.rados.rados.0.smithi160.stdout:1840: done (4 left) 2024-09-21T18:08:45.636 INFO:tasks.rados.rados.0.smithi160.stdout:1841: done (3 left) 2024-09-21T18:08:45.636 INFO:tasks.rados.rados.0.smithi160.stdout:1844: done (2 left) 2024-09-21T18:08:45.636 INFO:tasks.rados.rados.0.smithi160.stdout:1845: done (1 left) 2024-09-21T18:08:45.636 INFO:tasks.rados.rados.0.smithi160.stdout:1846: delete oid 338 current snap is 0 2024-09-21T18:08:45.717 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:45.716+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:45.725 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:45.725 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:45.725 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:45.725 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:45.725 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:45.735 INFO:tasks.rados.rados.0.smithi160.stdout:1842: finishing copy_from to smithi16030472-276 2024-09-21T18:08:45.735 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 307 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1842: done (1 left) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1846: done (0 left) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1847: read oid 348 snap -1 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1847: expect (ObjNum 506 snap 0 seq_num 506) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1848: read oid 55 snap -1 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1848: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1849: read oid 436 snap -1 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1849: expect (ObjNum 764 snap 0 seq_num 764) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1850: read oid 267 snap -1 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1850: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1851: write oid 445 current snap is 0 2024-09-21T18:08:45.739 INFO:tasks.rados.rados.0.smithi160.stdout:1851: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-21T18:08:45.740 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:45.754 INFO:tasks.rados.rados.0.smithi160.stdout:1851: writing smithi16030472-445 from 766650 to 1480972 tid 1 2024-09-21T18:08:45.767 INFO:tasks.rados.rados.0.smithi160.stdout:1851: writing smithi16030472-445 from 2075423 to 2764078 tid 2 2024-09-21T18:08:45.780 INFO:tasks.rados.rados.0.smithi160.stdout:1851: writing smithi16030472-445 from 3522196 to 3966650 tid 3 2024-09-21T18:08:45.780 INFO:tasks.rados.rados.0.smithi160.stdout:1852: cache_try_flush oid 108 snap -1 2024-09-21T18:08:45.780 INFO:tasks.rados.rados.0.smithi160.stdout:1853: cache_flush oid 224 snap -1 2024-09-21T18:08:45.780 INFO:tasks.rados.rados.0.smithi160.stdout:1854: delete oid 316 current snap is 0 2024-09-21T18:08:45.819 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:45.816+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:45.879 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:45.880 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:45.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:45.880 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:08:45.880 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:45.880 INFO:tasks.rados.rados.0.smithi160.stdout:1853: got (2) No such file or directory 2024-09-21T18:08:45.883 INFO:tasks.rados.rados.0.smithi160.stdout:1848: expect (ObjNum 54 snap 0 seq_num 54) 2024-09-21T18:08:45.892 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2156832 2024-09-21T18:08:45.900 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:45.907 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:45.919 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:45.916+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:45.971 INFO:tasks.rados.rados.0.smithi160.stdout:1849: expect (ObjNum 764 snap 0 seq_num 764) 2024-09-21T18:08:46.016 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:08:46.020 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:46.014+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:46.027 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:46.027 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:46.027 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:08:46.027 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:46.036 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2558778 2024-09-21T18:08:46.041 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:46.096 INFO:tasks.rados.rados.0.smithi160.stdout:1847: expect (ObjNum 506 snap 0 seq_num 506) 2024-09-21T18:08:46.122 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:46.114+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:46.155 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1897005 2024-09-21T18:08:46.163 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:46.163 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:08:46.164 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:08:46.164 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:08:46.164 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:46.178 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:46.203 INFO:tasks.rados.rados.0.smithi160.stdout:1850: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:08:46.223 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:46.217+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:46.245 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:46.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:46.271 INFO:tasks.rados.rados.0.smithi160.stdout:1851: finishing write tid 1 to smithi16030472-445 2024-09-21T18:08:46.271 INFO:tasks.rados.rados.0.smithi160.stdout:1851: finishing write tid 2 to smithi16030472-445 2024-09-21T18:08:46.271 INFO:tasks.rados.rados.0.smithi160.stdout:1851: finishing write tid 3 to smithi16030472-445 2024-09-21T18:08:46.271 INFO:tasks.rados.rados.0.smithi160.stdout:1847: done (7 left) 2024-09-21T18:08:46.271 INFO:tasks.rados.rados.0.smithi160.stdout:1848: done (6 left) 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1849: done (5 left) 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1850: done (4 left) 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1853: done (3 left) 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1854: done (2 left) 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1855: cache_flush oid 48 snap -1 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1856: write (excl) oid 195 current snap is 0 2024-09-21T18:08:46.272 INFO:tasks.rados.rados.0.smithi160.stdout:1856: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-21T18:08:46.287 INFO:tasks.rados.rados.0.smithi160.stdout:1856: writing smithi16030472-195 from 400264 to 981437 tid 1 2024-09-21T18:08:46.303 INFO:tasks.rados.rados.0.smithi160.stdout:1856: writing smithi16030472-195 from 1508740 to 2000264 tid 2 2024-09-21T18:08:46.303 INFO:tasks.rados.rados.0.smithi160.stdout:1857: copy_from oid 463 from oid 109 current snap is 0 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1855: got (0) Success 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 48 v 0 (ObjNum 80 snap 0 seq_num 80) dirty exists 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1855: done (4 left) 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1858: copy_from oid 276 from oid 142 current snap is 0 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1859: cache_try_flush oid 381 snap -1 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1860: write oid 305 current snap is 0 2024-09-21T18:08:46.304 INFO:tasks.rados.rados.0.smithi160.stdout:1860: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.158123395000004, 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.306 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:08:46.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.312 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.158077086000006, 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.313 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:08:46.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:46.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:08:46.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:46.319 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.155981173000001, 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:08:46.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:08:46.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:46.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:08:46.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:08:46.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:46.328 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.155318058000006, 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:08:46.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:08:46.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:08:46.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:08:46.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.155281559000002, 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:08:46.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:08:46.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:08:46.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:08:46.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.021643386999997, 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.342 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:08:46.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:08:46.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.020931595999997, 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.349 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:08:46.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:08:46.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:08:46.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:08:46.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:08:46.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:08:46.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 93.020894033000005, 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:46.359 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:08:46.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:08:46.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.223508034000005, 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.366 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:08:46.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.223313488000002, 2024-09-21T18:08:46.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.374 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.223027099999996, 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:46.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:08:46.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:08:46.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.223011020000001, 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.194083767999999, 2024-09-21T18:08:46.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.386 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.187864692999995, 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:46.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:08:46.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:08:46.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.182763097999995, 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.182429150000004, 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.398 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.182408671000005, 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.399 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:08:46.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:08:46.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.096080143999998, 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.087619958000005, 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 69.081692253, 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.411 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.412 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.413 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.414 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.415 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.415 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.415 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:46.416 INFO:tasks.rados.rados.0.smithi160.stdout:1860: writing smithi16030472-305 from 417071 to 1073493 tid 1 2024-09-21T18:08:46.416 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1860: writing smithi16030472-305 from 1567222 to 2017071 tid 2 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1861: copy_from oid 403 from oid 446 current snap is 0 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: finishing write tid 4 to smithi16030472-445 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: finishing write tid 5 to smithi16030472-445 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: oid 445 updating version 0 to 697 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: oid 445 version 697 is already newer than 695 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: oid 445 version 697 is already newer than 694 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: oid 445 version 697 is already newer than 696 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 697 (ObjNum 950 snap 0 seq_num 950) dirty exists 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: left oid 445 (ObjNum 950 snap 0 seq_num 950) 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1851: done (7 left) 2024-09-21T18:08:46.419 INFO:tasks.rados.rados.0.smithi160.stdout:1862: read oid 223 snap -1 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1862: expect deleted 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1863: write (excl) oid 45 current snap is 0 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1863: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1863: writing smithi16030472-45 from 433878 to 1165549 tid 1 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1863: writing smithi16030472-45 from 1909351 to 2033878 tid 2 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1864: write (excl) oid 100 current snap is 0 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1864: seq_num 955 ranges {450685=406920,1567833=482852} 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1864: writing smithi16030472-100 from 450685 to 857605 tid 1 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1864: writing smithi16030472-100 from 1567833 to 2050685 tid 2 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1865: cache_try_flush oid 416 snap -1 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1866: read oid 16 snap -1 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1866: expect deleted 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1867: copy_from oid 73 from oid 282 current snap is 0 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1868: write oid 91 current snap is 0 2024-09-21T18:08:46.420 INFO:tasks.rados.rados.0.smithi160.stdout:1868: seq_num 956 ranges {467492=482169,1626315=441177} 2024-09-21T18:08:46.421 INFO:tasks.rados.rados.0.smithi160.stdout:1868: writing smithi16030472-91 from 467492 to 949661 tid 1 2024-09-21T18:08:46.421 INFO:tasks.rados.rados.0.smithi160.stdout:1868: writing smithi16030472-91 from 1626315 to 2067492 tid 2 2024-09-21T18:08:46.421 INFO:tasks.rados.rados.0.smithi160.stdout:1869: cache_flush oid 487 snap -1 2024-09-21T18:08:46.421 INFO:tasks.rados.rados.0.smithi160.stdout:1870: delete oid 477 current snap is 0 2024-09-21T18:08:46.421 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:46.410+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:46.421 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:46.422 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:46.439 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:46.440 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:46.460 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:46.460 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:46.480 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:46.480 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:46.497 INFO:tasks.rados.rados.0.smithi160.stdout:1856: finishing write tid 1 to smithi16030472-195 2024-09-21T18:08:46.498 INFO:tasks.rados.rados.0.smithi160.stdout:1856: finishing write tid 2 to smithi16030472-195 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: finishing write tid 3 to smithi16030472-195 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: finishing write tid 4 to smithi16030472-195 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: oid 195 updating version 0 to 698 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: oid 195 version 698 is already newer than 696 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: oid 195 version 698 is already newer than 697 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 698 (ObjNum 952 snap 0 seq_num 952) dirty exists 2024-09-21T18:08:46.501 INFO:tasks.rados.rados.0.smithi160.stdout:1856: left oid 195 (ObjNum 952 snap 0 seq_num 952) 2024-09-21T18:08:46.502 INFO:tasks.rados.rados.0.smithi160.stdout:1859: got (0) Success 2024-09-21T18:08:46.502 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:08:46.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:46.514 INFO:tasks.rados.rados.0.smithi160.stdout:1863: finishing write tid 1 to smithi16030472-45 2024-09-21T18:08:46.516 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:46.510+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:46.534 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:46.534 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:46.555 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:46.555 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:46.573 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:5407 4.2 4:7c14251d:::smithi16030472-108:head [cache-try-flush] snapc 0=[] ondisk+read+skiprwlocks+known_if_redirected+supports_pool_eio e83)", 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:08:45.879423+0000", 2024-09-21T18:08:46.591 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.70989445900000003, 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.709943879, 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 5407 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879423+0000", 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879423+0000", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879426+0000", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8870000000000002e-06 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879441+0000", 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4153e-05 2024-09-21T18:08:46.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879469+0000", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8822999999999998e-05 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:45.879493+0000", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3686e-05 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:46.211505+0000", 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33201182499999998 2024-09-21T18:08:46.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:08:46.211595+0000", 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0489999999999996e-05 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-21T18:08:46.595 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:08:46.596 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:46.596 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:46.608 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:46.616 INFO:tasks.rados.rados.0.smithi160.stdout:1863: finishing write tid 2 to smithi16030472-45 2024-09-21T18:08:46.616 INFO:tasks.rados.rados.0.smithi160.stdout:1863: finishing write tid 3 to smithi16030472-45 2024-09-21T18:08:46.616 INFO:tasks.rados.rados.0.smithi160.stdout:1863: finishing write tid 4 to smithi16030472-45 2024-09-21T18:08:46.616 INFO:tasks.rados.rados.0.smithi160.stdout:1863: oid 45 updating version 0 to 701 2024-09-21T18:08:46.617 INFO:tasks.rados.rados.0.smithi160.stdout:1863: oid 45 version 701 is already newer than 699 2024-09-21T18:08:46.617 INFO:tasks.rados.rados.0.smithi160.stdout:1863: oid 45 version 701 is already newer than 700 2024-09-21T18:08:46.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 45 v 701 (ObjNum 954 snap 0 seq_num 954) dirty exists 2024-09-21T18:08:46.617 INFO:tasks.rados.rados.0.smithi160.stdout:1863: left oid 45 (ObjNum 954 snap 0 seq_num 954) 2024-09-21T18:08:46.619 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:46.610+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:46.621 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:46.621 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:1858: finishing copy_from racing read to smithi16030472-276 2024-09-21T18:08:46.622 INFO:tasks.rados.rados.0.smithi160.stdout:1864: finishing write tid 1 to smithi16030472-100 2024-09-21T18:08:46.631 INFO:tasks.rados.rados.0.smithi160.stdout:1860: finishing write tid 1 to smithi16030472-305 2024-09-21T18:08:46.631 INFO:tasks.rados.rados.0.smithi160.stdout:1865: got (0) Success 2024-09-21T18:08:46.631 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 0 (ObjNum 767 snap 0 seq_num 767) dirty exists 2024-09-21T18:08:46.631 INFO:tasks.rados.rados.0.smithi160.stdout:1861: finishing copy_from racing read to smithi16030472-403 2024-09-21T18:08:46.632 INFO:tasks.rados.rados.0.smithi160.stdout:1861: finishing copy_from to smithi16030472-403 2024-09-21T18:08:46.632 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 403 v 348 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:08:46.632 INFO:tasks.rados.rados.0.smithi160.stdout:1857: finishing copy_from racing read to smithi16030472-463 2024-09-21T18:08:46.637 INFO:tasks.rados.rados.0.smithi160.stdout:1857: finishing copy_from to smithi16030472-463 2024-09-21T18:08:46.637 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 463 v 68 (ObjNum 108 snap 0 seq_num 108) dirty exists 2024-09-21T18:08:46.638 INFO:tasks.rados.rados.0.smithi160.stdout:1852: got (0) Success 2024-09-21T18:08:46.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 108 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:08:46.651 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:46.651 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:46.671 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:46.673 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:46.694 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:46.695 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:46.715 INFO:tasks.rados.rados.0.smithi160.stdout:1864: finishing write tid 2 to smithi16030472-100 2024-09-21T18:08:46.715 INFO:tasks.rados.rados.0.smithi160.stdout:1864: finishing write tid 3 to smithi16030472-100 2024-09-21T18:08:46.718 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:46.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:46.719 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:46.714+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:46.733 INFO:tasks.rados.rados.0.smithi160.stdout:1860: finishing write tid 2 to smithi16030472-305 2024-09-21T18:08:46.734 INFO:tasks.rados.rados.0.smithi160.stdout:1868: finishing write tid 1 to smithi16030472-91 2024-09-21T18:08:46.734 INFO:tasks.rados.rados.0.smithi160.stdout:1868: finishing write tid 2 to smithi16030472-91 2024-09-21T18:08:46.734 INFO:tasks.rados.rados.0.smithi160.stdout:1868: finishing write tid 3 to smithi16030472-91 2024-09-21T18:08:46.735 INFO:tasks.rados.rados.0.smithi160.stdout:1852: done (15 left) 2024-09-21T18:08:46.735 INFO:tasks.rados.rados.0.smithi160.stdout:1856: done (14 left) 2024-09-21T18:08:46.735 INFO:tasks.rados.rados.0.smithi160.stdout:1857: done (13 left) 2024-09-21T18:08:46.735 INFO:tasks.rados.rados.0.smithi160.stdout:1859: done (12 left) 2024-09-21T18:08:46.735 INFO:tasks.rados.rados.0.smithi160.stdout:1861: done (11 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1862: done (10 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1863: done (9 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1865: done (8 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1866: done (7 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1870: done (6 left) 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1871: write (excl) oid 47 current snap is 0 2024-09-21T18:08:46.736 INFO:tasks.rados.rados.0.smithi160.stdout:1871: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-21T18:08:46.739 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:46.739 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:46.739 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:46.739 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:46.743 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:46.744 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:46.745 INFO:tasks.rados.rados.0.smithi160.stdout:1871: writing smithi16030472-47 from 501106 to 1250126 tid 1 2024-09-21T18:08:46.749 INFO:tasks.rados.rados.0.smithi160.stdout:1871: writing smithi16030472-47 from 1743279 to 2101106 tid 2 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1872: read oid 172 snap -1 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1872: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1873: read oid 58 snap -1 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1873: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1874: cache_try_flush oid 146 snap -1 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1860: finishing write tid 3 to smithi16030472-305 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1875: write (excl) oid 94 current snap is 0 2024-09-21T18:08:46.750 INFO:tasks.rados.rados.0.smithi160.stdout:1875: seq_num 959 ranges {517913=424269,1401761=446989} 2024-09-21T18:08:46.756 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:46.757 INFO:tasks.rados.rados.0.smithi160.stdout:1875: writing smithi16030472-94 from 517913 to 942182 tid 1 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1875: writing smithi16030472-94 from 1401761 to 1848750 tid 2 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1876: write oid 19 current snap is 0 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1860: finishing write tid 4 to smithi16030472-305 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1860: oid 305 updating version 0 to 629 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1860: oid 305 version 629 is already newer than 628 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1860: oid 305 updating version 629 to 630 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 305 v 630 (ObjNum 953 snap 0 seq_num 953) dirty exists 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1860: left oid 305 (ObjNum 953 snap 0 seq_num 953) 2024-09-21T18:08:46.765 INFO:tasks.rados.rados.0.smithi160.stdout:1876: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-21T18:08:46.773 INFO:tasks.rados.rados.0.smithi160.stdout:1876: writing smithi16030472-19 from 534720 to 1034238 tid 1 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1876: writing smithi16030472-19 from 1460243 to 2134720 tid 2 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1860: done (11 left) 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1877: read oid 178 snap -1 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1877: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1878: cache_evict oid 461 snap -1 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1879: write oid 362 current snap is 0 2024-09-21T18:08:46.783 INFO:tasks.rados.rados.0.smithi160.stdout:1879: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-21T18:08:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:1879: writing smithi16030472-362 from 551527 to 1126294 tid 1 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1879: writing smithi16030472-362 from 1918725 to 2151527 tid 2 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1880: cache_flush oid 267 snap -1 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1878: got (16) Device or resource busy 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1878: done (14 left) 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1881: cache_evict oid 7 snap -1 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1882: write (excl) oid 399 current snap is 0 2024-09-21T18:08:46.800 INFO:tasks.rados.rados.0.smithi160.stdout:1882: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-21T18:08:46.808 INFO:tasks.rados.rados.0.smithi160.stdout:1882: writing smithi16030472-399 from 568334 to 1218350 tid 1 2024-09-21T18:08:46.814 INFO:tasks.rados.rados.0.smithi160.stdout:1882: writing smithi16030472-399 from 1860854 to 2168334 tid 2 2024-09-21T18:08:46.814 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: finishing write tid 4 to smithi16030472-100 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: oid 100 updating version 0 to 704 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: oid 100 version 704 is already newer than 702 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: oid 100 version 704 is already newer than 703 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 100 v 704 (ObjNum 955 snap 0 seq_num 955) dirty exists 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: left oid 100 (ObjNum 955 snap 0 seq_num 955) 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1864: done (15 left) 2024-09-21T18:08:46.817 INFO:tasks.rados.rados.0.smithi160.stdout:1883: read oid 361 snap -1 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1883: expect (ObjNum 360 snap 0 seq_num 360) 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1858: finishing copy_from to smithi16030472-276 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 537 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1858: done (15 left) 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1884: cache_try_flush oid 429 snap -1 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1869: got (0) Success 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 487 v 0 (ObjNum 570 snap 0 seq_num 570) dirty exists 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1869: done (15 left) 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1885: write oid 85 current snap is 0 2024-09-21T18:08:46.818 INFO:tasks.rados.rados.0.smithi160.stdout:1885: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-21T18:08:46.819 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:46.813+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:46.826 INFO:tasks.rados.rados.0.smithi160.stdout:1885: writing smithi16030472-85 from 585141 to 1310406 tid 1 2024-09-21T18:08:46.832 INFO:tasks.rados.rados.0.smithi160.stdout:1885: writing smithi16030472-85 from 1919336 to 2185141 tid 2 2024-09-21T18:08:46.832 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: finishing write tid 4 to smithi16030472-91 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: oid 91 updating version 0 to 675 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: oid 91 updating version 675 to 676 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: oid 91 updating version 676 to 677 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 677 (ObjNum 956 snap 0 seq_num 956) dirty exists 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: left oid 91 (ObjNum 956 snap 0 seq_num 956) 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1868: done (15 left) 2024-09-21T18:08:46.836 INFO:tasks.rados.rados.0.smithi160.stdout:1886: cache_flush oid 438 snap -1 2024-09-21T18:08:46.837 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:46.873 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:46.873 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:46.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:46.920 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:46.916+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:46.929 INFO:tasks.rados.rados.0.smithi160.stdout:1877: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:46.938 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1864540 2024-09-21T18:08:46.992 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:47.021 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:47.020+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:47.028 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.028 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:47.051 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2182593 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1880: got (0) Success 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 267 v 0 (ObjNum 266 snap 0 seq_num 266) dirty exists 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1867: finishing copy_from racing read to smithi16030472-73 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1871: finishing write tid 1 to smithi16030472-47 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1871: finishing write tid 2 to smithi16030472-47 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1871: finishing write tid 3 to smithi16030472-47 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1877: done (15 left) 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1880: done (14 left) 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1887: write oid 82 current snap is 0 2024-09-21T18:08:47.063 INFO:tasks.rados.rados.0.smithi160.stdout:1887: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-21T18:08:47.071 INFO:tasks.rados.rados.0.smithi160.stdout:1887: writing smithi16030472-82 from 601948 to 1002462 tid 1 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1887: writing smithi16030472-82 from 1577818 to 2201948 tid 2 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1888: read oid 307 snap -1 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1888: expect deleted 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1874: got (0) Success 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 0 (ObjNum 725 snap 0 seq_num 725) dirty exists 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1867: finishing copy_from to smithi16030472-73 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 73 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1871: finishing write tid 4 to smithi16030472-47 2024-09-21T18:08:47.083 INFO:tasks.rados.rados.0.smithi160.stdout:1871: oid 47 updating version 0 to 700 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1871: oid 47 version 700 is already newer than 699 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1871: oid 47 version 700 is already newer than 698 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 700 (ObjNum 958 snap 0 seq_num 958) dirty exists 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1871: left oid 47 (ObjNum 958 snap 0 seq_num 958) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1867: done (15 left) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1871: done (14 left) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1874: done (13 left) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1889: cache_try_flush oid 163 snap -1 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1881: got (0) Success 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1881: done (13 left) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1890: read oid 90 snap -1 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1890: expect (ObjNum 766 snap 0 seq_num 766) 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1891: cache_flush oid 152 snap -1 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1892: write (excl) oid 223 current snap is 0 2024-09-21T18:08:47.084 INFO:tasks.rados.rados.0.smithi160.stdout:1892: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-21T18:08:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:1892: writing smithi16030472-223 from 618755 to 1094518 tid 1 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout:1892: writing smithi16030472-223 from 1636300 to 2094296 tid 2 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout:1888: done (15 left) 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout:1893: read oid 434 snap -1 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout:1893: expect deleted 2024-09-21T18:08:47.101 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:47.123 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:47.117+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:47.147 INFO:tasks.rados.rados.0.smithi160.stdout:1876: finishing write tid 1 to smithi16030472-19 2024-09-21T18:08:47.149 INFO:tasks.rados.rados.0.smithi160.stdout:1876: finishing write tid 2 to smithi16030472-19 2024-09-21T18:08:47.150 INFO:tasks.rados.rados.0.smithi160.stdout:1872: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:47.172 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.172 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:47.224 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:47.217+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:47.273 INFO:tasks.rados.rados.0.smithi160.stdout:1873: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-21T18:08:47.276 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 655318 2024-09-21T18:08:47.313 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.313 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.325 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:47.324+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:47.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:47.356 INFO:tasks.rados.rados.0.smithi160.stdout:1883: expect (ObjNum 360 snap 0 seq_num 360) 2024-09-21T18:08:47.359 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 653868 2024-09-21T18:08:47.388 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:47.409 INFO:tasks.rados.rados.0.smithi160.stdout:1884: got (0) Success 2024-09-21T18:08:47.409 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 0 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-21T18:08:47.409 INFO:tasks.rados.rados.0.smithi160.stdout:1882: finishing write tid 1 to smithi16030472-399 2024-09-21T18:08:47.409 INFO:tasks.rados.rados.0.smithi160.stdout:1886: got (0) Success 2024-09-21T18:08:47.410 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 438 v 0 (ObjNum 437 snap 0 seq_num 437) dirty exists 2024-09-21T18:08:47.410 INFO:tasks.rados.rados.0.smithi160.stdout:1891: got (2) No such file or directory 2024-09-21T18:08:47.410 INFO:tasks.rados.rados.0.smithi160.stdout:1882: finishing write tid 2 to smithi16030472-399 2024-09-21T18:08:47.410 INFO:tasks.rados.rados.0.smithi160.stdout:1876: finishing write tid 3 to smithi16030472-19 2024-09-21T18:08:47.410 INFO:tasks.rados.rados.0.smithi160.stdout:1890: expect (ObjNum 766 snap 0 seq_num 766) 2024-09-21T18:08:47.425 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:47.420+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:47.453 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.454 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:47.475 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2716790 2024-09-21T18:08:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: finishing write tid 1 to smithi16030472-94 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: finishing write tid 2 to smithi16030472-94 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: finishing write tid 3 to smithi16030472-94 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: finishing write tid 4 to smithi16030472-94 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: oid 94 updating version 0 to 631 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: oid 94 updating version 631 to 633 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: oid 94 version 633 is already newer than 632 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 633 (ObjNum 959 snap 0 seq_num 959) dirty exists 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1875: left oid 94 (ObjNum 959 snap 0 seq_num 959) 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1879: finishing write tid 1 to smithi16030472-362 2024-09-21T18:08:47.525 INFO:tasks.rados.rados.0.smithi160.stdout:1879: finishing write tid 2 to smithi16030472-362 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: finishing write tid 3 to smithi16030472-362 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: finishing write tid 4 to smithi16030472-362 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: oid 362 updating version 0 to 683 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: oid 362 version 683 is already newer than 682 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: oid 362 version 683 is already newer than 681 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 362 v 683 (ObjNum 961 snap 0 seq_num 961) dirty exists 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1879: left oid 362 (ObjNum 961 snap 0 seq_num 961) 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: finishing write tid 1 to smithi16030472-85 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: finishing write tid 2 to smithi16030472-85 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: finishing write tid 3 to smithi16030472-85 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: finishing write tid 4 to smithi16030472-85 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: oid 85 updating version 0 to 686 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: oid 85 version 686 is already newer than 685 2024-09-21T18:08:47.526 INFO:tasks.rados.rados.0.smithi160.stdout:1885: oid 85 version 686 is already newer than 684 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 686 (ObjNum 963 snap 0 seq_num 963) dirty exists 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1885: left oid 85 (ObjNum 963 snap 0 seq_num 963) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1872: done (15 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1873: done (14 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1875: done (13 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1879: done (12 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1883: done (11 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1884: done (10 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1885: done (9 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1886: done (8 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1890: done (7 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1891: done (6 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1893: done (5 left) 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1894: copy_from oid 76 from oid 358 current snap is 0 2024-09-21T18:08:47.527 INFO:tasks.rados.rados.0.smithi160.stdout:1895: delete oid 434 current snap is 0 2024-09-21T18:08:47.529 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:47.522+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:47.586 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.586 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: finishing write tid 3 to smithi16030472-399 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: finishing write tid 4 to smithi16030472-399 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: oid 399 updating version 0 to 670 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: oid 399 updating version 670 to 687 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: oid 399 version 687 is already newer than 671 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 399 v 687 (ObjNum 962 snap 0 seq_num 962) dirty exists 2024-09-21T18:08:47.625 INFO:tasks.rados.rados.0.smithi160.stdout:1882: left oid 399 (ObjNum 962 snap 0 seq_num 962) 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1876: finishing write tid 4 to smithi16030472-19 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1876: oid 19 updating version 0 to 680 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1876: oid 19 version 680 is already newer than 679 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1876: oid 19 version 680 is already newer than 678 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 19 v 680 (ObjNum 960 snap 0 seq_num 960) dirty exists 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1876: left oid 19 (ObjNum 960 snap 0 seq_num 960) 2024-09-21T18:08:47.626 INFO:tasks.rados.rados.0.smithi160.stdout:1889: got (2) No such file or directory 2024-09-21T18:08:47.629 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:47.624+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:1887: finishing write tid 1 to smithi16030472-82 2024-09-21T18:08:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:1887: finishing write tid 2 to smithi16030472-82 2024-09-21T18:08:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:1887: finishing write tid 3 to smithi16030472-82 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1887: finishing write tid 4 to smithi16030472-82 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1887: oid 82 updating version 0 to 689 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1887: oid 82 updating version 689 to 690 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1887: oid 82 version 690 is already newer than 688 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 690 (ObjNum 964 snap 0 seq_num 964) dirty exists 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1887: left oid 82 (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:08:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:1892: finishing write tid 1 to smithi16030472-223 2024-09-21T18:08:47.700 INFO:tasks.rados.rados.0.smithi160.stdout:1892: finishing write tid 2 to smithi16030472-223 2024-09-21T18:08:47.719 INFO:tasks.rados.rados.0.smithi160.stdout:1894: finishing copy_from racing read to smithi16030472-76 2024-09-21T18:08:47.731 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:47.728+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:47.764 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.764 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:47.824 INFO:tasks.rados.rados.0.smithi160.stdout:1892: finishing write tid 3 to smithi16030472-223 2024-09-21T18:08:47.824 INFO:tasks.rados.rados.0.smithi160.stdout:1892: finishing write tid 4 to smithi16030472-223 2024-09-21T18:08:47.824 INFO:tasks.rados.rados.0.smithi160.stdout:1892: oid 223 updating version 0 to 691 2024-09-21T18:08:47.824 INFO:tasks.rados.rados.0.smithi160.stdout:1892: oid 223 updating version 691 to 692 2024-09-21T18:08:47.825 INFO:tasks.rados.rados.0.smithi160.stdout:1892: oid 223 updating version 692 to 693 2024-09-21T18:08:47.825 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 223 v 693 (ObjNum 965 snap 0 seq_num 965) dirty exists 2024-09-21T18:08:47.825 INFO:tasks.rados.rados.0.smithi160.stdout:1892: left oid 223 (ObjNum 965 snap 0 seq_num 965) 2024-09-21T18:08:47.829 INFO:tasks.rados.rados.0.smithi160.stdout:1876: done (6 left) 2024-09-21T18:08:47.829 INFO:tasks.rados.rados.0.smithi160.stdout:1882: done (5 left) 2024-09-21T18:08:47.829 INFO:tasks.rados.rados.0.smithi160.stdout:1887: done (4 left) 2024-09-21T18:08:47.829 INFO:tasks.rados.rados.0.smithi160.stdout:1889: done (3 left) 2024-09-21T18:08:47.830 INFO:tasks.rados.rados.0.smithi160.stdout:1892: done (2 left) 2024-09-21T18:08:47.830 INFO:tasks.rados.rados.0.smithi160.stdout:1895: done (1 left) 2024-09-21T18:08:47.830 INFO:tasks.rados.rados.0.smithi160.stdout:1896: copy_from oid 165 from oid 344 current snap is 0 2024-09-21T18:08:47.830 INFO:tasks.rados.rados.0.smithi160.stdout:1897: delete oid 72 current snap is 0 2024-09-21T18:08:47.831 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:47.828+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:47.854 INFO:tasks.rados.rados.0.smithi160.stdout:1896: finishing copy_from racing read to smithi16030472-165 2024-09-21T18:08:47.903 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:47.903 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:47.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:47.923 INFO:tasks.rados.rados.0.smithi160.stdout:1894: finishing copy_from to smithi16030472-76 2024-09-21T18:08:47.923 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 255 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-21T18:08:47.932 INFO:tasks.rados.rados.0.smithi160.stdout:1896: finishing copy_from to smithi16030472-165 2024-09-21T18:08:47.932 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 165 v 620 (ObjNum 815 snap 0 seq_num 815) dirty exists 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1894: done (2 left) 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1896: done (1 left) 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1897: done (0 left) 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1898: cache_try_flush oid 138 snap -1 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1899: cache_evict oid 173 snap -1 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1900: copy_from oid 480 from oid 89 current snap is 0 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1901: cache_try_flush oid 333 snap -1 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1902: read oid 331 snap -1 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1902: expect deleted 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1903: copy_from oid 295 from oid 242 current snap is 0 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1904: read oid 337 snap -1 2024-09-21T18:08:47.962 INFO:tasks.rados.rados.0.smithi160.stdout:1904: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-21T18:08:47.963 INFO:tasks.rados.rados.0.smithi160.stdout:1905: cache_flush oid 246 snap -1 2024-09-21T18:08:47.963 INFO:tasks.rados.rados.0.smithi160.stdout:1906: write (excl) oid 367 current snap is 0 2024-09-21T18:08:47.963 INFO:tasks.rados.rados.0.smithi160.stdout:1906: seq_num 968 ranges {669176=417863,1811746=457430} 2024-09-21T18:08:47.963 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:47.958+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:47.979 INFO:tasks.rados.rados.0.smithi160.stdout:1906: writing smithi16030472-367 from 669176 to 1087039 tid 1 2024-09-21T18:08:47.999 INFO:tasks.rados.rados.0.smithi160.stdout:1906: writing smithi16030472-367 from 1811746 to 2269176 tid 2 2024-09-21T18:08:47.999 INFO:tasks.rados.rados.0.smithi160.stdout:1907: cache_evict oid 13 snap -1 2024-09-21T18:08:47.999 INFO:tasks.rados.rados.0.smithi160.stdout:1898: got (0) Success 2024-09-21T18:08:47.999 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 138 v 0 (ObjNum 137 snap 0 seq_num 137) dirty exists 2024-09-21T18:08:48.000 INFO:tasks.rados.rados.0.smithi160.stdout:1898: done (9 left) 2024-09-21T18:08:48.000 INFO:tasks.rados.rados.0.smithi160.stdout:1908: write oid 332 current snap is 0 2024-09-21T18:08:48.000 INFO:tasks.rados.rados.0.smithi160.stdout:1908: seq_num 969 ranges {685983=493112,1870228=415755} 2024-09-21T18:08:48.018 INFO:tasks.rados.rados.0.smithi160.stdout:1908: writing smithi16030472-332 from 685983 to 1179095 tid 1 2024-09-21T18:08:48.032 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:48.028+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:48.034 INFO:tasks.rados.rados.0.smithi160.stdout:1908: writing smithi16030472-332 from 1870228 to 2285983 tid 2 2024-09-21T18:08:48.034 INFO:tasks.rados.rados.0.smithi160.stdout:1909: write (excl) oid 192 current snap is 0 2024-09-21T18:08:48.034 INFO:tasks.rados.rados.0.smithi160.stdout:1909: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-21T18:08:48.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:08:48.045 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.045 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.048 INFO:tasks.rados.rados.0.smithi160.stdout:1909: writing smithi16030472-192 from 702790 to 1271151 tid 1 2024-09-21T18:08:48.058 INFO:tasks.rados.rados.0.smithi160.stdout:1909: writing smithi16030472-192 from 1812357 to 2302790 tid 2 2024-09-21T18:08:48.058 INFO:tasks.rados.rados.0.smithi160.stdout:1910: cache_evict oid 17 snap -1 2024-09-21T18:08:48.058 INFO:tasks.rados.rados.0.smithi160.stdout:1911: write (excl) oid 182 current snap is 0 2024-09-21T18:08:48.058 INFO:tasks.rados.rados.0.smithi160.stdout:1911: seq_num 971 ranges {719597=643610,1870839=448758} 2024-09-21T18:08:48.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:48.069 INFO:tasks.rados.rados.0.smithi160.stdout:1911: writing smithi16030472-182 from 719597 to 1363207 tid 1 2024-09-21T18:08:48.076 INFO:tasks.rados.rados.0.smithi160.stdout:1911: writing smithi16030472-182 from 1870839 to 2319597 tid 2 2024-09-21T18:08:48.076 INFO:tasks.rados.rados.0.smithi160.stdout:1912: delete oid 500 current snap is 0 2024-09-21T18:08:48.104 INFO:tasks.rados.rados.0.smithi160.stdout:1904: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-21T18:08:48.134 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:48.128+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:48.224 INFO:tasks.rados.rados.0.smithi160.stdout:1907: got (16) Device or resource busy 2024-09-21T18:08:48.224 INFO:tasks.rados.rados.0.smithi160.stdout:1903: finishing copy_from to smithi16030472-295 2024-09-21T18:08:48.224 INFO:tasks.rados.rados.0.smithi160.stdout:1903: got expected ENOENT (src dne) 2024-09-21T18:08:48.224 INFO:tasks.rados.rados.0.smithi160.stdout:1903: finishing copy_from racing read to smithi16030472-295 2024-09-21T18:08:48.225 INFO:tasks.rados.rados.0.smithi160.stdout:1899: got (0) Success 2024-09-21T18:08:48.225 INFO:tasks.rados.rados.0.smithi160.stdout:1905: got (2) No such file or directory 2024-09-21T18:08:48.225 INFO:tasks.rados.rados.0.smithi160.stdout:1901: got (0) Success 2024-09-21T18:08:48.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 0 (ObjNum 847 snap 0 seq_num 847) dirty exists 2024-09-21T18:08:48.228 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.228 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.235 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:48.230+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:48.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:48.254 INFO:tasks.rados.rados.0.smithi160.stdout:1910: got (0) Success 2024-09-21T18:08:48.271 INFO:tasks.rados.rados.0.smithi160.stdout:1908: finishing write tid 1 to smithi16030472-332 2024-09-21T18:08:48.274 INFO:tasks.rados.rados.0.smithi160.stdout:1899: done (13 left) 2024-09-21T18:08:48.274 INFO:tasks.rados.rados.0.smithi160.stdout:1901: done (12 left) 2024-09-21T18:08:48.274 INFO:tasks.rados.rados.0.smithi160.stdout:1902: done (11 left) 2024-09-21T18:08:48.274 INFO:tasks.rados.rados.0.smithi160.stdout:1903: done (10 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1904: done (9 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1905: done (8 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1907: done (7 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1910: done (6 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1912: done (5 left) 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1913: cache_flush oid 349 snap -1 2024-09-21T18:08:48.275 INFO:tasks.rados.rados.0.smithi160.stdout:1914: delete oid 329 current snap is 0 2024-09-21T18:08:48.336 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:48.330+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1914: done (6 left) 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1915: read oid 292 snap -1 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1915: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1916: cache_flush oid 300 snap -1 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1917: write (excl) oid 281 current snap is 0 2024-09-21T18:08:48.371 INFO:tasks.rados.rados.0.smithi160.stdout:1917: seq_num 974 ranges {770018=585710,2046285=323733} 2024-09-21T18:08:48.380 INFO:tasks.rados.rados.0.smithi160.stdout:1917: writing smithi16030472-281 from 770018 to 1355728 tid 1 2024-09-21T18:08:48.387 INFO:tasks.rados.rados.0.smithi160.stdout:1917: writing smithi16030472-281 from 2046285 to 2370018 tid 2 2024-09-21T18:08:48.388 INFO:tasks.rados.rados.0.smithi160.stdout:1918: write (excl) oid 55 current snap is 0 2024-09-21T18:08:48.388 INFO:tasks.rados.rados.0.smithi160.stdout:1918: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-21T18:08:48.399 INFO:tasks.rados.rados.0.smithi160.stdout:1918: writing smithi16030472-55 from 786825 to 1447784 tid 1 2024-09-21T18:08:48.405 INFO:tasks.rados.rados.0.smithi160.stdout:1918: writing smithi16030472-55 from 2104767 to 2386825 tid 2 2024-09-21T18:08:48.405 INFO:tasks.rados.rados.0.smithi160.stdout:1919: copy_from oid 159 from oid 22 current snap is 0 2024-09-21T18:08:48.406 INFO:tasks.rados.rados.0.smithi160.stdout:1908: finishing write tid 2 to smithi16030472-332 2024-09-21T18:08:48.406 INFO:tasks.rados.rados.0.smithi160.stdout:1920: write (excl) oid 353 current snap is 0 2024-09-21T18:08:48.406 INFO:tasks.rados.rados.0.smithi160.stdout:1920: seq_num 976 ranges {403632=736208,1763249=619999} 2024-09-21T18:08:48.406 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.406 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.413 INFO:tasks.rados.rados.0.smithi160.stdout:1920: writing smithi16030472-353 from 403632 to 1139840 tid 1 2024-09-21T18:08:48.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:48.422 INFO:tasks.rados.rados.0.smithi160.stdout:1920: writing smithi16030472-353 from 1763249 to 2383248 tid 2 2024-09-21T18:08:48.422 INFO:tasks.rados.rados.0.smithi160.stdout:1921: cache_try_flush oid 450 snap -1 2024-09-21T18:08:48.422 INFO:tasks.rados.rados.0.smithi160.stdout:1908: finishing write tid 3 to smithi16030472-332 2024-09-21T18:08:48.422 INFO:tasks.rados.rados.0.smithi160.stdout:1908: finishing write tid 4 to smithi16030472-332 2024-09-21T18:08:48.422 INFO:tasks.rados.rados.0.smithi160.stdout:1908: oid 332 updating version 0 to 706 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1908: oid 332 updating version 706 to 707 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1908: oid 332 version 707 is already newer than 705 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 707 (ObjNum 969 snap 0 seq_num 969) dirty exists 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1908: left oid 332 (ObjNum 969 snap 0 seq_num 969) 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1908: done (12 left) 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1922: read oid 293 snap -1 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1922: expect deleted 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1923: read oid 137 snap -1 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1923: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1924: read oid 149 snap -1 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1924: expect (ObjNum 789 snap 0 seq_num 789) 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1925: write oid 422 current snap is 0 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1909: finishing write tid 1 to smithi16030472-192 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1913: got (2) No such file or directory 2024-09-21T18:08:48.423 INFO:tasks.rados.rados.0.smithi160.stdout:1900: finishing copy_from racing read to smithi16030472-480 2024-09-21T18:08:48.424 INFO:tasks.rados.rados.0.smithi160.stdout:1925: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-21T18:08:48.428 INFO:tasks.rados.rados.0.smithi160.stdout:1925: writing smithi16030472-422 from 420439 to 831896 tid 1 2024-09-21T18:08:48.436 INFO:tasks.rados.rados.0.smithi160.stdout:1925: writing smithi16030472-422 from 1421731 to 1901741 tid 2 2024-09-21T18:08:48.436 INFO:tasks.rados.rados.0.smithi160.stdout:1913: done (15 left) 2024-09-21T18:08:48.436 INFO:tasks.rados.rados.0.smithi160.stdout:1926: write (excl) oid 139 current snap is 0 2024-09-21T18:08:48.436 INFO:tasks.rados.rados.0.smithi160.stdout:1926: seq_num 978 ranges {437246=486706,1363860=623668} 2024-09-21T18:08:48.442 INFO:tasks.rados.rados.0.smithi160.stdout:1926: writing smithi16030472-139 from 437246 to 923952 tid 1 2024-09-21T18:08:48.448 INFO:tasks.rados.rados.0.smithi160.stdout:1926: writing smithi16030472-139 from 1363860 to 1987528 tid 2 2024-09-21T18:08:48.449 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:48.467 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:48.462+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:48.486 INFO:tasks.rados.rados.0.smithi160.stdout:1915: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:48.500 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1776869 2024-09-21T18:08:48.537 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:48.532+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:48.541 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 412050 2024-09-21T18:08:48.585 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.585 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.593 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:48.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:48.638 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:48.636+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:48.647 INFO:tasks.rados.rados.0.smithi160.stdout:1924: expect (ObjNum 789 snap 0 seq_num 789) 2024-09-21T18:08:48.715 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.715 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:48.739 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:48.733+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:48.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:48.776 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1835730 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: finishing write tid 1 to smithi16030472-281 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: finishing write tid 2 to smithi16030472-281 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: finishing write tid 3 to smithi16030472-281 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: finishing write tid 4 to smithi16030472-281 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: oid 281 updating version 0 to 708 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: oid 281 updating version 708 to 709 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: oid 281 updating version 709 to 710 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 281 v 710 (ObjNum 974 snap 0 seq_num 974) dirty exists 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1917: left oid 281 (ObjNum 974 snap 0 seq_num 974) 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1918: finishing write tid 1 to smithi16030472-55 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1918: finishing write tid 2 to smithi16030472-55 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1918: finishing write tid 3 to smithi16030472-55 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1918: finishing write tid 4 to smithi16030472-55 2024-09-21T18:08:48.811 INFO:tasks.rados.rados.0.smithi160.stdout:1918: oid 55 updating version 0 to 703 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1918: oid 55 version 703 is already newer than 702 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1918: oid 55 updating version 703 to 704 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 55 v 704 (ObjNum 975 snap 0 seq_num 975) dirty exists 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1918: left oid 55 (ObjNum 975 snap 0 seq_num 975) 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: finishing write tid 1 to smithi16030472-353 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: finishing write tid 2 to smithi16030472-353 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1921: got (0) Success 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 450 v 0 (ObjNum 449 snap 0 seq_num 449) dirty exists 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: finishing write tid 3 to smithi16030472-353 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: finishing write tid 4 to smithi16030472-353 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: oid 353 updating version 0 to 706 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: oid 353 version 706 is already newer than 705 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: oid 353 updating version 706 to 707 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 353 v 707 (ObjNum 976 snap 0 seq_num 976) dirty exists 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1920: left oid 353 (ObjNum 976 snap 0 seq_num 976) 2024-09-21T18:08:48.812 INFO:tasks.rados.rados.0.smithi160.stdout:1900: finishing copy_from to smithi16030472-480 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 480 v 25 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1900: done (15 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1915: done (14 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1917: done (13 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1918: done (12 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1920: done (11 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1921: done (10 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1922: done (9 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1924: done (8 left) 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1927: cache_evict oid 449 snap -1 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1919: finishing copy_from racing read to smithi16030472-159 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1919: finishing copy_from to smithi16030472-159 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 159 v 27 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1906: finishing write tid 1 to smithi16030472-367 2024-09-21T18:08:48.813 INFO:tasks.rados.rados.0.smithi160.stdout:1911: finishing write tid 1 to smithi16030472-182 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: finishing write tid 2 to smithi16030472-192 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: finishing write tid 3 to smithi16030472-192 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: finishing write tid 4 to smithi16030472-192 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: oid 192 updating version 0 to 634 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: oid 192 version 634 is already newer than 622 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: oid 192 updating version 634 to 635 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 192 v 635 (ObjNum 970 snap 0 seq_num 970) dirty exists 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1909: left oid 192 (ObjNum 970 snap 0 seq_num 970) 2024-09-21T18:08:48.814 INFO:tasks.rados.rados.0.smithi160.stdout:1923: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:48.840 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:48.834+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:48.867 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:48.867 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:48.882 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:48.893 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:48.894 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:48.912 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:48.912 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1926: finishing write tid 1 to smithi16030472-139 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1926: finishing write tid 2 to smithi16030472-139 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: finishing write tid 1 to smithi16030472-422 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: finishing write tid 2 to smithi16030472-422 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: finishing write tid 3 to smithi16030472-422 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: finishing write tid 4 to smithi16030472-422 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: oid 422 updating version 0 to 638 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: oid 422 version 638 is already newer than 636 2024-09-21T18:08:48.922 INFO:tasks.rados.rados.0.smithi160.stdout:1925: oid 422 version 638 is already newer than 637 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 422 v 638 (ObjNum 977 snap 0 seq_num 977) dirty exists 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1925: left oid 422 (ObjNum 977 snap 0 seq_num 977) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1916: got (0) Success 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 300 v 0 (ObjNum 305 snap 0 seq_num 1) clean dne 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1909: done (8 left) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1916: done (7 left) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1919: done (6 left) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1923: done (5 left) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1925: done (4 left) 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1928: cache_try_flush oid 429 snap -1 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1929: cache_evict oid 350 snap -1 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1930: cache_try_flush oid 259 snap -1 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1931: read oid 195 snap -1 2024-09-21T18:08:48.923 INFO:tasks.rados.rados.0.smithi160.stdout:1931: expect (ObjNum 952 snap 0 seq_num 952) 2024-09-21T18:08:48.924 INFO:tasks.rados.rados.0.smithi160.stdout:1932: delete oid 89 current snap is 0 2024-09-21T18:08:48.932 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:48.933 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:48.940 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:48.936+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:48.954 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:48.954 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:48.999 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:49.000 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:49.000 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:49.000 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:49.000 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:49.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:49.022 INFO:tasks.rados.rados.0.smithi160.stdout:1928: got (0) Success 2024-09-21T18:08:49.022 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 0 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-21T18:08:49.022 INFO:tasks.rados.rados.0.smithi160.stdout:1926: finishing write tid 3 to smithi16030472-139 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1926: finishing write tid 4 to smithi16030472-139 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1926: oid 139 updating version 0 to 694 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1926: oid 139 updating version 694 to 695 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1926: oid 139 updating version 695 to 696 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 696 (ObjNum 978 snap 0 seq_num 978) dirty exists 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1926: left oid 139 (ObjNum 978 snap 0 seq_num 978) 2024-09-21T18:08:49.023 INFO:tasks.rados.rados.0.smithi160.stdout:1927: got (0) Success 2024-09-21T18:08:49.028 INFO:tasks.rados.rados.0.smithi160.stdout:1931: expect (ObjNum 952 snap 0 seq_num 952) 2024-09-21T18:08:49.035 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:08:49.035 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:49.030+0000 7ff40b999640 -1 osd.4 84 handle_osd_map injecting map crc failure 2024-09-21T18:08:49.041 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:49.036+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:49.048 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:49.048 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:49.049 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:49.069 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:49.080 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1756054 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: finishing write tid 2 to smithi16030472-182 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: finishing write tid 3 to smithi16030472-182 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: finishing write tid 4 to smithi16030472-182 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: oid 182 updating version 0 to 708 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: oid 182 updating version 708 to 709 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: oid 182 version 709 is already newer than 669 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 182 v 709 (ObjNum 971 snap 0 seq_num 971) dirty exists 2024-09-21T18:08:49.083 INFO:tasks.rados.rados.0.smithi160.stdout:1911: left oid 182 (ObjNum 971 snap 0 seq_num 971) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1911: done (8 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1926: done (7 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1927: done (6 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1928: done (5 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1931: done (4 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1932: done (3 left) 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1933: cache_try_flush oid 210 snap -1 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1934: write oid 57 current snap is 0 2024-09-21T18:08:49.084 INFO:tasks.rados.rados.0.smithi160.stdout:1934: seq_num 980 ranges {470860=637204,1880824=590036} 2024-09-21T18:08:49.089 INFO:tasks.rados.rados.0.smithi160.stdout:1934: writing smithi16030472-57 from 470860 to 1108064 tid 1 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1934: writing smithi16030472-57 from 1880824 to 2470860 tid 2 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1935: cache_try_flush oid 310 snap -1 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1936: copy_from oid 67 from oid 180 current snap is 0 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1937: cache_try_flush oid 335 snap -1 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1938: copy_from oid 75 from oid 352 current snap is 0 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1906: finishing write tid 2 to smithi16030472-367 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1939: write oid 285 current snap is 0 2024-09-21T18:08:49.097 INFO:tasks.rados.rados.0.smithi160.stdout:1939: seq_num 981 ranges {487667=428806,1655659=770995} 2024-09-21T18:08:49.098 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:49.098 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:49.102 INFO:tasks.rados.rados.0.smithi160.stdout:1939: writing smithi16030472-285 from 487667 to 916473 tid 1 2024-09-21T18:08:49.111 INFO:tasks.rados.rados.0.smithi160.stdout:1939: writing smithi16030472-285 from 1655659 to 2426654 tid 2 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1940: cache_evict oid 444 snap -1 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1937: got (0) Success 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 335 v 0 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1937: done (10 left) 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1941: copy_from oid 184 from oid 17 current snap is 0 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1942: cache_try_flush oid 239 snap -1 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1943: write (excl) oid 464 current snap is 0 2024-09-21T18:08:49.112 INFO:tasks.rados.rados.0.smithi160.stdout:1943: seq_num 982 ranges {504474=504055,1714141=631006} 2024-09-21T18:08:49.118 INFO:tasks.rados.rados.0.smithi160.stdout:1943: writing smithi16030472-464 from 504474 to 1008529 tid 1 2024-09-21T18:08:49.120 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:49.121 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:49.126 INFO:tasks.rados.rados.0.smithi160.stdout:1943: writing smithi16030472-464 from 1714141 to 2345147 tid 2 2024-09-21T18:08:49.126 INFO:tasks.rados.rados.0.smithi160.stdout:1944: delete oid 25 current snap is 0 2024-09-21T18:08:49.142 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:49.140+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:49.144 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:49.146 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:49.146 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.1036, 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:08:49.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:08:49.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.103360375999998, 2024-09-21T18:08:49.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:08:49.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:08:49.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:08:49.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.103324968999999, 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:08:49.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.162 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.103291795999993, 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:08:49.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:08:49.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.103262236000006, 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.168 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:08:49.169 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:08:49.170 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.171 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.172 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.103232579999997, 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.173 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.174 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.175 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.176 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.177 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.102746323999995, 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.178 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.179 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.180 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.182 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.102712730999997, 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.185 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.002495038999996, 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.187 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:08:49.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.001896134999996, 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.197 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.001181517999996, 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:49.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:08:49.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.991928786000003, 2024-09-21T18:08:49.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.991664084999996, 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:08:49.208 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.991569396000003, 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:08:49.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.991528106000004, 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:08:49.217 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:08:49.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.980258203000005, 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.222 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:08:49.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:08:49.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:08:49.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.971141970000005, 2024-09-21T18:08:49.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:08:49.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:08:49.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:49.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.970078603000005, 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:08:49.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:08:49.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:08:49.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.970038293000002, 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:08:49.237 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:08:49.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.241 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.207400450999998, 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:08:49.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:08:49.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.246 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:49.248 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:49.248 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:49.249 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:49.242+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:49.249 INFO:tasks.rados.rados.0.smithi160.stdout:1936: finishing copy_from racing read to smithi16030472-67 2024-09-21T18:08:49.249 INFO:tasks.rados.rados.0.smithi160.stdout:1906: finishing write tid 3 to smithi16030472-367 2024-09-21T18:08:49.249 INFO:tasks.rados.rados.0.smithi160.stdout:1906: finishing write tid 4 to smithi16030472-367 2024-09-21T18:08:49.249 INFO:tasks.rados.rados.0.smithi160.stdout:1906: oid 367 updating version 0 to 712 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:1906: oid 367 version 712 is already newer than 711 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:1906: oid 367 version 712 is already newer than 668 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 712 (ObjNum 968 snap 0 seq_num 968) dirty exists 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:1906: left oid 367 (ObjNum 968 snap 0 seq_num 968) 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:1936: finishing copy_from to smithi16030472-67 2024-09-21T18:08:49.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 67 v 385 (ObjNum 591 snap 0 seq_num 591) dirty exists 2024-09-21T18:08:49.250 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:49.299 INFO:tasks.rados.rados.0.smithi160.stdout:1929: got (0) Success 2024-09-21T18:08:49.299 INFO:tasks.rados.rados.0.smithi160.stdout:1930: got (0) Success 2024-09-21T18:08:49.299 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 259 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:08:49.300 INFO:tasks.rados.rados.0.smithi160.stdout:1940: got (0) Success 2024-09-21T18:08:49.325 INFO:tasks.rados.rados.0.smithi160.stdout:1934: finishing write tid 1 to smithi16030472-57 2024-09-21T18:08:49.327 INFO:tasks.rados.rados.0.smithi160.stdout:1934: finishing write tid 2 to smithi16030472-57 2024-09-21T18:08:49.327 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:49.327 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:49.349 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:49.341+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:49.350 INFO:tasks.rados.rados.0.smithi160.stdout:1939: finishing write tid 1 to smithi16030472-285 2024-09-21T18:08:49.350 INFO:tasks.rados.rados.0.smithi160.stdout:1939: finishing write tid 2 to smithi16030472-285 2024-09-21T18:08:49.364 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:49.364 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:49.371 INFO:tasks.rados.rados.0.smithi160.stdout:1935: got (0) Success 2024-09-21T18:08:49.371 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 0 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-21T18:08:49.383 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:5614 4.3 e84/33)", 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:49.299114+0000", 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.083689694999999995, 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.083723061000000001, 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "no flag points reached", 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:49.299114+0000", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:49.299114+0000", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:49.299118+0000", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9709999999999998e-06 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:49.299554+0000", 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00043580799999999999 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:49.299574+0000", 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0655000000000001e-05 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.386 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:08:49.387 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-09-21T18:08:49.387 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:49.395 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:49.526 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:49.526 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:49.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:49.641 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:49.636+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:49.643 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:49.644 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:49.396+0000 7f4976023640 -1 osd.5 84 handle_osd_map injecting map crc failure 2024-09-21T18:08:49.645 INFO:tasks.rados.rados.0.smithi160.stdout:1934: finishing write tid 3 to smithi16030472-57 2024-09-21T18:08:49.645 INFO:tasks.rados.rados.0.smithi160.stdout:1933: got (0) Success 2024-09-21T18:08:49.645 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 0 (ObjNum 676 snap 0 seq_num 676) dirty exists 2024-09-21T18:08:49.645 INFO:tasks.rados.rados.0.smithi160.stdout:1934: finishing write tid 4 to smithi16030472-57 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:1934: oid 57 updating version 0 to 673 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:1934: oid 57 updating version 673 to 674 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:1934: oid 57 updating version 674 to 675 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 57 v 675 (ObjNum 980 snap 0 seq_num 980) dirty exists 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:1934: left oid 57 (ObjNum 980 snap 0 seq_num 980) 2024-09-21T18:08:49.646 INFO:tasks.rados.rados.0.smithi160.stdout:1938: finishing copy_from to smithi16030472-75 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1938: got expected ENOENT (src dne) 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1938: finishing copy_from racing read to smithi16030472-75 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1939: finishing write tid 3 to smithi16030472-285 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1939: finishing write tid 4 to smithi16030472-285 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1939: oid 285 updating version 0 to 713 2024-09-21T18:08:49.647 INFO:tasks.rados.rados.0.smithi160.stdout:1939: oid 285 version 713 is already newer than 712 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:1939: oid 285 version 713 is already newer than 711 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 285 v 713 (ObjNum 981 snap 0 seq_num 981) dirty exists 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:1939: left oid 285 (ObjNum 981 snap 0 seq_num 981) 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:1943: finishing write tid 1 to smithi16030472-464 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:1943: finishing write tid 2 to smithi16030472-464 2024-09-21T18:08:49.648 INFO:tasks.rados.rados.0.smithi160.stdout:1941: finishing copy_from racing read to smithi16030472-184 2024-09-21T18:08:49.651 INFO:tasks.rados.rados.0.smithi160.stdout:1943: finishing write tid 3 to smithi16030472-464 2024-09-21T18:08:49.651 INFO:tasks.rados.rados.0.smithi160.stdout:1943: finishing write tid 4 to smithi16030472-464 2024-09-21T18:08:49.651 INFO:tasks.rados.rados.0.smithi160.stdout:1943: oid 464 updating version 0 to 639 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:1943: oid 464 updating version 639 to 641 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:1943: oid 464 version 641 is already newer than 640 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 641 (ObjNum 982 snap 0 seq_num 982) dirty exists 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:1943: left oid 464 (ObjNum 982 snap 0 seq_num 982) 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:1906: done (13 left) 2024-09-21T18:08:49.652 INFO:tasks.rados.rados.0.smithi160.stdout:1929: done (12 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1930: done (11 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1933: done (10 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1934: done (9 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1935: done (8 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1936: done (7 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1938: done (6 left) 2024-09-21T18:08:49.653 INFO:tasks.rados.rados.0.smithi160.stdout:1939: done (5 left) 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1940: done (4 left) 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1943: done (3 left) 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1944: done (2 left) 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1945: cache_try_flush oid 98 snap -1 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1946: copy_from oid 310 from oid 497 current snap is 0 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1947: write (excl) oid 14 current snap is 0 2024-09-21T18:08:49.654 INFO:tasks.rados.rados.0.smithi160.stdout:1947: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-21T18:08:49.655 INFO:tasks.rados.rados.0.smithi160.stdout:1947: writing smithi16030472-14 from 538088 to 1192641 tid 1 2024-09-21T18:08:49.663 INFO:tasks.rados.rados.0.smithi160.stdout:1947: writing smithi16030472-14 from 1714752 to 2349427 tid 2 2024-09-21T18:08:49.663 INFO:tasks.rados.rados.0.smithi160.stdout:1948: write (excl) oid 177 current snap is 0 2024-09-21T18:08:49.663 INFO:tasks.rados.rados.0.smithi160.stdout:1948: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-21T18:08:49.672 INFO:tasks.rados.rados.0.smithi160.stdout:1948: writing smithi16030472-177 from 554895 to 1284697 tid 1 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1948: writing smithi16030472-177 from 1773234 to 2551567 tid 2 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1949: copy_from oid 434 from oid 306 current snap is 0 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1942: got (0) Success 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 239 v 0 (ObjNum 860 snap 0 seq_num 860) dirty exists 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1946: finishing copy_from to smithi16030472-310 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1946: got expected ENOENT (src dne) 2024-09-21T18:08:49.682 INFO:tasks.rados.rados.0.smithi160.stdout:1947: finishing write tid 1 to smithi16030472-14 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1942: done (6 left) 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1950: cache_flush oid 157 snap -1 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1951: read oid 121 snap -1 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1951: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1952: write (excl) oid 101 current snap is 0 2024-09-21T18:08:49.683 INFO:tasks.rados.rados.0.smithi160.stdout:1952: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-21T18:08:49.689 INFO:tasks.rados.rados.0.smithi160.stdout:1952: writing smithi16030472-101 from 571702 to 976753 tid 1 2024-09-21T18:08:49.699 INFO:tasks.rados.rados.0.smithi160.stdout:1952: writing smithi16030472-101 from 1431716 to 2070060 tid 2 2024-09-21T18:08:49.703 INFO:tasks.rados.rados.0.smithi160.stdout:1952: writing smithi16030472-101 from 2560365 to 2571702 tid 3 2024-09-21T18:08:49.703 INFO:tasks.rados.rados.0.smithi160.stdout:1953: read oid 74 snap -1 2024-09-21T18:08:49.703 INFO:tasks.rados.rados.0.smithi160.stdout:1953: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:08:49.703 INFO:tasks.rados.rados.0.smithi160.stdout:1954: cache_try_flush oid 485 snap -1 2024-09-21T18:08:49.704 INFO:tasks.rados.rados.0.smithi160.stdout:1955: delete oid 277 current snap is 0 2024-09-21T18:08:49.740 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:49.733+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:49.785 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:49.785 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:49.785 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:49.785 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:49.785 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:49.799 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:49.804 INFO:tasks.rados.rados.0.smithi160.stdout:1946: finishing copy_from racing read to smithi16030472-310 2024-09-21T18:08:49.805 INFO:tasks.rados.rados.0.smithi160.stdout:1941: finishing copy_from to smithi16030472-184 2024-09-21T18:08:49.805 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 184 v 329 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:08:49.847 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:49.844+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:49.860 INFO:tasks.rados.rados.0.smithi160.stdout:1951: expect (ObjNum 120 snap 0 seq_num 120) 2024-09-21T18:08:49.860 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 125.627592991, 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.930 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:49.934 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 125.62637893199999, 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:08:49.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 125.625028012, 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.942 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 119.337447375, 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:49.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:08:49.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:08:49.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:08:49.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 97.238854744999998, 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:08:49.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 97.238377534999998, 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.951 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:49.955 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 97.237942582000002, 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:08:49.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:08:49.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.959 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.938208134999996, 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.963 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.837638742999999, 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.967 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.837280676000006, 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:49.968 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:08:49.969 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:08:49.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:08:49.971 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.837243842999996, 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:49.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.836784111, 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.976 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:08:49.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:08:49.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.836352583999997, 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.980 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:49.984 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.835415968999996, 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:08:49.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:08:49.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.835377285999996, 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:08:49.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:08:49.992 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.834090634000006, 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:49.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.731946170000001, 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:49.997 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:08:49.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:08:49.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:08:50.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.731222376999995, 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:08:50.001 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:50.005 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 70.605512137999995, 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:08:50.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:08:50.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:08:50.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 45.965717650999999, 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:08:50.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:08:50.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:08:50.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.014 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.016 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: finishing write tid 2 to smithi16030472-14 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: finishing write tid 3 to smithi16030472-14 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: finishing write tid 4 to smithi16030472-14 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: oid 14 updating version 0 to 700 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: oid 14 version 700 is already newer than 698 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: oid 14 version 700 is already newer than 699 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 700 (ObjNum 984 snap 0 seq_num 984) dirty exists 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1947: left oid 14 (ObjNum 984 snap 0 seq_num 984) 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1954: got (0) Success 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 485 v 0 (ObjNum 257 snap 0 seq_num 3153698528) clean dne 2024-09-21T18:08:50.018 INFO:tasks.rados.rados.0.smithi160.stdout:1949: finishing copy_from to smithi16030472-434 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1949: got expected ENOENT (src dne) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1941: done (11 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1946: done (10 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1947: done (9 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1951: done (8 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1954: done (7 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1955: done (6 left) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1956: read oid 172 snap -1 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1956: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1957: copy_from oid 165 from oid 150 current snap is 0 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1958: read oid 351 snap -1 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1958: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1959: cache_evict oid 138 snap -1 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1960: write (excl) oid 222 current snap is 0 2024-09-21T18:08:50.019 INFO:tasks.rados.rados.0.smithi160.stdout:1960: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1960: writing smithi16030472-222 from 605316 to 1160865 tid 1 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1960: writing smithi16030472-222 from 1832327 to 2474340 tid 2 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1961: cache_flush oid 1 snap -1 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1959: got (0) Success 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1959: done (11 left) 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1962: cache_flush oid 290 snap -1 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1945: got (0) Success 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 98 v 0 (ObjNum 113601 snap 0 seq_num 2436187248) clean dne 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: finishing write tid 1 to smithi16030472-177 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: finishing write tid 2 to smithi16030472-177 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: finishing write tid 3 to smithi16030472-177 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: finishing write tid 4 to smithi16030472-177 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: oid 177 updating version 0 to 713 2024-09-21T18:08:50.020 INFO:tasks.rados.rados.0.smithi160.stdout:1948: oid 177 updating version 713 to 715 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1948: oid 177 version 715 is already newer than 714 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 177 v 715 (ObjNum 985 snap 0 seq_num 985) dirty exists 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1948: left oid 177 (ObjNum 985 snap 0 seq_num 985) 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1952: finishing write tid 1 to smithi16030472-101 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1952: finishing write tid 2 to smithi16030472-101 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1945: done (11 left) 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1948: done (10 left) 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1963: write (excl) oid 430 current snap is 0 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1963: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-21T18:08:50.021 INFO:tasks.rados.rados.0.smithi160.stdout:1963: writing smithi16030472-430 from 622123 to 1369274 tid 1 2024-09-21T18:08:50.021 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:50.012+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:50.022 INFO:tasks.rados.rados.0.smithi160.stdout:1963: writing smithi16030472-430 from 2007162 to 2622123 tid 2 2024-09-21T18:08:50.022 INFO:tasks.rados.rados.0.smithi160.stdout:1964: write (excl) oid 447 current snap is 0 2024-09-21T18:08:50.022 INFO:tasks.rados.rados.0.smithi160.stdout:1964: seq_num 990 ranges {638930=422400,1665644=645682} 2024-09-21T18:08:50.032 INFO:tasks.rados.rados.0.smithi160.stdout:1964: writing smithi16030472-447 from 638930 to 1061330 tid 1 2024-09-21T18:08:50.043 INFO:tasks.rados.rados.0.smithi160.stdout:1964: writing smithi16030472-447 from 1665644 to 2311326 tid 2 2024-09-21T18:08:50.043 INFO:tasks.rados.rados.0.smithi160.stdout:1965: cache_flush oid 22 snap -1 2024-09-21T18:08:50.043 INFO:tasks.rados.rados.0.smithi160.stdout:1966: cache_try_flush oid 79 snap -1 2024-09-21T18:08:50.043 INFO:tasks.rados.rados.0.smithi160.stdout:1967: write oid 457 current snap is 0 2024-09-21T18:08:50.043 INFO:tasks.rados.rados.0.smithi160.stdout:1967: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-21T18:08:50.051 INFO:tasks.rados.rados.0.smithi160.stdout:1967: writing smithi16030472-457 from 655737 to 1153386 tid 1 2024-09-21T18:08:50.059 INFO:tasks.rados.rados.0.smithi160.stdout:1967: writing smithi16030472-457 from 1607773 to 2397113 tid 2 2024-09-21T18:08:50.059 INFO:tasks.rados.rados.0.smithi160.stdout:1968: cache_flush oid 402 snap -1 2024-09-21T18:08:50.059 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout:1950: got (0) Success 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 0 (ObjNum 844 snap 0 seq_num 844) dirty exists 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout:1950: done (15 left) 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout:1969: read oid 178 snap -1 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout:1969: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:50.073 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:50.099 INFO:tasks.rados.rados.0.smithi160.stdout:1953: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:08:50.116 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:50.112+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:50.179 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:50.191 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:50.191 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:50.191 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:50.191 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.206 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:50.217 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:50.210+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:50.251 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1898348 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: finishing write tid 3 to smithi16030472-101 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: finishing write tid 4 to smithi16030472-101 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: finishing write tid 5 to smithi16030472-101 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: oid 101 updating version 0 to 716 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: oid 101 updating version 716 to 717 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: oid 101 updating version 717 to 718 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: oid 101 updating version 718 to 719 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 101 v 719 (ObjNum 986 snap 0 seq_num 986) dirty exists 2024-09-21T18:08:50.287 INFO:tasks.rados.rados.0.smithi160.stdout:1952: left oid 101 (ObjNum 986 snap 0 seq_num 986) 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1949: finishing copy_from racing read to smithi16030472-434 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: finishing write tid 1 to smithi16030472-430 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1965: got (0) Success 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 22 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: finishing write tid 2 to smithi16030472-430 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: finishing write tid 3 to smithi16030472-430 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: finishing write tid 4 to smithi16030472-430 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: oid 430 updating version 0 to 642 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: oid 430 updating version 642 to 644 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: oid 430 version 644 is already newer than 643 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 430 v 644 (ObjNum 989 snap 0 seq_num 989) dirty exists 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1963: left oid 430 (ObjNum 989 snap 0 seq_num 989) 2024-09-21T18:08:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:1968: got (0) Success 2024-09-21T18:08:50.289 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 402 v 0 (ObjNum 401 snap 0 seq_num 401) dirty exists 2024-09-21T18:08:50.289 INFO:tasks.rados.rados.0.smithi160.stdout:1967: finishing write tid 1 to smithi16030472-457 2024-09-21T18:08:50.289 INFO:tasks.rados.rados.0.smithi160.stdout:1969: expect (ObjNum 177 snap 0 seq_num 177) 2024-09-21T18:08:50.294 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1360154 2024-09-21T18:08:50.318 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:50.313+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1961: got (0) Success 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 1 v 0 (ObjNum 49 snap 0 seq_num 3111024432) clean dne 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1967: finishing write tid 2 to smithi16030472-457 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1967: finishing write tid 3 to smithi16030472-457 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1949: done (15 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1952: done (14 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1953: done (13 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1961: done (12 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1963: done (11 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1965: done (10 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1968: done (9 left) 2024-09-21T18:08:50.365 INFO:tasks.rados.rados.0.smithi160.stdout:1969: done (8 left) 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1970: cache_try_flush oid 380 snap -1 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1971: read oid 82 snap -1 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1971: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1972: cache_flush oid 320 snap -1 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1973: cache_flush oid 368 snap -1 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1974: cache_try_flush oid 164 snap -1 2024-09-21T18:08:50.366 INFO:tasks.rados.rados.0.smithi160.stdout:1975: delete oid 199 current snap is 0 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:1967: finishing write tid 4 to smithi16030472-457 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:1967: oid 457 updating version 0 to 720 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:1967: oid 457 updating version 720 to 721 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:1967: oid 457 updating version 721 to 722 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 457 v 722 (ObjNum 991 snap 0 seq_num 991) dirty exists 2024-09-21T18:08:50.367 INFO:tasks.rados.rados.0.smithi160.stdout:1967: left oid 457 (ObjNum 991 snap 0 seq_num 991) 2024-09-21T18:08:50.376 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:50.376 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:50.376 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:50.376 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:50.376 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.391 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:50.419 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:50.413+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:50.520 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:50.518+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.486960151999995, 2024-09-21T18:08:50.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:08:50.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:08:50.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:08:50.525 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.483349601, 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:08:50.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.480543061999995, 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.530 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:08:50.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:50.534 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 79.476794862000006, 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:08:50.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.855944058000006, 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:08:50.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:08:50.543 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.755089986000002, 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:08:50.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:08:50.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.754333705999997, 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:08:50.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:08:50.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:08:50.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.753942148999997, 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:08:50.556 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.753641024999993, 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:08:50.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:50.561 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.752917174999993, 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:08:50.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:08:50.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:08:50.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.652020336000007, 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:08:50.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:08:50.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.651926052999997, 2024-09-21T18:08:50.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.651012858000001, 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:08:50.574 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:08:50.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.578 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.650443170000003, 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:50.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:08:50.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:08:50.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:08:50.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.649715685999993, 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.583 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:08:50.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:08:50.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 78.583533777, 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:08:50.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.327487972, 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:08:50.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:50.596 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.327205173999999, 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:08:50.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.327126188999998, 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:08:50.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 77.324507858000004, 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:50.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:50.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:50.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:50.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:50.610 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.611 INFO:tasks.rados.rados.0.smithi160.stdout:1956: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:50.611 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:50.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:50.620 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:50.620+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:50.644 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1960: finishing write tid 1 to smithi16030472-222 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1962: got (2) No such file or directory 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1964: finishing write tid 1 to smithi16030472-447 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1964: finishing write tid 2 to smithi16030472-447 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1957: finishing copy_from to smithi16030472-165 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1957: got expected ENOENT (src dne) 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1957: finishing copy_from racing read to smithi16030472-165 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1964: finishing write tid 3 to smithi16030472-447 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1970: got (2) No such file or directory 2024-09-21T18:08:50.706 INFO:tasks.rados.rados.0.smithi160.stdout:1971: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:08:50.721 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:50.716+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:50.741 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:50.741 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:50.741 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:50.741 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.756 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:50.836 INFO:tasks.rados.rados.0.smithi160.stdout:1960: finishing write tid 2 to smithi16030472-222 2024-09-21T18:08:50.836 INFO:tasks.rados.rados.0.smithi160.stdout:1960: finishing write tid 3 to smithi16030472-222 2024-09-21T18:08:50.836 INFO:tasks.rados.rados.0.smithi160.stdout:1956: done (13 left) 2024-09-21T18:08:50.836 INFO:tasks.rados.rados.0.smithi160.stdout:1957: done (12 left) 2024-09-21T18:08:50.836 INFO:tasks.rados.rados.0.smithi160.stdout:1962: done (11 left) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1967: done (10 left) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1970: done (9 left) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1971: done (8 left) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1975: done (7 left) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1976: write (excl) oid 416 current snap is 0 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1972: got (2) No such file or directory 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1960: finishing write tid 4 to smithi16030472-222 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1960: oid 222 updating version 0 to 676 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1960: oid 222 updating version 676 to 703 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1960: oid 222 version 703 is already newer than 702 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 222 v 703 (ObjNum 988 snap 0 seq_num 988) dirty exists 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1960: left oid 222 (ObjNum 988 snap 0 seq_num 988) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:1958: expect (ObjNum 350 snap 0 seq_num 350) 2024-09-21T18:08:50.837 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 126754 2024-09-21T18:08:50.843 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:50.840+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:50.922 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:50.918+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:50.929 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:50.929 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:50.929 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:50.929 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:50.930 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:50.942 INFO:tasks.rados.rados.0.smithi160.stdout:1964: finishing write tid 4 to smithi16030472-447 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:1964: oid 447 updating version 0 to 624 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:1964: oid 447 version 624 is already newer than 623 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:1964: oid 447 updating version 624 to 625 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 447 v 625 (ObjNum 990 snap 0 seq_num 990) dirty exists 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:1964: left oid 447 (ObjNum 990 snap 0 seq_num 990) 2024-09-21T18:08:50.943 INFO:tasks.rados.rados.0.smithi160.stdout:1976: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-21T18:08:50.944 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:50.950 INFO:tasks.rados.rados.0.smithi160.stdout:1976: writing smithi16030472-416 from 689351 to 1337498 tid 1 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1976: writing smithi16030472-416 from 2124737 to 2689351 tid 2 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1958: done (7 left) 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1960: done (6 left) 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1964: done (5 left) 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1972: done (4 left) 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1977: write oid 484 current snap is 0 2024-09-21T18:08:50.958 INFO:tasks.rados.rados.0.smithi160.stdout:1977: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-21T18:08:50.965 INFO:tasks.rados.rados.0.smithi160.stdout:1977: writing smithi16030472-484 from 706158 to 1429554 tid 1 2024-09-21T18:08:50.973 INFO:tasks.rados.rados.0.smithi160.stdout:1977: writing smithi16030472-484 from 2183219 to 2706158 tid 2 2024-09-21T18:08:50.973 INFO:tasks.rados.rados.0.smithi160.stdout:1978: cache_evict oid 84 snap -1 2024-09-21T18:08:50.973 INFO:tasks.rados.rados.0.smithi160.stdout:1979: write oid 176 current snap is 0 2024-09-21T18:08:50.973 INFO:tasks.rados.rados.0.smithi160.stdout:1979: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-21T18:08:50.984 INFO:tasks.rados.rados.0.smithi160.stdout:1979: writing smithi16030472-176 from 722965 to 1521610 tid 1 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1979: writing smithi16030472-176 from 2125348 to 2722965 tid 2 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1980: copy_from oid 46 from oid 200 current snap is 0 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1978: got (16) Device or resource busy 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1978: done (7 left) 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1981: copy_from oid 366 from oid 42 current snap is 0 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1982: copy_from oid 150 from oid 83 current snap is 0 2024-09-21T18:08:50.993 INFO:tasks.rados.rados.0.smithi160.stdout:1983: delete oid 86 current snap is 0 2024-09-21T18:08:50.997 INFO:tasks.rados.rados.0.smithi160.stdout:1966: got (0) Success 2024-09-21T18:08:50.997 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 0 (ObjNum 785 snap 0 seq_num 785) dirty exists 2024-09-21T18:08:51.023 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:51.021+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:51.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:08:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:1979: finishing write tid 1 to smithi16030472-176 2024-09-21T18:08:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:1979: finishing write tid 2 to smithi16030472-176 2024-09-21T18:08:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:1973: got (0) Success 2024-09-21T18:08:51.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 368 v 0 (ObjNum 934 snap 0 seq_num 934) dirty exists 2024-09-21T18:08:51.113 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:51.113 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:08:51.113 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:51.124 INFO:tasks.rados.rados.0.smithi160.stdout:1980: finishing copy_from racing read to smithi16030472-46 2024-09-21T18:08:51.124 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:51.120+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:51.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:51.210 INFO:tasks.rados.rados.0.smithi160.stdout:1977: finishing write tid 1 to smithi16030472-484 2024-09-21T18:08:51.216 INFO:tasks.rados.rados.0.smithi160.stdout:1976: finishing write tid 1 to smithi16030472-416 2024-09-21T18:08:51.216 INFO:tasks.rados.rados.0.smithi160.stdout:1976: finishing write tid 2 to smithi16030472-416 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: finishing write tid 3 to smithi16030472-416 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: finishing write tid 4 to smithi16030472-416 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: oid 416 updating version 0 to 647 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: oid 416 version 647 is already newer than 646 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: oid 416 version 647 is already newer than 645 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 647 (ObjNum 993 snap 0 seq_num 993) dirty exists 2024-09-21T18:08:51.217 INFO:tasks.rados.rados.0.smithi160.stdout:1976: left oid 416 (ObjNum 993 snap 0 seq_num 993) 2024-09-21T18:08:51.221 INFO:tasks.rados.rados.0.smithi160.stdout:1966: done (9 left) 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1973: done (8 left) 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1976: done (7 left) 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1983: done (6 left) 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1984: copy_from oid 246 from oid 342 current snap is 0 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1985: cache_flush oid 226 snap -1 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1986: copy_from oid 103 from oid 457 current snap is 0 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1987: write oid 31 current snap is 0 2024-09-21T18:08:51.222 INFO:tasks.rados.rados.0.smithi160.stdout:1987: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-21T18:08:51.225 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:51.224+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:51.231 INFO:tasks.rados.rados.0.smithi160.stdout:1987: writing smithi16030472-31 from 756579 to 1422075 tid 1 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1987: writing smithi16030472-31 from 1958665 to 2359012 tid 2 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1988: read oid 411 snap -1 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1988: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1989: cache_flush oid 60 snap -1 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1985: got (0) Success 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 226 v 0 (ObjNum 225 snap 0 seq_num 225) dirty exists 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1980: finishing copy_from to smithi16030472-46 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 151 (ObjNum 199 snap 0 seq_num 199) dirty exists 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1980: done (11 left) 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1985: done (10 left) 2024-09-21T18:08:51.238 INFO:tasks.rados.rados.0.smithi160.stdout:1990: delete oid 269 current snap is 0 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.11.22:2133 3.3 e86/31)", 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:51.204349+0000", 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.057405019000000002, 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.057423167999999997, 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.262 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "no flag points reached", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:51.204349+0000", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:51.204349+0000", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:51.204363+0000", 2024-09-21T18:08:51.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3745e-05 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:51.207921+0000", 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035574790000000001 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:51.207949+0000", 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8311999999999999e-05 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:08:51.264 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:08:51.265 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:51.276 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:51.311 INFO:tasks.rados.rados.0.smithi160.stdout:1977: finishing write tid 2 to smithi16030472-484 2024-09-21T18:08:51.327 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:51.322+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:51.411 INFO:tasks.rados.rados.0.smithi160.stdout:1981: finishing copy_from racing read to smithi16030472-366 2024-09-21T18:08:51.428 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:51.424+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:51.444 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:51.444 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:51.444 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:51.444 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:51.444 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:51.460 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:51.528 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:51.521+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:51.547 INFO:tasks.rados.rados.0.smithi160.stdout:1987: finishing write tid 1 to smithi16030472-31 2024-09-21T18:08:51.582 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:51.582 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:08:51.582 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:08:51.582 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:08:51.582 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 93.249737750999998, 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:08:51.583 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:08:51.584 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.585 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:08:51.586 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.409885755000005, 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.587 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.588 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:08:51.589 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:08:51.590 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.306165307000001, 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.591 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:08:51.593 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:08:51.594 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.595 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.305390564000007, 2024-09-21T18:08:51.596 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.597 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.598 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.599 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.304742544999996, 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:08:51.600 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.601 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.602 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.603 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:51.604 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.290870850000005, 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:08:51.605 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:08:51.606 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:08:51.607 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.608 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.287929833999996, 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.609 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:08:51.610 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:08:51.611 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.612 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.284912985999995, 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.613 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.614 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.615 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.616 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.282701403999994, 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.617 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.618 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.619 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:08:51.620 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.177364311999995, 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:51.621 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:08:51.622 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.623 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:08:51.624 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 86.485941937999996, 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.625 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.626 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.627 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.628 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 86.482911938000001, 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.629 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.630 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.631 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:08:51.632 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 81.516347976000006, 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.633 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:08:51.634 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:08:51.635 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:08:51.636 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 79.849512540999996, 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:08:51.637 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.638 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.639 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:08:51.640 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.641 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.546021460000006, 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.642 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.643 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.644 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.645 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.543724427000001, 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.646 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:08:51.647 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:08:51.648 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:08:51.649 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.541673978999995, 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.650 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.651 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.652 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:08:51.653 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.539518618000002, 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.654 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.655 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.656 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:08:51.657 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.538755859999995, 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.658 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:08:51.659 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.660 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.661 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.436850815, 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.662 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.663 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.664 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.665 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.666 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:08:51.666 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:08:51.666 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:51.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:51.668 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:51.662+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:51.669 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:51.669 INFO:tasks.rados.rados.0.smithi160.stdout:1974: got (0) Success 2024-09-21T18:08:51.669 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 0 (ObjNum 801 snap 0 seq_num 801) dirty exists 2024-09-21T18:08:51.669 INFO:tasks.rados.rados.0.smithi160.stdout:1986: finishing copy_from racing read to smithi16030472-103 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: finishing write tid 3 to smithi16030472-176 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: finishing write tid 4 to smithi16030472-176 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: oid 176 updating version 0 to 677 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: oid 176 updating version 677 to 704 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: oid 176 version 704 is already newer than 678 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 176 v 704 (ObjNum 995 snap 0 seq_num 995) dirty exists 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1979: left oid 176 (ObjNum 995 snap 0 seq_num 995) 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1986: finishing copy_from to smithi16030472-103 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 722 (ObjNum 991 snap 0 seq_num 991) dirty exists 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1977: finishing write tid 3 to smithi16030472-484 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1977: finishing write tid 4 to smithi16030472-484 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1977: oid 484 updating version 0 to 724 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1977: oid 484 updating version 724 to 726 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:1977: oid 484 version 726 is already newer than 725 2024-09-21T18:08:51.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 726 (ObjNum 994 snap 0 seq_num 994) dirty exists 2024-09-21T18:08:51.671 INFO:tasks.rados.rados.0.smithi160.stdout:1977: left oid 484 (ObjNum 994 snap 0 seq_num 994) 2024-09-21T18:08:51.671 INFO:tasks.rados.rados.0.smithi160.stdout:1981: finishing copy_from to smithi16030472-366 2024-09-21T18:08:51.671 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 348 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:08:51.671 INFO:tasks.rados.rados.0.smithi160.stdout:1987: finishing write tid 2 to smithi16030472-31 2024-09-21T18:08:51.734 INFO:tasks.rados.rados.0.smithi160.stdout:1984: finishing copy_from to smithi16030472-246 2024-09-21T18:08:51.734 INFO:tasks.rados.rados.0.smithi160.stdout:1984: got expected ENOENT (src dne) 2024-09-21T18:08:51.735 INFO:tasks.rados.rados.0.smithi160.stdout:1982: finishing copy_from racing read to smithi16030472-150 2024-09-21T18:08:51.735 INFO:tasks.rados.rados.0.smithi160.stdout:1982: finishing copy_from to smithi16030472-150 2024-09-21T18:08:51.735 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 150 v 397 (ObjNum 631 snap 0 seq_num 631) dirty exists 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:1987: finishing write tid 3 to smithi16030472-31 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:1987: finishing write tid 4 to smithi16030472-31 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:1987: oid 31 updating version 0 to 716 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:1987: oid 31 version 716 is already newer than 714 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:1987: oid 31 version 716 is already newer than 715 2024-09-21T18:08:51.748 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 716 (ObjNum 997 snap 0 seq_num 997) dirty exists 2024-09-21T18:08:51.749 INFO:tasks.rados.rados.0.smithi160.stdout:1987: left oid 31 (ObjNum 997 snap 0 seq_num 997) 2024-09-21T18:08:51.768 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:51.761+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:51.802 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:51.802 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:51.802 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:08:51.802 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:51.815 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:51.824 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:51.824 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:51.832 INFO:tasks.rados.rados.0.smithi160.stdout:1984: finishing copy_from racing read to smithi16030472-246 2024-09-21T18:08:51.844 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:51.844 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:51.848 INFO:tasks.rados.rados.0.smithi160.stdout:1974: done (10 left) 2024-09-21T18:08:51.848 INFO:tasks.rados.rados.0.smithi160.stdout:1977: done (9 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1979: done (8 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1981: done (7 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1982: done (6 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1984: done (5 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1986: done (4 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1987: done (3 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1990: done (2 left) 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1991: cache_evict oid 2 snap -1 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1992: cache_try_flush oid 370 snap -1 2024-09-21T18:08:51.849 INFO:tasks.rados.rados.0.smithi160.stdout:1993: cache_try_flush oid 350 snap -1 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1994: read oid 184 snap -1 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1994: expect (ObjNum 532 snap 0 seq_num 532) 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1995: read oid 59 snap -1 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1995: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1996: cache_flush oid 25 snap -1 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1997: delete oid 112 current snap is 0 2024-09-21T18:08:51.850 INFO:tasks.rados.rados.0.smithi160.stdout:1991: got (2) No such file or directory 2024-09-21T18:08:51.851 INFO:tasks.rados.rados.0.smithi160.stdout:1993: got (0) Success 2024-09-21T18:08:51.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 0 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:08:51.861 INFO:tasks.rados.rados.0.smithi160.stdout:1995: expect (ObjNum 171 snap 0 seq_num 171) 2024-09-21T18:08:51.862 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:51.862 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:51.869 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:51.862+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:51.884 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:51.884 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:51.937 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:51.940 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:08:51.940 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:08:51.940 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:08:51.940 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:08:51.941 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:08:51.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:51.970 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:51.962+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:51.996 INFO:tasks.rados.rados.0.smithi160.stdout:1989: got (0) Success 2024-09-21T18:08:51.996 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 0 (ObjNum 904 snap 0 seq_num 904) dirty exists 2024-09-21T18:08:51.996 INFO:tasks.rados.rados.0.smithi160.stdout:1988: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-21T18:08:52.010 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:52.026 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.038 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:08:52.071 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:52.068+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:52.071 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:52.082 INFO:tasks.rados.rados.0.smithi160.stdout:1994: expect (ObjNum 532 snap 0 seq_num 532) 2024-09-21T18:08:52.082 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.084 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.085 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.098 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:52.099 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:52.122 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:52.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:52.123 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.146 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:52.170 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:52.171 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:52.166+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:52.189 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:52.189 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:1992: got (0) Success 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 370 v 0 (ObjNum 731 snap 0 seq_num 731) dirty exists 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:1996: got (0) Success 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 25 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:1988: done (8 left) 2024-09-21T18:08:52.195 INFO:tasks.rados.rados.0.smithi160.stdout:1989: done (7 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1991: done (6 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1992: done (5 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1993: done (4 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1994: done (3 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1995: done (2 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1996: done (1 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1997: done (0 left) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1998: cache_evict oid 303 snap -1 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:1999: cache_evict oid 302 snap -1 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2000: cache_evict oid 145 snap -1 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2001: read oid 249 snap -1 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2001: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2002: copy_from oid 241 from oid 283 current snap is 0 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2003: read oid 358 snap -1 2024-09-21T18:08:52.196 INFO:tasks.rados.rados.0.smithi160.stdout:2003: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-21T18:08:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:2004: cache_evict oid 323 snap -1 2024-09-21T18:08:52.197 INFO:tasks.rados.rados.0.smithi160.stdout:2005: delete oid 210 current snap is 0 2024-09-21T18:08:52.211 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:52.228 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:52.264 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.265 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.271 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:52.268+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:52.278 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:52.295 INFO:tasks.rados.rados.0.smithi160.stdout:1999: got (0) Success 2024-09-21T18:08:52.295 INFO:tasks.rados.rados.0.smithi160.stdout:2000: got (0) Success 2024-09-21T18:08:52.295 INFO:tasks.rados.rados.0.smithi160.stdout:2002: finishing copy_from to smithi16030472-241 2024-09-21T18:08:52.295 INFO:tasks.rados.rados.0.smithi160.stdout:2002: got expected ENOENT (src dne) 2024-09-21T18:08:52.296 INFO:tasks.rados.rados.0.smithi160.stdout:1998: got (16) Device or resource busy 2024-09-21T18:08:52.297 INFO:tasks.rados.rados.0.smithi160.stdout:2002: finishing copy_from racing read to smithi16030472-241 2024-09-21T18:08:52.300 INFO:tasks.rados.rados.0.smithi160.stdout:2003: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-21T18:08:52.316 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.321 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:52.326 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:52.337 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.358 INFO:tasks.rados.rados.0.smithi160.stdout:2004: got (2) No such file or directory 2024-09-21T18:08:52.359 INFO:tasks.rados.rados.0.smithi160.stdout:2001: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:08:52.372 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:52.366+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:52.399 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.399 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.415 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:52.471 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.473 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:52.466+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:1998: done (7 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:1999: done (6 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2000: done (5 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2001: done (4 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2002: done (3 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2003: done (2 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2004: done (1 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2005: done (0 left) 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2006: cache_try_flush oid 225 snap -1 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2007: read oid 481 snap -1 2024-09-21T18:08:52.531 INFO:tasks.rados.rados.0.smithi160.stdout:2007: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:52.532 INFO:tasks.rados.rados.0.smithi160.stdout:2008: cache_try_flush oid 199 snap -1 2024-09-21T18:08:52.532 INFO:tasks.rados.rados.0.smithi160.stdout:2009: read oid 389 snap -1 2024-09-21T18:08:52.532 INFO:tasks.rados.rados.0.smithi160.stdout:2009: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-21T18:08:52.532 INFO:tasks.rados.rados.0.smithi160.stdout:2010: delete oid 368 current snap is 0 2024-09-21T18:08:52.537 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.538 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.553 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:52.574 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:52.566+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:52.634 INFO:tasks.rados.rados.0.smithi160.stdout:2007: expect (ObjNum 480 snap 0 seq_num 480) 2024-09-21T18:08:52.673 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.673 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.675 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:52.670+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:52.687 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:08:52.737 INFO:tasks.rados.rados.0.smithi160.stdout:2009: expect (ObjNum 388 snap 0 seq_num 388) 2024-09-21T18:08:52.775 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:52.772+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:52.779 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2008: got (0) Success 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 0 (ObjNum 81 snap 0 seq_num 2436185376) clean dne 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2007: done (4 left) 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2008: done (3 left) 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2009: done (2 left) 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2010: done (1 left) 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2011: read oid 425 snap -1 2024-09-21T18:08:52.806 INFO:tasks.rados.rados.0.smithi160.stdout:2011: expect deleted 2024-09-21T18:08:52.807 INFO:tasks.rados.rados.0.smithi160.stdout:2012: write oid 42 current snap is 0 2024-09-21T18:08:52.807 INFO:tasks.rados.rados.0.smithi160.stdout:2012: seq_num 1002 ranges {440614=641741,1734722=551343} 2024-09-21T18:08:52.808 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.808 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:08:52.823 INFO:tasks.rados.rados.0.smithi160.stdout:2012: writing smithi16030472-42 from 440614 to 1082355 tid 1 2024-09-21T18:08:52.843 INFO:tasks.rados.rados.0.smithi160.stdout:2012: writing smithi16030472-42 from 1734722 to 2286065 tid 2 2024-09-21T18:08:52.843 INFO:tasks.rados.rados.0.smithi160.stdout:2013: write oid 313 current snap is 0 2024-09-21T18:08:52.843 INFO:tasks.rados.rados.0.smithi160.stdout:2006: got (0) Success 2024-09-21T18:08:52.844 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 0 (ObjNum 913 snap 0 seq_num 913) dirty exists 2024-09-21T18:08:52.844 INFO:tasks.rados.rados.0.smithi160.stdout:2013: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-21T18:08:52.862 INFO:tasks.rados.rados.0.smithi160.stdout:2013: writing smithi16030472-313 from 457421 to 1174411 tid 1 2024-09-21T18:08:52.876 INFO:tasks.rados.rados.0.smithi160.stdout:2013: writing smithi16030472-313 from 1676851 to 2088205 tid 2 2024-09-21T18:08:52.885 INFO:tasks.rados.rados.0.smithi160.stdout:2013: writing smithi16030472-313 from 2677497 to 2857421 tid 3 2024-09-21T18:08:52.885 INFO:tasks.rados.rados.0.smithi160.stdout:2006: done (3 left) 2024-09-21T18:08:52.885 INFO:tasks.rados.rados.0.smithi160.stdout:2014: write oid 439 current snap is 0 2024-09-21T18:08:52.885 INFO:tasks.rados.rados.0.smithi160.stdout:2014: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-21T18:08:52.895 INFO:tasks.rados.rados.0.smithi160.stdout:2014: writing smithi16030472-439 from 474228 to 982820 tid 1 2024-09-21T18:08:52.904 INFO:tasks.rados.rados.0.smithi160.stdout:2014: writing smithi16030472-439 from 1451686 to 2006698 tid 2 2024-09-21T18:08:52.910 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:52.908+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:52.911 INFO:tasks.rados.rados.0.smithi160.stdout:2014: writing smithi16030472-439 from 2704920 to 2874228 tid 3 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2015: cache_flush oid 40 snap -1 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2016: read oid 366 snap -1 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2016: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2017: cache_flush oid 199 snap -1 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2018: cache_try_flush oid 246 snap -1 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2019: cache_flush oid 63 snap -1 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2020: write oid 413 current snap is 0 2024-09-21T18:08:52.912 INFO:tasks.rados.rados.0.smithi160.stdout:2020: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-21T18:08:52.919 INFO:tasks.rados.rados.0.smithi160.stdout:2020: writing smithi16030472-413 from 491035 to 1074876 tid 1 2024-09-21T18:08:52.927 INFO:tasks.rados.rados.0.smithi160.stdout:2020: writing smithi16030472-413 from 1510168 to 2208838 tid 2 2024-09-21T18:08:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:2020: writing smithi16030472-413 from 2732343 to 2891035 tid 3 2024-09-21T18:08:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:2021: read oid 290 snap -1 2024-09-21T18:08:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:2021: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-21T18:08:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:2022: read oid 62 snap -1 2024-09-21T18:08:52.931 INFO:tasks.rados.rados.0.smithi160.stdout:2022: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout:2023: read oid 137 snap -1 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout:2023: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout:2024: cache_flush oid 274 snap -1 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout:2025: cache_try_flush oid 80 snap -1 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout:2026: cache_evict oid 87 snap -1 2024-09-21T18:08:52.932 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:52.945 INFO:tasks.rados.rados.0.smithi160.stdout:2012: finishing write tid 1 to smithi16030472-42 2024-09-21T18:08:52.945 INFO:tasks.rados.rados.0.smithi160.stdout:2012: finishing write tid 2 to smithi16030472-42 2024-09-21T18:08:52.945 INFO:tasks.rados.rados.0.smithi160.stdout:2012: finishing write tid 3 to smithi16030472-42 2024-09-21T18:08:52.945 INFO:tasks.rados.rados.0.smithi160.stdout:2012: finishing write tid 4 to smithi16030472-42 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2012: oid 42 updating version 0 to 717 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2012: oid 42 updating version 717 to 718 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2012: oid 42 updating version 718 to 719 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 719 (ObjNum 1002 snap 0 seq_num 1002) dirty exists 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2012: left oid 42 (ObjNum 1002 snap 0 seq_num 1002) 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2012: done (15 left) 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout:2027: cache_evict oid 330 snap -1 2024-09-21T18:08:52.946 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:52.950 INFO:tasks.rados.rados.0.smithi160.stdout:2026: got (16) Device or resource busy 2024-09-21T18:08:52.951 INFO:tasks.rados.rados.0.smithi160.stdout:2026: done (15 left) 2024-09-21T18:08:52.951 INFO:tasks.rados.rados.0.smithi160.stdout:2028: delete oid 394 current snap is 0 2024-09-21T18:08:52.977 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:52.970+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:52.985 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:52.985 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:52.997 INFO:tasks.rados.rados.0.smithi160.stdout:2016: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:08:52.999 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:08:53.067 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:53.078 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:53.072+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:53.104 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 104666 2024-09-21T18:08:53.130 INFO:tasks.rados.rados.0.smithi160.stdout:2017: got (0) Success 2024-09-21T18:08:53.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 0 (ObjNum 81 snap 0 seq_num 3131249056) clean dne 2024-09-21T18:08:53.130 INFO:tasks.rados.rados.0.smithi160.stdout:2018: got (0) Success 2024-09-21T18:08:53.131 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 246 v 0 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-09-21T18:08:53.131 INFO:tasks.rados.rados.0.smithi160.stdout:2022: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:08:53.131 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:53.133 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.133 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.147 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:08:53.179 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:53.176+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:53.267 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.267 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.280 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:53.273+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:53.281 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:08:53.294 INFO:tasks.rados.rados.0.smithi160.stdout:2024: got (2) No such file or directory 2024-09-21T18:08:53.294 INFO:tasks.rados.rados.0.smithi160.stdout:2013: finishing write tid 1 to smithi16030472-313 2024-09-21T18:08:53.294 INFO:tasks.rados.rados.0.smithi160.stdout:2027: got (16) Device or resource busy 2024-09-21T18:08:53.294 INFO:tasks.rados.rados.0.smithi160.stdout:2016: done (15 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2017: done (14 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2018: done (13 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2022: done (12 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2024: done (11 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2027: done (10 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2028: done (9 left) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2029: read oid 37 snap -1 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2029: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2030: delete oid 125 current snap is 0 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: finishing write tid 2 to smithi16030472-313 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: finishing write tid 3 to smithi16030472-313 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: finishing write tid 4 to smithi16030472-313 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: finishing write tid 5 to smithi16030472-313 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: oid 313 updating version 0 to 730 2024-09-21T18:08:53.295 INFO:tasks.rados.rados.0.smithi160.stdout:2013: oid 313 updating version 730 to 732 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:2013: oid 313 version 732 is already newer than 729 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:2013: oid 313 version 732 is already newer than 731 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 313 v 732 (ObjNum 1003 snap 0 seq_num 1003) dirty exists 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:2013: left oid 313 (ObjNum 1003 snap 0 seq_num 1003) 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:2015: got (0) Success 2024-09-21T18:08:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 0 (ObjNum 841 snap 0 seq_num 841) dirty exists 2024-09-21T18:08:53.381 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:53.376+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:53.420 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.420 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.436 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:08:53.482 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:53.474+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:53.563 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.563 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.577 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:08:53.583 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:53.578+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2019: got (2) No such file or directory 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2020: finishing write tid 1 to smithi16030472-413 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: finishing write tid 1 to smithi16030472-439 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2020: finishing write tid 2 to smithi16030472-413 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2020: finishing write tid 3 to smithi16030472-413 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2020: finishing write tid 4 to smithi16030472-413 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: finishing write tid 2 to smithi16030472-439 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: finishing write tid 3 to smithi16030472-439 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: finishing write tid 4 to smithi16030472-439 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: finishing write tid 5 to smithi16030472-439 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: oid 439 updating version 0 to 680 2024-09-21T18:08:53.615 INFO:tasks.rados.rados.0.smithi160.stdout:2014: oid 439 updating version 680 to 682 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2014: oid 439 version 682 is already newer than 681 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2014: oid 439 updating version 682 to 683 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 439 v 683 (ObjNum 1004 snap 0 seq_num 1004) dirty exists 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2014: left oid 439 (ObjNum 1004 snap 0 seq_num 1004) 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: finishing write tid 5 to smithi16030472-413 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: oid 413 updating version 0 to 707 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: oid 413 updating version 707 to 708 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: oid 413 version 708 is already newer than 706 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: oid 413 version 708 is already newer than 705 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 708 (ObjNum 1005 snap 0 seq_num 1005) dirty exists 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2020: left oid 413 (ObjNum 1005 snap 0 seq_num 1005) 2024-09-21T18:08:53.616 INFO:tasks.rados.rados.0.smithi160.stdout:2023: expect (ObjNum 505 snap 0 seq_num 505) 2024-09-21T18:08:53.684 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:53.682+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:53.702 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.703 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:08:53.723 INFO:tasks.rados.rados.0.smithi160.stdout:2011: done (10 left) 2024-09-21T18:08:53.723 INFO:tasks.rados.rados.0.smithi160.stdout:2013: done (9 left) 2024-09-21T18:08:53.723 INFO:tasks.rados.rados.0.smithi160.stdout:2014: done (8 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2015: done (7 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2019: done (6 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2020: done (5 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2023: done (4 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2030: done (3 left) 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2031: copy_from oid 110 from oid 499 current snap is 0 2024-09-21T18:08:53.724 INFO:tasks.rados.rados.0.smithi160.stdout:2021: expect (ObjNum 289 snap 0 seq_num 289) 2024-09-21T18:08:53.776 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1382161 2024-09-21T18:08:53.785 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:53.780+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:53.796 INFO:tasks.rados.rados.0.smithi160.stdout:2029: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:08:53.804 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2024764 2024-09-21T18:08:53.813 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:53.840 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:53.851 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.851 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.866 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:08:53.868 INFO:tasks.rados.rados.0.smithi160.stdout:2025: got (0) Success 2024-09-21T18:08:53.868 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 0 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:08:53.869 INFO:tasks.rados.rados.0.smithi160.stdout:2021: done (3 left) 2024-09-21T18:08:53.869 INFO:tasks.rados.rados.0.smithi160.stdout:2025: done (2 left) 2024-09-21T18:08:53.869 INFO:tasks.rados.rados.0.smithi160.stdout:2029: done (1 left) 2024-09-21T18:08:53.869 INFO:tasks.rados.rados.0.smithi160.stdout:2032: write (excl) oid 351 current snap is 0 2024-09-21T18:08:53.869 INFO:tasks.rados.rados.0.smithi160.stdout:2032: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-21T18:08:53.873 INFO:tasks.rados.rados.0.smithi160.stdout:2032: writing smithi16030472-351 from 541456 to 951044 tid 1 2024-09-21T18:08:53.879 INFO:tasks.rados.rados.0.smithi160.stdout:2032: writing smithi16030472-351 from 1569261 to 2131611 tid 2 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2032: writing smithi16030472-351 from 2814612 to 2941456 tid 3 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2033: cache_evict oid 267 snap -1 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2034: read oid 323 snap -1 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2034: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2035: read oid 342 snap -1 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2035: expect deleted 2024-09-21T18:08:53.885 INFO:tasks.rados.rados.0.smithi160.stdout:2036: copy_from oid 206 from oid 49 current snap is 0 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2037: copy_from oid 306 from oid 135 current snap is 0 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2038: cache_try_flush oid 126 snap -1 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2039: cache_flush oid 497 snap -1 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2040: read oid 82 snap -1 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2040: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2041: write (excl) oid 201 current snap is 0 2024-09-21T18:08:53.886 INFO:tasks.rados.rados.0.smithi160.stdout:2041: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-21T18:08:53.886 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:53.878+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:53.890 INFO:tasks.rados.rados.0.smithi160.stdout:2041: writing smithi16030472-201 from 558263 to 1043100 tid 1 2024-09-21T18:08:53.898 INFO:tasks.rados.rados.0.smithi160.stdout:2041: writing smithi16030472-201 from 1627743 to 2333751 tid 2 2024-09-21T18:08:53.898 INFO:tasks.rados.rados.0.smithi160.stdout:2042: write (excl) oid 156 current snap is 0 2024-09-21T18:08:53.898 INFO:tasks.rados.rados.0.smithi160.stdout:2042: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2024-09-21T18:08:53.904 INFO:tasks.rados.rados.0.smithi160.stdout:2042: writing smithi16030472-156 from 575070 to 1135156 tid 1 2024-09-21T18:08:53.912 INFO:tasks.rados.rados.0.smithi160.stdout:2042: writing smithi16030472-156 from 1686225 to 2252244 tid 2 2024-09-21T18:08:53.918 INFO:tasks.rados.rados.0.smithi160.stdout:2042: writing smithi16030472-156 from 2869458 to 2975070 tid 3 2024-09-21T18:08:53.918 INFO:tasks.rados.rados.0.smithi160.stdout:2043: write (excl) oid 178 current snap is 0 2024-09-21T18:08:53.918 INFO:tasks.rados.rados.0.smithi160.stdout:2043: seq_num 1011 ranges {591877=635335,1628354=709677} 2024-09-21T18:08:53.926 INFO:tasks.rados.rados.0.smithi160.stdout:2043: writing smithi16030472-178 from 591877 to 1227212 tid 1 2024-09-21T18:08:53.932 INFO:tasks.rados.rados.0.smithi160.stdout:2043: writing smithi16030472-178 from 1628354 to 2338031 tid 2 2024-09-21T18:08:53.933 INFO:tasks.rados.rados.0.smithi160.stdout:2044: cache_flush oid 184 snap -1 2024-09-21T18:08:53.933 INFO:tasks.rados.rados.0.smithi160.stdout:2045: write (excl) oid 114 current snap is 0 2024-09-21T18:08:53.933 INFO:tasks.rados.rados.0.smithi160.stdout:2045: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-21T18:08:53.939 INFO:tasks.rados.rados.0.smithi160.stdout:2045: writing smithi16030472-114 from 608684 to 1035621 tid 1 2024-09-21T18:08:53.946 INFO:tasks.rados.rados.0.smithi160.stdout:2045: writing smithi16030472-114 from 1803189 to 2372877 tid 2 2024-09-21T18:08:53.949 INFO:tasks.rados.rados.0.smithi160.stdout:2045: writing smithi16030472-114 from 2924304 to 3008684 tid 3 2024-09-21T18:08:53.949 INFO:tasks.rados.rados.0.smithi160.stdout:2046: cache_try_flush oid 355 snap -1 2024-09-21T18:08:53.950 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:53.986 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:53.984+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:53.989 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:53.989 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:53.992 INFO:tasks.rados.rados.0.smithi160.stdout:2032: finishing write tid 1 to smithi16030472-351 2024-09-21T18:08:53.993 INFO:tasks.rados.rados.0.smithi160.stdout:2035: done (15 left) 2024-09-21T18:08:53.993 INFO:tasks.rados.rados.0.smithi160.stdout:2047: cache_flush oid 417 snap -1 2024-09-21T18:08:53.993 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2031: finishing copy_from to smithi16030472-110 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2031: got expected ENOENT (src dne) 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2031: finishing copy_from racing read to smithi16030472-110 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2038: got (0) Success 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 126 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2031: done (15 left) 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2038: done (14 left) 2024-09-21T18:08:53.997 INFO:tasks.rados.rados.0.smithi160.stdout:2048: delete oid 284 current snap is 0 2024-09-21T18:08:54.000 INFO:tasks.rados.rados.0.smithi160.stdout:2047: got (0) Success 2024-09-21T18:08:54.000 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 0 (ObjNum 502 snap 0 seq_num 502) dirty exists 2024-09-21T18:08:54.000 INFO:tasks.rados.rados.0.smithi160.stdout:2040: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:08:54.009 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:08:54.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:08:54.097 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:54.093+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:54.110 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1245561 2024-09-21T18:08:54.132 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:54.170 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:54.170 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:54.180 INFO:tasks.rados.rados.0.smithi160.stdout:2032: finishing write tid 2 to smithi16030472-351 2024-09-21T18:08:54.184 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:08:54.188 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:54.184+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:54.190 INFO:tasks.rados.rados.0.smithi160.stdout:2033: got (0) Success 2024-09-21T18:08:54.192 INFO:tasks.rados.rados.0.smithi160.stdout:2046: got (0) Success 2024-09-21T18:08:54.192 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 0 (ObjNum 354 snap 0 seq_num 354) dirty exists 2024-09-21T18:08:54.206 INFO:tasks.rados.rados.0.smithi160.stdout:2037: finishing copy_from to smithi16030472-306 2024-09-21T18:08:54.206 INFO:tasks.rados.rados.0.smithi160.stdout:2037: got expected ENOENT (src dne) 2024-09-21T18:08:54.206 INFO:tasks.rados.rados.0.smithi160.stdout:2037: finishing copy_from racing read to smithi16030472-306 2024-09-21T18:08:54.234 INFO:tasks.rados.rados.0.smithi160.stdout:2039: got (0) Success 2024-09-21T18:08:54.234 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 497 v 0 (ObjNum 129 snap 0 seq_num 3125799944) clean dne 2024-09-21T18:08:54.289 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:54.284+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:54.303 INFO:tasks.rados.rados.0.smithi160.stdout:2041: finishing write tid 1 to smithi16030472-201 2024-09-21T18:08:54.304 INFO:tasks.rados.rados.0.smithi160.stdout:2036: finishing copy_from racing read to smithi16030472-206 2024-09-21T18:08:54.304 INFO:tasks.rados.rados.0.smithi160.stdout:2036: finishing copy_from to smithi16030472-206 2024-09-21T18:08:54.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 206 v 417 (ObjNum 678 snap 0 seq_num 678) dirty exists 2024-09-21T18:08:54.312 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:54.312 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:54.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:54.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:08:54.390 INFO:tasks.rados.rados.0.smithi160.stdout:2032: finishing write tid 3 to smithi16030472-351 2024-09-21T18:08:54.390 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:54.384+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:54.392 INFO:tasks.rados.rados.0.smithi160.stdout:2043: finishing write tid 1 to smithi16030472-178 2024-09-21T18:08:54.392 INFO:tasks.rados.rados.0.smithi160.stdout:2043: finishing write tid 2 to smithi16030472-178 2024-09-21T18:08:54.406 INFO:tasks.rados.rados.0.smithi160.stdout:2041: finishing write tid 2 to smithi16030472-201 2024-09-21T18:08:54.406 INFO:tasks.rados.rados.0.smithi160.stdout:2041: finishing write tid 3 to smithi16030472-201 2024-09-21T18:08:54.449 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:54.449 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:54.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:08:54.468 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:54.487 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:54.488 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:54.490 INFO:tasks.rados.rados.0.smithi160.stdout:2032: finishing write tid 4 to smithi16030472-351 2024-09-21T18:08:54.490 INFO:tasks.rados.rados.0.smithi160.stdout:2042: finishing write tid 1 to smithi16030472-156 2024-09-21T18:08:54.490 INFO:tasks.rados.rados.0.smithi160.stdout:2042: finishing write tid 2 to smithi16030472-156 2024-09-21T18:08:54.491 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:54.488+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:54.497 INFO:tasks.rados.rados.0.smithi160.stdout:2042: finishing write tid 3 to smithi16030472-156 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2043: finishing write tid 3 to smithi16030472-178 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2044: got (0) Success 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 184 v 0 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2043: finishing write tid 4 to smithi16030472-178 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2043: oid 178 updating version 0 to 723 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2043: oid 178 version 723 is already newer than 722 2024-09-21T18:08:54.500 INFO:tasks.rados.rados.0.smithi160.stdout:2043: oid 178 version 723 is already newer than 721 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 178 v 723 (ObjNum 1011 snap 0 seq_num 1011) dirty exists 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2043: left oid 178 (ObjNum 1011 snap 0 seq_num 1011) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2033: done (14 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2036: done (13 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2037: done (12 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2039: done (11 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2040: done (10 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2043: done (9 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2044: done (8 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2046: done (7 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2047: done (6 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2048: done (5 left) 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2049: copy_from oid 444 from oid 52 current snap is 0 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2050: copy_from oid 159 from oid 170 current snap is 0 2024-09-21T18:08:54.501 INFO:tasks.rados.rados.0.smithi160.stdout:2051: cache_evict oid 269 snap -1 2024-09-21T18:08:54.502 INFO:tasks.rados.rados.0.smithi160.stdout:2052: cache_try_flush oid 384 snap -1 2024-09-21T18:08:54.502 INFO:tasks.rados.rados.0.smithi160.stdout:2053: write oid 431 current snap is 0 2024-09-21T18:08:54.502 INFO:tasks.rados.rados.0.smithi160.stdout:2053: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-21T18:08:54.507 INFO:tasks.rados.rados.0.smithi160.stdout:2053: writing smithi16030472-431 from 642298 to 1219733 tid 1 2024-09-21T18:08:54.509 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:54.509 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:54.515 INFO:tasks.rados.rados.0.smithi160.stdout:2053: writing smithi16030472-431 from 1920153 to 2493510 tid 2 2024-09-21T18:08:54.518 INFO:tasks.rados.rados.0.smithi160.stdout:2053: writing smithi16030472-431 from 2979150 to 3042298 tid 3 2024-09-21T18:08:54.518 INFO:tasks.rados.rados.0.smithi160.stdout:2054: cache_evict oid 469 snap -1 2024-09-21T18:08:54.519 INFO:tasks.rados.rados.0.smithi160.stdout:2034: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:08:54.519 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:54.530 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:54.530 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:54.547 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:54.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:54.577 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 450115 2024-09-21T18:08:54.586 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:54.587 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:54.587 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:54.587 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:54.587 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:54.591 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:54.585+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:54.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:08:54.628 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:54.628 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:54.637 INFO:tasks.rados.rados.0.smithi160.stdout:2032: finishing write tid 5 to smithi16030472-351 2024-09-21T18:08:54.637 INFO:tasks.rados.rados.0.smithi160.stdout:2032: oid 351 updating version 0 to 736 2024-09-21T18:08:54.637 INFO:tasks.rados.rados.0.smithi160.stdout:2032: oid 351 version 736 is already newer than 733 2024-09-21T18:08:54.637 INFO:tasks.rados.rados.0.smithi160.stdout:2032: oid 351 version 736 is already newer than 734 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2032: oid 351 version 736 is already newer than 735 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 351 v 736 (ObjNum 1008 snap 0 seq_num 1008) dirty exists 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2032: left oid 351 (ObjNum 1008 snap 0 seq_num 1008) 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2051: got (16) Device or resource busy 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2054: got (2) No such file or directory 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2041: finishing write tid 4 to smithi16030472-201 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2041: oid 201 updating version 0 to 650 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2041: oid 201 updating version 650 to 651 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2041: oid 201 updating version 651 to 652 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 201 v 652 (ObjNum 1009 snap 0 seq_num 1009) dirty exists 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2041: left oid 201 (ObjNum 1009 snap 0 seq_num 1009) 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2045: finishing write tid 1 to smithi16030472-114 2024-09-21T18:08:54.638 INFO:tasks.rados.rados.0.smithi160.stdout:2045: finishing write tid 2 to smithi16030472-114 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2032: done (10 left) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2034: done (9 left) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2041: done (8 left) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2051: done (7 left) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2054: done (6 left) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2055: copy_from oid 166 from oid 391 current snap is 0 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: finishing write tid 4 to smithi16030472-156 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: finishing write tid 5 to smithi16030472-156 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: oid 156 updating version 0 to 686 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: oid 156 version 686 is already newer than 685 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: oid 156 updating version 686 to 687 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: oid 156 version 687 is already newer than 684 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 156 v 687 (ObjNum 1010 snap 0 seq_num 1010) dirty exists 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2042: left oid 156 (ObjNum 1010 snap 0 seq_num 1010) 2024-09-21T18:08:54.639 INFO:tasks.rados.rados.0.smithi160.stdout:2045: finishing write tid 3 to smithi16030472-114 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: finishing write tid 4 to smithi16030472-114 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: finishing write tid 5 to smithi16030472-114 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: oid 114 updating version 0 to 656 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: oid 114 version 656 is already newer than 653 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: oid 114 version 656 is already newer than 654 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: oid 114 version 656 is already newer than 655 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 656 (ObjNum 1012 snap 0 seq_num 1012) dirty exists 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: left oid 114 (ObjNum 1012 snap 0 seq_num 1012) 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2042: done (6 left) 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2045: done (5 left) 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2056: read oid 230 snap -1 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2056: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2057: cache_evict oid 267 snap -1 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2058: write oid 384 current snap is 0 2024-09-21T18:08:54.640 INFO:tasks.rados.rados.0.smithi160.stdout:2058: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-21T18:08:54.646 INFO:tasks.rados.rados.0.smithi160.stdout:2058: writing smithi16030472-384 from 659105 to 1311789 tid 1 2024-09-21T18:08:54.655 INFO:tasks.rados.rados.0.smithi160.stdout:2058: writing smithi16030472-384 from 1862282 to 2579297 tid 2 2024-09-21T18:08:54.655 INFO:tasks.rados.rados.0.smithi160.stdout:2059: write oid 138 current snap is 0 2024-09-21T18:08:54.656 INFO:tasks.rados.rados.0.smithi160.stdout:2059: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-21T18:08:54.656 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:54.656 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:54.657 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:54.665 INFO:tasks.rados.rados.0.smithi160.stdout:2059: writing smithi16030472-138 from 675912 to 1403845 tid 1 2024-09-21T18:08:54.672 INFO:tasks.rados.rados.0.smithi160.stdout:2059: writing smithi16030472-138 from 1920764 to 2381437 tid 2 2024-09-21T18:08:54.678 INFO:tasks.rados.rados.0.smithi160.stdout:2059: writing smithi16030472-138 from 2801290 to 3075912 tid 3 2024-09-21T18:08:54.678 INFO:tasks.rados.rados.0.smithi160.stdout:2060: cache_flush oid 259 snap -1 2024-09-21T18:08:54.678 INFO:tasks.rados.rados.0.smithi160.stdout:2056: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:08:54.680 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:54.680 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:54.692 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:54.688+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:54.706 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:54.706 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:54.731 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:54.731 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:54.749 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:54.749 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:54.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:54.749 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:54.749 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:54.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:08:54.808 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:54.812 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:54.806+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:54.846 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:54.846 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:54.858 INFO:tasks.rados.rados.0.smithi160.stdout:2053: finishing write tid 1 to smithi16030472-431 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2053: finishing write tid 2 to smithi16030472-431 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2056: done (9 left) 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2061: copy_from oid 141 from oid 246 current snap is 0 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2060: got (0) Success 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 259 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2053: finishing write tid 3 to smithi16030472-431 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2057: got (0) Success 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2055: finishing copy_from racing read to smithi16030472-166 2024-09-21T18:08:54.859 INFO:tasks.rados.rados.0.smithi160.stdout:2059: finishing write tid 1 to smithi16030472-138 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2057: done (9 left) 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2060: done (8 left) 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2062: read oid 409 snap -1 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2062: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2063: write oid 396 current snap is 0 2024-09-21T18:08:54.860 INFO:tasks.rados.rados.0.smithi160.stdout:2063: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-09-21T18:08:54.869 INFO:tasks.rados.rados.0.smithi160.stdout:2063: writing smithi16030472-396 from 692719 to 1095901 tid 1 2024-09-21T18:08:54.874 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:54.874 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:54.879 INFO:tasks.rados.rados.0.smithi160.stdout:2063: writing smithi16030472-396 from 1579246 to 2299930 tid 2 2024-09-21T18:08:54.886 INFO:tasks.rados.rados.0.smithi160.stdout:2063: writing smithi16030472-396 from 2828713 to 3092719 tid 3 2024-09-21T18:08:54.886 INFO:tasks.rados.rados.0.smithi160.stdout:2064: cache_try_flush oid 216 snap -1 2024-09-21T18:08:54.886 INFO:tasks.rados.rados.0.smithi160.stdout:2065: write oid 45 current snap is 0 2024-09-21T18:08:54.886 INFO:tasks.rados.rados.0.smithi160.stdout:2065: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-21T18:08:54.895 INFO:tasks.rados.rados.0.smithi160.stdout:2065: writing smithi16030472-45 from 709526 to 1187957 tid 1 2024-09-21T18:08:54.895 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:54.891+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:54.899 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:54.899 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:54.902 INFO:tasks.rados.rados.0.smithi160.stdout:2065: writing smithi16030472-45 from 1637728 to 2102070 tid 2 2024-09-21T18:08:54.911 INFO:tasks.rados.rados.0.smithi160.stdout:2065: writing smithi16030472-45 from 2856136 to 3109526 tid 3 2024-09-21T18:08:54.911 INFO:tasks.rados.rados.0.smithi160.stdout:2066: cache_try_flush oid 273 snap -1 2024-09-21T18:08:54.911 INFO:tasks.rados.rados.0.smithi160.stdout:2067: write oid 156 current snap is 0 2024-09-21T18:08:54.911 INFO:tasks.rados.rados.0.smithi160.stdout:2067: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-21T18:08:54.922 INFO:tasks.rados.rados.0.smithi160.stdout:2067: writing smithi16030472-156 from 726333 to 1396366 tid 1 2024-09-21T18:08:54.926 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:54.926 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:54.931 INFO:tasks.rados.rados.0.smithi160.stdout:2067: writing smithi16030472-156 from 2096210 to 2820563 tid 2 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2068: cache_evict oid 280 snap -1 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2069: copy_from oid 114 from oid 40 current snap is 0 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2055: finishing copy_from to smithi16030472-166 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 283 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2055: done (15 left) 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2070: read oid 190 snap -1 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2070: expect deleted 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:54.932 INFO:tasks.rados.rados.0.smithi160.stdout:2062: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:08:54.936 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5869 4.0 4:2fc17355:::smithi16030472-138:head [write 1920764~460673 in=460673b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.672270+0000", 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.26213626299999998, 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.26218186199999999, 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5869 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672270+0000", 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672270+0000", 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672272+0000", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6920000000000001e-06 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672673+0000", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00040101299999999998 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672686+0000", 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3715e-05 2024-09-21T18:08:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.672694+0000", 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2019999999999998e-06 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.746099+0000", 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.073405409000000005 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.746218+0000", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011937700000000001 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.866029+0000", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.119811129 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.866147+0000", 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011773 2024-09-21T18:08:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.866283+0000", 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136055 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870738+0000", 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044545110000000004 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5870 4.0 4:2fc17355:::smithi16030472-138:head [write 2801290~274622 in=274622b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.677395+0000", 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.25701069100000001, 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25720620300000002, 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5870 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677395+0000", 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677395+0000", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677397+0000", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6929999999999999e-06 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677678+0000", 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00028036400000000001 2024-09-21T18:08:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677694+0000", 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6824999999999999e-05 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677704+0000", 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5990000000000008e-06 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846540+0000", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.168835859 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846622+0000", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2076999999999994e-05 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.868147+0000", 2024-09-21T18:08:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021525131999999999 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.868263+0000", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115995 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.868407+0000", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014366300000000001 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870744+0000", 2024-09-21T18:08:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002337338 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5871 4.0 4:2fc17355:::smithi16030472-138:head [setxattr _header (57) in=64b,truncate 3075912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.677721+0000", 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.25668549600000001, 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25701957399999997, 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5871 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677721+0000", 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677721+0000", 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677722+0000", 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.066e-06 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677727+0000", 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0300000000000001e-06 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677735+0000", 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4940000000000003e-06 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677747+0000", 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.206e-05 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846695+0000", 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.16894815799999999 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846753+0000", 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7599000000000002e-05 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.869699+0000", 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.022946040000000001 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.869851+0000", 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015158000000000001 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870043+0000", 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000192015 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870962+0000", 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091973900000000004 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5872 4.0 4:2fc17355:::smithi16030472-138:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.677756+0000", 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.256650137, 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25711909799999999, 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5872 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677756+0000", 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677756+0000", 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677757+0000", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8699999999999995e-07 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677759+0000", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.813e-06 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677768+0000", 2024-09-21T18:08:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.677774+0000", 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7120000000000003e-06 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846803+0000", 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.169028912 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.846848+0000", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4347000000000001e-05 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870629+0000", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.023781673999999999 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.870711+0000", 2024-09-21T18:08:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2003999999999996e-05 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5885 4.0 4:16d9fc1d:::smithi16030472-45:head [write 709526~478431 in=478431b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.894864+0000", 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.039541807999999998, 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.040127491000000001, 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5885 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.894864+0000", 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.894864+0000", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.894868+0000", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9890000000000003e-06 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895297+0000", 2024-09-21T18:08:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00042868799999999999 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895322+0000", 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4752999999999998e-05 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895336+0000", 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4568e-05 2024-09-21T18:08:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895437+0000", 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010016100000000001 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895522+0000", 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5796000000000003e-05 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.895618+0000", 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5662999999999995e-05 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.897919+0000", 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023007119999999999 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:5886 4.0 4.b83f9b68 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e87)", 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:08:54.902311+0000", 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.032095626000000002, 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.032791133, 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "no flag points reached", 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 5886 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.902311+0000", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.902311+0000", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.902313+0000", 2024-09-21T18:08:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1610000000000001e-06 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.902711+0000", 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039873799999999999 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:08:54.902727+0000", 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5237e-05 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:08:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 6 2024-09-21T18:08:54.970 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:54.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:08:54.971 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:54.972 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:55.009 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2384276 2024-09-21T18:08:55.014 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:55.015 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:55.010+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:55.047 INFO:tasks.rados.rados.0.smithi160.stdout:2052: got (16) Device or resource busy 2024-09-21T18:08:55.047 INFO:tasks.rados.rados.0.smithi160.stdout:2058: finishing write tid 1 to smithi16030472-384 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: finishing write tid 2 to smithi16030472-384 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: finishing write tid 3 to smithi16030472-384 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: finishing write tid 4 to smithi16030472-384 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: oid 384 updating version 0 to 737 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: oid 384 updating version 737 to 738 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: oid 384 updating version 738 to 739 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 384 v 739 (ObjNum 1015 snap 0 seq_num 1015) dirty exists 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2058: left oid 384 (ObjNum 1015 snap 0 seq_num 1015) 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2068: got (2) No such file or directory 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: finishing write tid 4 to smithi16030472-431 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: finishing write tid 5 to smithi16030472-431 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: oid 431 updating version 0 to 709 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: oid 431 updating version 709 to 712 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: oid 431 version 712 is already newer than 710 2024-09-21T18:08:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:2053: oid 431 version 712 is already newer than 711 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 431 v 712 (ObjNum 1014 snap 0 seq_num 1014) dirty exists 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2053: left oid 431 (ObjNum 1014 snap 0 seq_num 1014) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2049: finishing copy_from to smithi16030472-444 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2049: got expected ENOENT (src dne) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2064: got (2) No such file or directory 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2050: finishing copy_from to smithi16030472-159 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2050: got expected ENOENT (src dne) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2066: got (2) No such file or directory 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2050: finishing copy_from racing read to smithi16030472-159 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2050: done (15 left) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2052: done (14 left) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2053: done (13 left) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2058: done (12 left) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2062: done (11 left) 2024-09-21T18:08:55.049 INFO:tasks.rados.rados.0.smithi160.stdout:2064: done (10 left) 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2066: done (9 left) 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2068: done (8 left) 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2071: read oid 337 snap -1 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2071: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2072: write oid 466 current snap is 0 2024-09-21T18:08:55.050 INFO:tasks.rados.rados.0.smithi160.stdout:2072: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-21T18:08:55.057 INFO:tasks.rados.rados.0.smithi160.stdout:2072: writing smithi16030472-466 from 743140 to 1488422 tid 1 2024-09-21T18:08:55.066 INFO:tasks.rados.rados.0.smithi160.stdout:2072: writing smithi16030472-466 from 2154692 to 2622703 tid 2 2024-09-21T18:08:55.066 INFO:tasks.rados.rados.0.smithi160.stdout:2073: cache_try_flush oid 448 snap -1 2024-09-21T18:08:55.066 INFO:tasks.rados.rados.0.smithi160.stdout:2074: read oid 145 snap -1 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2074: expect (ObjNum 144 snap 0 seq_num 144) 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2075: copy_from oid 293 from oid 273 current snap is 0 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2076: read oid 1 snap -1 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2076: expect deleted 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2077: write (excl) oid 409 current snap is 0 2024-09-21T18:08:55.067 INFO:tasks.rados.rados.0.smithi160.stdout:2077: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-09-21T18:08:55.074 INFO:tasks.rados.rados.0.smithi160.stdout:2077: writing smithi16030472-409 from 759947 to 1180478 tid 1 2024-09-21T18:08:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:08:55.084 INFO:tasks.rados.rados.0.smithi160.stdout:2077: writing smithi16030472-409 from 1813174 to 2541196 tid 2 2024-09-21T18:08:55.084 INFO:tasks.rados.rados.0.smithi160.stdout:2078: copy_from oid 46 from oid 50 current snap is 0 2024-09-21T18:08:55.084 INFO:tasks.rados.rados.0.smithi160.stdout:2061: finishing copy_from racing read to smithi16030472-141 2024-09-21T18:08:55.084 INFO:tasks.rados.rados.0.smithi160.stdout:2072: finishing write tid 1 to smithi16030472-466 2024-09-21T18:08:55.084 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:55.097 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:55.091+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:55.141 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:55.142 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:55.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout:2061: finishing copy_from to smithi16030472-141 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 141 v 141 (ObjNum 245 snap 0 seq_num 245) dirty exists 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout:2061: done (15 left) 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout:2079: read oid 127 snap -1 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout:2079: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:55.178 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:55.179 INFO:tasks.rados.rados.0.smithi160.stdout:2072: finishing write tid 2 to smithi16030472-466 2024-09-21T18:08:55.199 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:55.194+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: finishing write tid 2 to smithi16030472-138 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: finishing write tid 3 to smithi16030472-138 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: finishing write tid 4 to smithi16030472-138 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: finishing write tid 5 to smithi16030472-138 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: oid 138 updating version 0 to 727 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: oid 138 version 727 is already newer than 725 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: oid 138 version 727 is already newer than 655 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: oid 138 version 727 is already newer than 726 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 138 v 727 (ObjNum 1016 snap 0 seq_num 1016) dirty exists 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: left oid 138 (ObjNum 1016 snap 0 seq_num 1016) 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2065: finishing write tid 1 to smithi16030472-45 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2059: done (15 left) 2024-09-21T18:08:55.205 INFO:tasks.rados.rados.0.smithi160.stdout:2080: cache_flush oid 47 snap -1 2024-09-21T18:08:55.206 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:55.206 INFO:tasks.rados.rados.0.smithi160.stdout:2072: finishing write tid 3 to smithi16030472-466 2024-09-21T18:08:55.207 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:55.203+0000 7f7283507640 -1 osd.14 87 handle_osd_map injecting map crc failure 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:2072: finishing write tid 4 to smithi16030472-466 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:2072: oid 466 updating version 0 to 741 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:2072: oid 466 version 741 is already newer than 740 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:2072: oid 466 updating version 741 to 742 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 466 v 742 (ObjNum 1020 snap 0 seq_num 1020) dirty exists 2024-09-21T18:08:55.282 INFO:tasks.rados.rados.0.smithi160.stdout:2072: left oid 466 (ObjNum 1020 snap 0 seq_num 1020) 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2072: done (15 left) 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2081: read oid 452 snap -1 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2081: expect (ObjNum 451 snap 0 seq_num 451) 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2069: finishing copy_from racing read to smithi16030472-114 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2049: finishing copy_from racing read to smithi16030472-444 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2063: finishing write tid 1 to smithi16030472-396 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2063: finishing write tid 2 to smithi16030472-396 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2049: done (15 left) 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2082: write oid 54 current snap is 0 2024-09-21T18:08:55.283 INFO:tasks.rados.rados.0.smithi160.stdout:2082: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-21T18:08:55.298 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:55.292+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:55.302 INFO:tasks.rados.rados.0.smithi160.stdout:2082: writing smithi16030472-54 from 776754 to 1272534 tid 1 2024-09-21T18:08:55.317 INFO:tasks.rados.rados.0.smithi160.stdout:2082: writing smithi16030472-54 from 1871656 to 2343336 tid 2 2024-09-21T18:08:55.329 INFO:tasks.rados.rados.0.smithi160.stdout:2082: writing smithi16030472-54 from 2965828 to 3176754 tid 3 2024-09-21T18:08:55.329 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:55.329 INFO:tasks.rados.rados.0.smithi160.stdout:2079: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:55.329 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:08:55.333 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 102.119216169, 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:55.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:08:55.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:08:55.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:08:55.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.339 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.575278620000006, 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.414355155999999, 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.345 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:08:55.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:08:55.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:08:55.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.413680561000007, 2024-09-21T18:08:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:55.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:08:55.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.353 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.413654023000007, 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.403820280999994, 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.358 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:08:55.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:08:55.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:08:55.361 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.391408493, 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.362 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:08:55.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:08:55.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.365 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:08:55.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:08:55.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:08:55.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:08:55.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:08:55.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.373 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.373 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.374 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:55.374 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.382518422999993, 2024-09-21T18:08:55.374 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:08:55.374 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:08:55.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:08:55.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:08:55.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.381706406000006, 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.382 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:08:55.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:55.388 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.381598194000006, 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:08:55.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:08:55.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:08:55.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:08:55.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:08:55.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.397 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.281327047999994, 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.398 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:08:55.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.401 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.280430788999993, 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:08:55.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:08:55.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.279949176000002, 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.406 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.279496750999996, 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.410 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:08:55.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.279446425000003, 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.414 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:08:55.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:08:55.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:08:55.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.153713644999996, 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.418 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:08:55.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:08:55.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.149435588000003, 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.424 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:08:55.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.149065347999993, 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.430 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 84.148995618000001, 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.436 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:08:55.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:08:55.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:08:55.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:08:55.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 83.557418627999994, 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:08:55.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:55.446 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:55.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:08:55.448 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:55.442+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:55.481 INFO:tasks.rados.rados.0.smithi160.stdout:2075: finishing copy_from racing read to smithi16030472-293 2024-09-21T18:08:55.481 INFO:tasks.rados.rados.0.smithi160.stdout:2067: finishing write tid 1 to smithi16030472-156 2024-09-21T18:08:55.482 INFO:tasks.rados.rados.0.smithi160.stdout:2074: expect (ObjNum 144 snap 0 seq_num 144) 2024-09-21T18:08:55.509 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 537903 2024-09-21T18:08:55.549 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:55.543+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:55.553 INFO:tasks.rados.rados.0.smithi160.stdout:2065: finishing write tid 2 to smithi16030472-45 2024-09-21T18:08:55.553 INFO:tasks.rados.rados.0.smithi160.stdout:2065: finishing write tid 3 to smithi16030472-45 2024-09-21T18:08:55.553 INFO:tasks.rados.rados.0.smithi160.stdout:2065: finishing write tid 4 to smithi16030472-45 2024-09-21T18:08:55.553 INFO:tasks.rados.rados.0.smithi160.stdout:2065: finishing write tid 5 to smithi16030472-45 2024-09-21T18:08:55.553 INFO:tasks.rados.rados.0.smithi160.stdout:2065: oid 45 updating version 0 to 730 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2065: oid 45 version 730 is already newer than 729 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2065: oid 45 updating version 730 to 731 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2065: oid 45 version 731 is already newer than 728 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 45 v 731 (ObjNum 1018 snap 0 seq_num 1018) dirty exists 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2065: left oid 45 (ObjNum 1018 snap 0 seq_num 1018) 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2067: finishing write tid 2 to smithi16030472-156 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2067: finishing write tid 3 to smithi16030472-156 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2073: got (0) Success 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 0 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2077: finishing write tid 1 to smithi16030472-409 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2069: finishing copy_from to smithi16030472-114 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 633 (ObjNum 841 snap 0 seq_num 841) dirty exists 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2077: finishing write tid 2 to smithi16030472-409 2024-09-21T18:08:55.554 INFO:tasks.rados.rados.0.smithi160.stdout:2077: finishing write tid 3 to smithi16030472-409 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2065: done (15 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2069: done (14 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2070: done (13 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2073: done (12 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2074: done (11 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2079: done (10 left) 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2083: write oid 487 current snap is 0 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2077: finishing write tid 4 to smithi16030472-409 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2077: oid 409 updating version 0 to 657 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2077: oid 409 updating version 657 to 658 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2077: oid 409 updating version 658 to 659 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 659 (ObjNum 1021 snap 0 seq_num 1021) dirty exists 2024-09-21T18:08:55.555 INFO:tasks.rados.rados.0.smithi160.stdout:2077: left oid 409 (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2067: finishing write tid 4 to smithi16030472-156 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2067: oid 156 updating version 0 to 690 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2067: oid 156 version 690 is already newer than 689 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2067: oid 156 version 690 is already newer than 688 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 156 v 690 (ObjNum 1019 snap 0 seq_num 1019) dirty exists 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2067: left oid 156 (ObjNum 1019 snap 0 seq_num 1019) 2024-09-21T18:08:55.556 INFO:tasks.rados.rados.0.smithi160.stdout:2071: expect (ObjNum 369 snap 0 seq_num 369) 2024-09-21T18:08:55.583 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1152671 2024-09-21T18:08:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:2075: finishing copy_from to smithi16030472-293 2024-09-21T18:08:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 395 (ObjNum 523 snap 0 seq_num 523) dirty exists 2024-09-21T18:08:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:2063: finishing write tid 3 to smithi16030472-396 2024-09-21T18:08:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:2078: finishing copy_from racing read to smithi16030472-46 2024-09-21T18:08:55.618 INFO:tasks.rados.rados.0.smithi160.stdout:2083: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2024-09-21T18:08:55.626 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:55.626 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:55.626 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:55.626 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:55.640 INFO:tasks.rados.rados.0.smithi160.stdout:2083: writing smithi16030472-487 from 793561 to 1364590 tid 1 2024-09-21T18:08:55.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:08:55.650 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:55.647+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:55.659 INFO:tasks.rados.rados.0.smithi160.stdout:2083: writing smithi16030472-487 from 1813785 to 2545476 tid 2 2024-09-21T18:08:55.669 INFO:tasks.rados.rados.0.smithi160.stdout:2083: writing smithi16030472-487 from 2993251 to 3193561 tid 3 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2067: done (10 left) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2071: done (9 left) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2075: done (8 left) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2076: done (7 left) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2077: done (6 left) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2084: read oid 167 snap -1 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2084: expect deleted 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2085: cache_evict oid 64 snap -1 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2086: cache_flush oid 289 snap -1 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2087: copy_from oid 178 from oid 322 current snap is 0 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2088: read oid 364 snap -1 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2088: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2089: copy_from oid 324 from oid 120 current snap is 0 2024-09-21T18:08:55.670 INFO:tasks.rados.rados.0.smithi160.stdout:2090: copy_from oid 204 from oid 496 current snap is 0 2024-09-21T18:08:55.671 INFO:tasks.rados.rados.0.smithi160.stdout:2091: cache_evict oid 329 snap -1 2024-09-21T18:08:55.671 INFO:tasks.rados.rados.0.smithi160.stdout:2092: delete oid 237 current snap is 0 2024-09-21T18:08:55.692 INFO:tasks.rados.rados.0.smithi160.stdout:2081: expect (ObjNum 451 snap 0 seq_num 451) 2024-09-21T18:08:55.750 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:55.748+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:55.756 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:55.762 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:55.762 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:55.763 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:55.763 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:55.763 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:55.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:08:55.850 INFO:tasks.rados.rados.0.smithi160.stdout:2082: finishing write tid 1 to smithi16030472-54 2024-09-21T18:08:55.850 INFO:tasks.rados.rados.0.smithi160.stdout:2086: got (0) Success 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 0 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: finishing write tid 4 to smithi16030472-396 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: finishing write tid 5 to smithi16030472-396 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: oid 396 updating version 0 to 724 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: oid 396 version 724 is already newer than 723 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: oid 396 updating version 724 to 726 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: oid 396 version 726 is already newer than 725 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 396 v 726 (ObjNum 1017 snap 0 seq_num 1017) dirty exists 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2063: left oid 396 (ObjNum 1017 snap 0 seq_num 1017) 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:2088: expect (ObjNum 670 snap 0 seq_num 670) 2024-09-21T18:08:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:55.852 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:55.848+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:55.897 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:55.898 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:55.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:55.898 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:08:55.898 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:55.910 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1227462 2024-09-21T18:08:55.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:08:55.952 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:08:55.947+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:08:55.960 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1548111 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2091: got (16) Device or resource busy 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2078: finishing copy_from to smithi16030472-46 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 40 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2082: finishing write tid 2 to smithi16030472-54 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2085: got (16) Device or resource busy 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2090: finishing copy_from racing read to smithi16030472-204 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2082: finishing write tid 3 to smithi16030472-54 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2080: got (0) Success 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 0 (ObjNum 958 snap 0 seq_num 958) dirty exists 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2087: finishing copy_from to smithi16030472-178 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2087: got expected ENOENT (src dne) 2024-09-21T18:08:55.988 INFO:tasks.rados.rados.0.smithi160.stdout:2087: finishing copy_from racing read to smithi16030472-178 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: finishing write tid 4 to smithi16030472-54 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: finishing write tid 5 to smithi16030472-54 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: oid 54 updating version 0 to 744 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: oid 54 updating version 744 to 746 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: oid 54 version 746 is already newer than 745 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:2082: oid 54 version 746 is already newer than 743 2024-09-21T18:08:56.003 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 746 (ObjNum 1022 snap 0 seq_num 1022) dirty exists 2024-09-21T18:08:56.004 INFO:tasks.rados.rados.0.smithi160.stdout:2082: left oid 54 (ObjNum 1022 snap 0 seq_num 1022) 2024-09-21T18:08:56.009 INFO:tasks.rados.rados.0.smithi160.stdout:2083: finishing write tid 1 to smithi16030472-487 2024-09-21T18:08:56.009 INFO:tasks.rados.rados.0.smithi160.stdout:2083: finishing write tid 2 to smithi16030472-487 2024-09-21T18:08:56.009 INFO:tasks.rados.rados.0.smithi160.stdout:2083: finishing write tid 3 to smithi16030472-487 2024-09-21T18:08:56.011 INFO:tasks.rados.rados.0.smithi160.stdout:2083: finishing write tid 4 to smithi16030472-487 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: finishing write tid 5 to smithi16030472-487 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: oid 487 updating version 0 to 732 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: oid 487 updating version 732 to 734 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: oid 487 version 734 is already newer than 733 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: oid 487 updating version 734 to 735 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 487 v 735 (ObjNum 1023 snap 0 seq_num 1023) dirty exists 2024-09-21T18:08:56.012 INFO:tasks.rados.rados.0.smithi160.stdout:2083: left oid 487 (ObjNum 1023 snap 0 seq_num 1023) 2024-09-21T18:08:56.020 INFO:tasks.rados.rados.0.smithi160.stdout:2090: finishing copy_from to smithi16030472-204 2024-09-21T18:08:56.020 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 204 v 383 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:08:56.041 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:08:56.041 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:56.053 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:08:56.046+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:08:56.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2063: done (14 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2078: done (13 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2080: done (12 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2081: done (11 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2082: done (10 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2083: done (9 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2084: done (8 left) 2024-09-21T18:08:56.111 INFO:tasks.rados.rados.0.smithi160.stdout:2085: done (7 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2086: done (6 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2087: done (5 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2088: done (4 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2090: done (3 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2091: done (2 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2092: done (1 left) 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2093: cache_try_flush oid 79 snap -1 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2094: write oid 35 current snap is 0 2024-09-21T18:08:56.112 INFO:tasks.rados.rados.0.smithi160.stdout:2094: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-21T18:08:56.126 INFO:tasks.rados.rados.0.smithi160.stdout:2094: writing smithi16030472-35 from 427175 to 1148702 tid 1 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2094: writing smithi16030472-35 from 1930749 to 2666109 tid 2 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2095: read oid 39 snap -1 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2095: expect (ObjNum 858 snap 0 seq_num 858) 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2096: cache_flush oid 134 snap -1 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2097: write (excl) oid 73 current snap is 0 2024-09-21T18:08:56.146 INFO:tasks.rados.rados.0.smithi160.stdout:2097: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-21T18:08:56.153 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:56.146+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:56.161 INFO:tasks.rados.rados.0.smithi160.stdout:2097: writing smithi16030472-73 from 443982 to 1240758 tid 1 2024-09-21T18:08:56.174 INFO:tasks.rados.rados.0.smithi160.stdout:2097: writing smithi16030472-73 from 1989231 to 2468249 tid 2 2024-09-21T18:08:56.182 INFO:tasks.rados.rados.0.smithi160.stdout:2097: writing smithi16030472-73 from 2959167 to 3243982 tid 3 2024-09-21T18:08:56.182 INFO:tasks.rados.rados.0.smithi160.stdout:2098: read oid 292 snap -1 2024-09-21T18:08:56.182 INFO:tasks.rados.rados.0.smithi160.stdout:2098: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:56.182 INFO:tasks.rados.rados.0.smithi160.stdout:2099: delete oid 219 current snap is 0 2024-09-21T18:08:56.225 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:56.225 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:08:56.225 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:08:56.225 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:08:56.225 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:56.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:08:56.262 INFO:tasks.rados.rados.0.smithi160.stdout:2095: expect (ObjNum 858 snap 0 seq_num 858) 2024-09-21T18:08:56.262 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:56.283 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:56.278+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:56.355 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:56.350+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:56.384 INFO:tasks.rados.rados.0.smithi160.stdout:2098: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:08:56.385 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:56.410 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:08:56.410 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:08:56.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:08:56.410 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 130.512589265, 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:56.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 129.42719687600001, 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:56.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:08:56.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:56.418 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:56.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:56.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:08:56.419 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:08:56.419 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:08:56.425 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:08:56.451 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1414177 2024-09-21T18:08:56.456 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:56.449+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:56.483 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:2089: finishing copy_from to smithi16030472-324 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 324 v 79 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:2093: got (0) Success 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 0 (ObjNum 785 snap 0 seq_num 785) dirty exists 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:2089: finishing copy_from racing read to smithi16030472-324 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:2097: finishing write tid 1 to smithi16030472-73 2024-09-21T18:08:56.535 INFO:tasks.rados.rados.0.smithi160.stdout:2097: finishing write tid 2 to smithi16030472-73 2024-09-21T18:08:56.536 INFO:tasks.rados.rados.0.smithi160.stdout:2097: finishing write tid 3 to smithi16030472-73 2024-09-21T18:08:56.552 INFO:tasks.rados.rados.0.smithi160.stdout:2097: finishing write tid 4 to smithi16030472-73 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: finishing write tid 5 to smithi16030472-73 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: oid 73 updating version 0 to 739 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: oid 73 updating version 739 to 740 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: oid 73 version 740 is already newer than 737 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: oid 73 version 740 is already newer than 738 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 73 v 740 (ObjNum 1026 snap 0 seq_num 1026) dirty exists 2024-09-21T18:08:56.553 INFO:tasks.rados.rados.0.smithi160.stdout:2097: left oid 73 (ObjNum 1026 snap 0 seq_num 1026) 2024-09-21T18:08:56.556 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:56.552+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:2094: finishing write tid 1 to smithi16030472-35 2024-09-21T18:08:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:2094: finishing write tid 2 to smithi16030472-35 2024-09-21T18:08:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:2094: finishing write tid 3 to smithi16030472-35 2024-09-21T18:08:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:2094: finishing write tid 4 to smithi16030472-35 2024-09-21T18:08:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:2094: oid 35 updating version 0 to 729 2024-09-21T18:08:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:2094: oid 35 version 729 is already newer than 728 2024-09-21T18:08:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:2094: oid 35 version 729 is already newer than 727 2024-09-21T18:08:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 729 (ObjNum 1025 snap 0 seq_num 1025) dirty exists 2024-09-21T18:08:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:2094: left oid 35 (ObjNum 1025 snap 0 seq_num 1025) 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:868 3.0 3:317cb061:::smithi16030472-134:head [copy-from ver 486 in=56b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e88)", 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:56.246651+0000", 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.313414627, 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31347597399999999, 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246651+0000", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246651+0000", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246655+0000", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3170000000000003e-06 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246671+0000", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6540000000000001e-05 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246703+0000", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1696000000000003e-05 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246721+0000", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8012999999999998e-05 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.246903+0000", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000182072 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.247137+0000", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000233365 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.355149+0000", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10801214200000001 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.369471+0000", 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.014322477 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(osd.4.21:868 3.0 e88/31)", 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:56.466810+0000", 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.093254791000000004, 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.093416846999999997, 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466810+0000", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466810+0000", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466814+0000", 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8060000000000001e-06 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466826+0000", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1311000000000001e-05 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466849+0000", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3621e-05 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:56.466862+0000", 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3234000000000001e-05 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.567 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.568 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.568 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:08:56.568 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 2 2024-09-21T18:08:56.568 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:56.568 INFO:tasks.rados.rados.0.smithi160.stdout:2089: done (7 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2093: done (6 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2094: done (5 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2095: done (4 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2097: done (3 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2098: done (2 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2099: done (1 left) 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2100: cache_try_flush oid 247 snap -1 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2101: cache_try_flush oid 116 snap -1 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2102: delete oid 415 current snap is 0 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:2100: got (0) Success 2024-09-21T18:08:56.569 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 0 (ObjNum 246 snap 0 seq_num 246) dirty exists 2024-09-21T18:08:56.570 INFO:tasks.rados.rados.0.smithi160.stdout:2096: got (0) Success 2024-09-21T18:08:56.570 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 134 v 0 (ObjNum 650 snap 0 seq_num 650) dirty exists 2024-09-21T18:08:56.576 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:08:56.663 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:56.660+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:56.669 INFO:tasks.rados.rados.0.smithi160.stdout:2096: done (3 left) 2024-09-21T18:08:56.669 INFO:tasks.rados.rados.0.smithi160.stdout:2100: done (2 left) 2024-09-21T18:08:56.669 INFO:tasks.rados.rados.0.smithi160.stdout:2102: done (1 left) 2024-09-21T18:08:56.669 INFO:tasks.rados.rados.0.smithi160.stdout:2103: cache_flush oid 236 snap -1 2024-09-21T18:08:56.669 INFO:tasks.rados.rados.0.smithi160.stdout:2104: copy_from oid 500 from oid 64 current snap is 0 2024-09-21T18:08:56.670 INFO:tasks.rados.rados.0.smithi160.stdout:2105: read oid 140 snap -1 2024-09-21T18:08:56.670 INFO:tasks.rados.rados.0.smithi160.stdout:2105: expect (ObjNum 924 snap 0 seq_num 924) 2024-09-21T18:08:56.670 INFO:tasks.rados.rados.0.smithi160.stdout:2106: read oid 129 snap -1 2024-09-21T18:08:56.670 INFO:tasks.rados.rados.0.smithi160.stdout:2106: expect (ObjNum 651 snap 0 seq_num 651) 2024-09-21T18:08:56.670 INFO:tasks.rados.rados.0.smithi160.stdout:2107: delete oid 147 current snap is 0 2024-09-21T18:08:56.671 INFO:tasks.rados.rados.0.smithi160.stdout:2103: got (2) No such file or directory 2024-09-21T18:08:56.745 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:56.745 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:56.745 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:08:56.745 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:08:56.746 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:56.758 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:56.755+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:56.760 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:08:56.773 INFO:tasks.rados.rados.0.smithi160.stdout:2103: done (5 left) 2024-09-21T18:08:56.774 INFO:tasks.rados.rados.0.smithi160.stdout:2107: done (4 left) 2024-09-21T18:08:56.774 INFO:tasks.rados.rados.0.smithi160.stdout:2108: read oid 360 snap -1 2024-09-21T18:08:56.774 INFO:tasks.rados.rados.0.smithi160.stdout:2108: expect (ObjNum 915 snap 0 seq_num 915) 2024-09-21T18:08:56.774 INFO:tasks.rados.rados.0.smithi160.stdout:2109: write oid 206 current snap is 0 2024-09-21T18:08:56.774 INFO:tasks.rados.rados.0.smithi160.stdout:2109: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-21T18:08:56.787 INFO:tasks.rados.rados.0.smithi160.stdout:2109: writing smithi16030472-206 from 511210 to 925335 tid 1 2024-09-21T18:08:56.803 INFO:tasks.rados.rados.0.smithi160.stdout:2109: writing smithi16030472-206 from 1423159 to 1909515 tid 2 2024-09-21T18:08:56.823 INFO:tasks.rados.rados.0.smithi160.stdout:2109: writing smithi16030472-206 from 2668859 to 3258444 tid 3 2024-09-21T18:08:56.823 INFO:tasks.rados.rados.0.smithi160.stdout:2110: cache_try_flush oid 380 snap -1 2024-09-21T18:08:56.823 INFO:tasks.rados.rados.0.smithi160.stdout:2111: write oid 345 current snap is 0 2024-09-21T18:08:56.823 INFO:tasks.rados.rados.0.smithi160.stdout:2105: expect (ObjNum 924 snap 0 seq_num 924) 2024-09-21T18:08:56.859 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:56.853+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:56.918 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1389414 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885826690000002, 2024-09-21T18:08:56.928 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.929 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:08:56.930 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.931 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.932 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.933 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885587065999999, 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.934 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.935 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.936 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:08:56.937 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.938 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.939 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885551659000001, 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.940 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.941 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.942 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.943 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885518485999995, 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.944 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.945 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.946 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.947 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885488925999994, 2024-09-21T18:08:56.948 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.949 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.950 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.951 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.952 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.885459269999998, 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:56.953 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.954 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:08:56.955 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:08:56.956 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.957 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.884973013999996, 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:08:56.958 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.959 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.960 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.961 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.962 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.884939420999999, 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.963 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:08:56.964 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.965 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:08:56.966 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:08:56.967 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.968 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.784721728999997, 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.969 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.970 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:08:56.971 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.972 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.973 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:08:56.974 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.784122824999997, 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.975 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:08:56.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.980 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.783408207999997, 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:08:56.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.774155476000004, 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:56.988 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:08:56.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:08:56.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:08:56.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:08:56.993 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.773890774999998, 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:56.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:08:56.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:56.998 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.773796086000004, 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:56.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.773754796000006, 2024-09-21T18:08:57.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:08:57.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.007 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.762484893000007, 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:08:57.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:08:57.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:08:57.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:57.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.012 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.753368660000007, 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:08:57.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.018 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.752305293000006, 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:57.023 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 85.752264983000003, 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:08:57.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 84.989627141, 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:08:57.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:08:57.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:08:57.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:08:57.034 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:57.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:08:57.036 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:57.031+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:57.036 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 737591 2024-09-21T18:08:57.036 INFO:tasks.rados.rados.0.smithi160.stdout:2106: expect (ObjNum 651 snap 0 seq_num 651) 2024-09-21T18:08:57.037 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:08:57.048 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 905300 2024-09-21T18:08:57.092 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1728711 2024-09-21T18:08:57.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2104: finishing copy_from racing read to smithi16030472-500 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2104: finishing copy_from to smithi16030472-500 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 500 v 40 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2101: got (0) Success 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 0 (ObjNum 869 snap 0 seq_num 869) dirty exists 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2109: finishing write tid 1 to smithi16030472-206 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2109: finishing write tid 2 to smithi16030472-206 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2110: got (2) No such file or directory 2024-09-21T18:08:57.119 INFO:tasks.rados.rados.0.smithi160.stdout:2108: expect (ObjNum 915 snap 0 seq_num 915) 2024-09-21T18:08:57.122 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 530427 2024-09-21T18:08:57.137 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:57.132+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:57.169 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:08:57.169 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:08:57.169 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:08:57.169 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:08:57.184 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:08:57.196 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:57.217 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:57.217 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:57.219 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:57.238 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:57.235+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:57.241 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:57.242 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:57.260 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:57.260 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:57.272 INFO:tasks.rados.rados.0.smithi160.stdout:2109: finishing write tid 3 to smithi16030472-206 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: finishing write tid 4 to smithi16030472-206 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: finishing write tid 5 to smithi16030472-206 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: oid 206 updating version 0 to 714 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: oid 206 version 714 is already newer than 713 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: oid 206 updating version 714 to 715 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: oid 206 updating version 715 to 716 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 206 v 716 (ObjNum 1030 snap 0 seq_num 1030) dirty exists 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2109: left oid 206 (ObjNum 1030 snap 0 seq_num 1030) 2024-09-21T18:08:57.273 INFO:tasks.rados.rados.0.smithi160.stdout:2111: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-09-21T18:08:57.276 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:57.277 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:57.290 INFO:tasks.rados.rados.0.smithi160.stdout:2111: writing smithi16030472-345 from 528017 to 1017391 tid 1 2024-09-21T18:08:57.294 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:57.294 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:57.308 INFO:tasks.rados.rados.0.smithi160.stdout:2111: writing smithi16030472-345 from 1481641 to 2111655 tid 2 2024-09-21T18:08:57.315 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2111: writing smithi16030472-345 from 2696282 to 3097139 tid 3 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2101: done (7 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2104: done (6 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2105: done (5 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2106: done (4 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2108: done (3 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2109: done (2 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2110: done (1 left) 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2112: cache_evict oid 163 snap -1 2024-09-21T18:08:57.324 INFO:tasks.rados.rados.0.smithi160.stdout:2113: delete oid 434 current snap is 0 2024-09-21T18:08:57.325 INFO:tasks.rados.rados.0.smithi160.stdout:2112: got (2) No such file or directory 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2112: done (2 left) 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2113: done (1 left) 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2114: read oid 424 snap -1 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2114: expect deleted 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2115: write (excl) oid 207 current snap is 0 2024-09-21T18:08:57.328 INFO:tasks.rados.rados.0.smithi160.stdout:2115: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-09-21T18:08:57.334 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:57.334 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:57.338 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:57.333+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:57.343 INFO:tasks.rados.rados.0.smithi160.stdout:2115: writing smithi16030472-207 from 561631 to 1201503 tid 1 2024-09-21T18:08:57.354 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:57.355 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:57.355 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:57.360 INFO:tasks.rados.rados.0.smithi160.stdout:2115: writing smithi16030472-207 from 1882252 to 2515935 tid 2 2024-09-21T18:08:57.370 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:08:57.371 INFO:tasks.rados.rados.0.smithi160.stdout:2115: writing smithi16030472-207 from 3034775 to 3361631 tid 3 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2116: cache_try_flush oid 65 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2117: read oid 331 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2117: expect deleted 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2118: read oid 86 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2118: expect deleted 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2119: read oid 110 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2119: expect (ObjNum 109 snap 0 seq_num 109) 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2120: cache_flush oid 343 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2121: cache_flush oid 373 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2122: cache_evict oid 322 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2123: cache_evict oid 104 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2124: read oid 133 snap -1 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2124: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-21T18:08:57.372 INFO:tasks.rados.rados.0.smithi160.stdout:2125: read oid 455 snap -1 2024-09-21T18:08:57.373 INFO:tasks.rados.rados.0.smithi160.stdout:2125: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-21T18:08:57.373 INFO:tasks.rados.rados.0.smithi160.stdout:2126: delete oid 73 current snap is 0 2024-09-21T18:08:57.374 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:57.374 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:57.439 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:57.437+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:57.472 INFO:tasks.rados.rados.0.smithi160.stdout:2115: finishing write tid 1 to smithi16030472-207 2024-09-21T18:08:57.472 INFO:tasks.rados.rados.0.smithi160.stdout:2115: finishing write tid 2 to smithi16030472-207 2024-09-21T18:08:57.496 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:08:57.497 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:08:57.500 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:57.500 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:57.514 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:08:57.515 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:08:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:08:57.535 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:08:57.535 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:08:57.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:08:57.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:08:57.559 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:57.556+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:57.582 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:08:57.582 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:08:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:08:57.587 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:08:57.610 INFO:tasks.rados.rados.0.smithi160.stdout:2120: got (0) Success 2024-09-21T18:08:57.610 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 0 (ObjNum 33 snap 0 seq_num 3137130280) clean dne 2024-09-21T18:08:57.610 INFO:tasks.rados.rados.0.smithi160.stdout:2121: got (0) Success 2024-09-21T18:08:57.610 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 373 v 0 (ObjNum 372 snap 0 seq_num 372) dirty exists 2024-09-21T18:08:57.636 INFO:tasks.rados.rados.0.smithi160.stdout:2122: got (16) Device or resource busy 2024-09-21T18:08:57.636 INFO:tasks.rados.rados.0.smithi160.stdout:2115: finishing write tid 3 to smithi16030472-207 2024-09-21T18:08:57.641 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:08:57.634+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:08:57.643 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:57.643 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:57.657 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:08:57.676 INFO:tasks.rados.rados.0.smithi160.stdout:2115: finishing write tid 4 to smithi16030472-207 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: finishing write tid 5 to smithi16030472-207 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: oid 207 updating version 0 to 719 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: oid 207 version 719 is already newer than 717 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: oid 207 updating version 719 to 720 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: oid 207 version 720 is already newer than 718 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 207 v 720 (ObjNum 1033 snap 0 seq_num 1033) dirty exists 2024-09-21T18:08:57.677 INFO:tasks.rados.rados.0.smithi160.stdout:2115: left oid 207 (ObjNum 1033 snap 0 seq_num 1033) 2024-09-21T18:08:57.743 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:57.737+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:57.773 INFO:tasks.rados.rados.0.smithi160.stdout:2125: expect (ObjNum 454 snap 0 seq_num 454) 2024-09-21T18:08:57.773 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:57.824 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:57.824 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:57.842 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:08:57.844 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:57.839+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:57.860 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 442030 2024-09-21T18:08:57.897 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:57.935 INFO:tasks.rados.rados.0.smithi160.stdout:2119: expect (ObjNum 109 snap 0 seq_num 109) 2024-09-21T18:08:57.936 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:57.946 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:57.941+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:57.970 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:57.970 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:57.984 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:08:58.047 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:58.041+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: finishing write tid 1 to smithi16030472-345 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: finishing write tid 2 to smithi16030472-345 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: finishing write tid 3 to smithi16030472-345 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: finishing write tid 4 to smithi16030472-345 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: finishing write tid 5 to smithi16030472-345 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: oid 345 updating version 0 to 742 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: oid 345 version 742 is already newer than 741 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: oid 345 updating version 742 to 743 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: oid 345 updating version 743 to 744 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 345 v 744 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2111: left oid 345 (ObjNum 1031 snap 0 seq_num 1031) 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2116: got (0) Success 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 65 v 0 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:08:58.056 INFO:tasks.rados.rados.0.smithi160.stdout:2123: got (0) Success 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2111: done (13 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2114: done (12 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2115: done (11 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2116: done (10 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2117: done (9 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2118: done (8 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2119: done (7 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2120: done (6 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2121: done (5 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2122: done (4 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2123: done (3 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2125: done (2 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2126: done (1 left) 2024-09-21T18:08:58.057 INFO:tasks.rados.rados.0.smithi160.stdout:2127: cache_try_flush oid 220 snap -1 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2128: read oid 127 snap -1 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2128: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2129: cache_try_flush oid 316 snap -1 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2130: read oid 68 snap -1 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2130: expect deleted 2024-09-21T18:08:58.058 INFO:tasks.rados.rados.0.smithi160.stdout:2131: delete oid 458 current snap is 0 2024-09-21T18:08:58.110 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.111 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.127 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:08:58.147 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:08:58.141+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:08:58.151 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:08:58.162 INFO:tasks.rados.rados.0.smithi160.stdout:2128: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:08:58.248 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:58.242+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:58.250 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 691980 2024-09-21T18:08:58.257 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.257 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.272 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:08:58.285 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1368076 2024-09-21T18:08:58.311 INFO:tasks.rados.rados.0.smithi160.stdout:2124: expect (ObjNum 132 snap 0 seq_num 132) 2024-09-21T18:08:58.311 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:58.349 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:08:58.344+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:08:58.371 INFO:tasks.rados.rados.0.smithi160.stdout:2129: got (0) Success 2024-09-21T18:08:58.371 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 316 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:08:58.371 INFO:tasks.rados.rados.0.smithi160.stdout:2124: done (5 left) 2024-09-21T18:08:58.371 INFO:tasks.rados.rados.0.smithi160.stdout:2128: done (4 left) 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2129: done (3 left) 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2130: done (2 left) 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2131: done (1 left) 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2132: copy_from oid 145 from oid 179 current snap is 0 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2133: cache_flush oid 282 snap -1 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2134: cache_flush oid 82 snap -1 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2135: read oid 281 snap -1 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2135: expect (ObjNum 974 snap 0 seq_num 974) 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2136: read oid 296 snap -1 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2136: expect deleted 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2137: copy_from oid 437 from oid 401 current snap is 0 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2138: write oid 78 current snap is 0 2024-09-21T18:08:58.372 INFO:tasks.rados.rados.0.smithi160.stdout:2138: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-21T18:08:58.390 INFO:tasks.rados.rados.0.smithi160.stdout:2138: writing smithi16030472-78 from 612052 to 1194024 tid 1 2024-09-21T18:08:58.400 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.400 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.406 INFO:tasks.rados.rados.0.smithi160.stdout:2138: writing smithi16030472-78 from 1657698 to 2155061 tid 2 2024-09-21T18:08:58.415 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:08:58.425 INFO:tasks.rados.rados.0.smithi160.stdout:2138: writing smithi16030472-78 from 2717044 to 3412052 tid 3 2024-09-21T18:08:58.425 INFO:tasks.rados.rados.0.smithi160.stdout:2139: write (excl) oid 210 current snap is 0 2024-09-21T18:08:58.425 INFO:tasks.rados.rados.0.smithi160.stdout:2139: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-21T18:08:58.440 INFO:tasks.rados.rados.0.smithi160.stdout:2139: writing smithi16030472-210 from 628859 to 1286080 tid 1 2024-09-21T18:08:58.450 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:58.445+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:58.450 INFO:tasks.rados.rados.0.smithi160.stdout:2139: writing smithi16030472-210 from 1716180 to 2357201 tid 2 2024-09-21T18:08:58.459 INFO:tasks.rados.rados.0.smithi160.stdout:2139: writing smithi16030472-210 from 3144467 to 3428859 tid 3 2024-09-21T18:08:58.459 INFO:tasks.rados.rados.0.smithi160.stdout:2140: write (excl) oid 264 current snap is 0 2024-09-21T18:08:58.459 INFO:tasks.rados.rados.0.smithi160.stdout:2140: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-21T18:08:58.469 INFO:tasks.rados.rados.0.smithi160.stdout:2140: writing smithi16030472-264 from 645666 to 1378136 tid 1 2024-09-21T18:08:58.477 INFO:tasks.rados.rados.0.smithi160.stdout:2140: writing smithi16030472-264 from 2174662 to 2675694 tid 2 2024-09-21T18:08:58.482 INFO:tasks.rados.rados.0.smithi160.stdout:2140: writing smithi16030472-264 from 3288243 to 3445666 tid 3 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2141: cache_evict oid 450 snap -1 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2142: read oid 47 snap -1 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2142: expect (ObjNum 958 snap 0 seq_num 958) 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2143: cache_try_flush oid 279 snap -1 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2127: got (0) Success 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 0 (ObjNum 947 snap 0 seq_num 947) dirty exists 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2133: got (0) Success 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 282 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:2135: expect (ObjNum 974 snap 0 seq_num 974) 2024-09-21T18:08:58.483 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:58.547 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.547 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.550 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:58.545+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:58.562 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:08:58.651 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:58.645+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2137: finishing copy_from racing read to smithi16030472-437 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2127: done (12 left) 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2133: done (11 left) 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2135: done (10 left) 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2136: done (9 left) 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2144: cache_evict oid 7 snap -1 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2145: write oid 152 current snap is 0 2024-09-21T18:08:58.662 INFO:tasks.rados.rados.0.smithi160.stdout:2145: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2024-09-21T18:08:58.671 INFO:tasks.rados.rados.0.smithi160.stdout:2145: writing smithi16030472-152 from 662473 to 1070192 tid 1 2024-09-21T18:08:58.676 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:58.671+0000 7f3ce267c640 -1 osd.6 88 handle_osd_map injecting map crc failure 2024-09-21T18:08:58.681 INFO:tasks.rados.rados.0.smithi160.stdout:2145: writing smithi16030472-152 from 1833144 to 2477834 tid 2 2024-09-21T18:08:58.688 INFO:tasks.rados.rados.0.smithi160.stdout:2145: writing smithi16030472-152 from 3199313 to 3462473 tid 3 2024-09-21T18:08:58.688 INFO:tasks.rados.rados.0.smithi160.stdout:2146: cache_flush oid 163 snap -1 2024-09-21T18:08:58.688 INFO:tasks.rados.rados.0.smithi160.stdout:2147: read oid 409 snap -1 2024-09-21T18:08:58.689 INFO:tasks.rados.rados.0.smithi160.stdout:2147: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:08:58.689 INFO:tasks.rados.rados.0.smithi160.stdout:2148: delete oid 153 current snap is 0 2024-09-21T18:08:58.689 INFO:tasks.rados.rados.0.smithi160.stdout:2146: got (2) No such file or directory 2024-09-21T18:08:58.690 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.690 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.704 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:08:58.748 INFO:tasks.rados.rados.0.smithi160.stdout:2138: finishing write tid 1 to smithi16030472-78 2024-09-21T18:08:58.748 INFO:tasks.rados.rados.0.smithi160.stdout:2138: finishing write tid 2 to smithi16030472-78 2024-09-21T18:08:58.748 INFO:tasks.rados.rados.0.smithi160.stdout:2138: finishing write tid 3 to smithi16030472-78 2024-09-21T18:08:58.749 INFO:tasks.rados.rados.0.smithi160.stdout:2138: finishing write tid 4 to smithi16030472-78 2024-09-21T18:08:58.749 INFO:tasks.rados.rados.0.smithi160.stdout:2138: finishing write tid 5 to smithi16030472-78 2024-09-21T18:08:58.749 INFO:tasks.rados.rados.0.smithi160.stdout:2138: oid 78 updating version 0 to 747 2024-09-21T18:08:58.749 INFO:tasks.rados.rados.0.smithi160.stdout:2138: oid 78 updating version 747 to 749 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:2138: oid 78 version 749 is already newer than 746 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:2138: oid 78 version 749 is already newer than 748 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 749 (ObjNum 1036 snap 0 seq_num 1036) dirty exists 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:2138: left oid 78 (ObjNum 1036 snap 0 seq_num 1036) 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:2141: got (0) Success 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:2137: finishing copy_from to smithi16030472-437 2024-09-21T18:08:58.750 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 437 v 295 (ObjNum 400 snap 0 seq_num 400) dirty exists 2024-09-21T18:08:58.751 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:08:58.745+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:08:58.754 INFO:tasks.rados.rados.0.smithi160.stdout:2139: finishing write tid 1 to smithi16030472-210 2024-09-21T18:08:58.829 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.829 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.844 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:08:58.852 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:08:58.846+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:08:58.856 INFO:tasks.rados.rados.0.smithi160.stdout:2132: finishing copy_from to smithi16030472-145 2024-09-21T18:08:58.856 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 145 v 115 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-21T18:08:58.856 INFO:tasks.rados.rados.0.smithi160.stdout:2134: got (0) Success 2024-09-21T18:08:58.856 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 0 (ObjNum 964 snap 0 seq_num 964) dirty exists 2024-09-21T18:08:58.860 INFO:tasks.rados.rados.0.smithi160.stdout:2142: expect (ObjNum 958 snap 0 seq_num 958) 2024-09-21T18:08:58.953 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:08:58.951+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:08:58.970 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:58.970 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:58.984 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:08:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:2139: finishing write tid 2 to smithi16030472-210 2024-09-21T18:08:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:2139: finishing write tid 3 to smithi16030472-210 2024-09-21T18:08:58.985 INFO:tasks.rados.rados.0.smithi160.stdout:2147: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:08:59.043 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 915336 2024-09-21T18:08:59.054 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:08:59.046+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:08:59.116 INFO:tasks.rados.rados.0.smithi160.stdout:2132: finishing copy_from racing read to smithi16030472-145 2024-09-21T18:08:59.116 INFO:tasks.rados.rados.0.smithi160.stdout:2145: finishing write tid 1 to smithi16030472-152 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: finishing write tid 4 to smithi16030472-210 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: finishing write tid 5 to smithi16030472-210 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: oid 210 updating version 0 to 751 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: oid 210 version 751 is already newer than 750 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: oid 210 version 751 is already newer than 749 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: oid 210 updating version 751 to 752 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 752 (ObjNum 1037 snap 0 seq_num 1037) dirty exists 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2139: left oid 210 (ObjNum 1037 snap 0 seq_num 1037) 2024-09-21T18:08:59.117 INFO:tasks.rados.rados.0.smithi160.stdout:2144: got (0) Success 2024-09-21T18:08:59.143 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:59.143 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:59.155 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:59.151+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:59.158 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:08:59.162 INFO:tasks.rados.rados.0.smithi160.stdout:2140: finishing write tid 1 to smithi16030472-264 2024-09-21T18:08:59.162 INFO:tasks.rados.rados.0.smithi160.stdout:2145: finishing write tid 2 to smithi16030472-152 2024-09-21T18:08:59.162 INFO:tasks.rados.rados.0.smithi160.stdout:2145: finishing write tid 3 to smithi16030472-152 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: finishing write tid 4 to smithi16030472-152 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: finishing write tid 5 to smithi16030472-152 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: oid 152 updating version 0 to 731 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: oid 152 version 731 is already newer than 726 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: oid 152 updating version 731 to 733 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: oid 152 version 733 is already newer than 732 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 152 v 733 (ObjNum 1039 snap 0 seq_num 1039) dirty exists 2024-09-21T18:08:59.163 INFO:tasks.rados.rados.0.smithi160.stdout:2145: left oid 152 (ObjNum 1039 snap 0 seq_num 1039) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2132: done (13 left) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2134: done (12 left) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2137: done (11 left) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2138: done (10 left) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2139: done (9 left) 2024-09-21T18:08:59.202 INFO:tasks.rados.rados.0.smithi160.stdout:2141: done (8 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2142: done (7 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2144: done (6 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2145: done (5 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2146: done (4 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2147: done (3 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2148: done (2 left) 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2149: cache_try_flush oid 273 snap -1 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2150: write (excl) oid 53 current snap is 0 2024-09-21T18:08:59.203 INFO:tasks.rados.rados.0.smithi160.stdout:2150: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-21T18:08:59.214 INFO:tasks.rados.rados.0.smithi160.stdout:2150: writing smithi16030472-53 from 696087 to 1254304 tid 1 2024-09-21T18:08:59.223 INFO:tasks.rados.rados.0.smithi160.stdout:2150: writing smithi16030472-53 from 1833755 to 2482114 tid 2 2024-09-21T18:08:59.231 INFO:tasks.rados.rados.0.smithi160.stdout:2150: writing smithi16030472-53 from 3137806 to 3496087 tid 3 2024-09-21T18:08:59.232 INFO:tasks.rados.rados.0.smithi160.stdout:2151: cache_flush oid 88 snap -1 2024-09-21T18:08:59.232 INFO:tasks.rados.rados.0.smithi160.stdout:2149: got (2) No such file or directory 2024-09-21T18:08:59.232 INFO:tasks.rados.rados.0.smithi160.stdout:2149: done (4 left) 2024-09-21T18:08:59.232 INFO:tasks.rados.rados.0.smithi160.stdout:2152: delete oid 45 current snap is 0 2024-09-21T18:08:59.256 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:59.251+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:59.279 INFO:tasks.rados.rados.0.smithi160.stdout:2140: finishing write tid 2 to smithi16030472-264 2024-09-21T18:08:59.281 INFO:tasks.rados.rados.0.smithi160.stdout:2140: finishing write tid 3 to smithi16030472-264 2024-09-21T18:08:59.323 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:59.323 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:59.336 INFO:tasks.rados.rados.0.smithi160.stdout:2152: done (4 left) 2024-09-21T18:08:59.336 INFO:tasks.rados.rados.0.smithi160.stdout:2153: write (excl) oid 278 current snap is 0 2024-09-21T18:08:59.336 INFO:tasks.rados.rados.0.smithi160.stdout:2153: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-21T18:08:59.338 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:08:59.349 INFO:tasks.rados.rados.0.smithi160.stdout:2153: writing smithi16030472-278 from 729701 to 1154769 tid 1 2024-09-21T18:08:59.357 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:08:59.352+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:08:59.361 INFO:tasks.rados.rados.0.smithi160.stdout:2153: writing smithi16030472-278 from 1667072 to 2319100 tid 2 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2153: writing smithi16030472-278 from 2909005 to 3394185 tid 3 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2154: write oid 215 current snap is 0 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2150: finishing write tid 1 to smithi16030472-53 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2150: finishing write tid 2 to smithi16030472-53 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2151: got (0) Success 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 0 (ObjNum 87 snap 0 seq_num 87) dirty exists 2024-09-21T18:08:59.372 INFO:tasks.rados.rados.0.smithi160.stdout:2154: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-21T18:08:59.385 INFO:tasks.rados.rados.0.smithi160.stdout:2154: writing smithi16030472-215 from 746508 to 1246825 tid 1 2024-09-21T18:08:59.401 INFO:tasks.rados.rados.0.smithi160.stdout:2154: writing smithi16030472-215 from 2009201 to 2804887 tid 2 2024-09-21T18:08:59.408 INFO:tasks.rados.rados.0.smithi160.stdout:2154: writing smithi16030472-215 from 3220075 to 3546508 tid 3 2024-09-21T18:08:59.408 INFO:tasks.rados.rados.0.smithi160.stdout:2151: done (5 left) 2024-09-21T18:08:59.408 INFO:tasks.rados.rados.0.smithi160.stdout:2155: read oid 489 snap -1 2024-09-21T18:08:59.409 INFO:tasks.rados.rados.0.smithi160.stdout:2155: expect (ObjNum 488 snap 0 seq_num 488) 2024-09-21T18:08:59.409 INFO:tasks.rados.rados.0.smithi160.stdout:2156: cache_flush oid 381 snap -1 2024-09-21T18:08:59.409 INFO:tasks.rados.rados.0.smithi160.stdout:2140: finishing write tid 4 to smithi16030472-264 2024-09-21T18:08:59.409 INFO:tasks.rados.rados.0.smithi160.stdout:2157: write oid 108 current snap is 0 2024-09-21T18:08:59.409 INFO:tasks.rados.rados.0.smithi160.stdout:2157: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-09-21T18:08:59.418 INFO:tasks.rados.rados.0.smithi160.stdout:2157: writing smithi16030472-108 from 763315 to 1338881 tid 1 2024-09-21T18:08:59.428 INFO:tasks.rados.rados.0.smithi160.stdout:2157: writing smithi16030472-108 from 2067683 to 2723380 tid 2 2024-09-21T18:08:59.433 INFO:tasks.rados.rados.0.smithi160.stdout:2157: writing smithi16030472-108 from 3247498 to 3563315 tid 3 2024-09-21T18:08:59.433 INFO:tasks.rados.rados.0.smithi160.stdout:2158: write (excl) oid 402 current snap is 0 2024-09-21T18:08:59.433 INFO:tasks.rados.rados.0.smithi160.stdout:2158: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-21T18:08:59.442 INFO:tasks.rados.rados.0.smithi160.stdout:2158: writing smithi16030472-402 from 780122 to 1430937 tid 1 2024-09-21T18:08:59.450 INFO:tasks.rados.rados.0.smithi160.stdout:2158: writing smithi16030472-402 from 2126165 to 2925520 tid 2 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2159: cache_evict oid 435 snap -1 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2156: got (0) Success 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2156: done (9 left) 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2160: cache_try_flush oid 122 snap -1 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2161: read oid 454 snap -1 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2161: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2162: cache_evict oid 291 snap -1 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2163: write oid 133 current snap is 0 2024-09-21T18:08:59.451 INFO:tasks.rados.rados.0.smithi160.stdout:2163: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-21T18:08:59.458 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:08:59.452+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:08:59.459 INFO:tasks.rados.rados.0.smithi160.stdout:2163: writing smithi16030472-133 from 796929 to 1522993 tid 1 2024-09-21T18:08:59.466 INFO:tasks.rados.rados.0.smithi160.stdout:2163: writing smithi16030472-133 from 2184647 to 2844013 tid 2 2024-09-21T18:08:59.470 INFO:tasks.rados.rados.0.smithi160.stdout:2163: writing smithi16030472-133 from 3302344 to 3596929 tid 3 2024-09-21T18:08:59.471 INFO:tasks.rados.rados.0.smithi160.stdout:2164: copy_from oid 82 from oid 72 current snap is 0 2024-09-21T18:08:59.471 INFO:tasks.rados.rados.0.smithi160.stdout:2165: cache_flush oid 398 snap -1 2024-09-21T18:08:59.471 INFO:tasks.rados.rados.0.smithi160.stdout:2166: read oid 349 snap -1 2024-09-21T18:08:59.471 INFO:tasks.rados.rados.0.smithi160.stdout:2166: expect deleted 2024-09-21T18:08:59.471 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:2160: got (0) Success 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 122 v 0 (ObjNum 742 snap 0 seq_num 742) dirty exists 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:2153: finishing write tid 1 to smithi16030472-278 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:2160: done (15 left) 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:2167: cache_flush oid 428 snap -1 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.479 INFO:tasks.rados.rados.0.smithi160.stdout:2153: finishing write tid 2 to smithi16030472-278 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: finishing write tid 3 to smithi16030472-53 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: finishing write tid 4 to smithi16030472-53 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: finishing write tid 5 to smithi16030472-53 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: oid 53 updating version 0 to 722 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: oid 53 version 722 is already newer than 721 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: oid 53 updating version 722 to 724 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: oid 53 version 724 is already newer than 723 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 53 v 724 (ObjNum 1041 snap 0 seq_num 1041) dirty exists 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2150: left oid 53 (ObjNum 1041 snap 0 seq_num 1041) 2024-09-21T18:08:59.505 INFO:tasks.rados.rados.0.smithi160.stdout:2154: finishing write tid 1 to smithi16030472-215 2024-09-21T18:08:59.506 INFO:tasks.rados.rados.0.smithi160.stdout:2150: done (15 left) 2024-09-21T18:08:59.506 INFO:tasks.rados.rados.0.smithi160.stdout:2168: write oid 195 current snap is 0 2024-09-21T18:08:59.506 INFO:tasks.rados.rados.0.smithi160.stdout:2168: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-21T18:08:59.507 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:59.507 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:59.509 INFO:tasks.rados.rados.0.smithi160.stdout:2168: writing smithi16030472-195 from 413736 to 815049 tid 1 2024-09-21T18:08:59.515 INFO:tasks.rados.rados.0.smithi160.stdout:2168: writing smithi16030472-195 from 1326776 to 1729800 tid 2 2024-09-21T18:08:59.521 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:08:59.522 INFO:tasks.rados.rados.0.smithi160.stdout:2168: writing smithi16030472-195 from 2413414 to 2871307 tid 3 2024-09-21T18:08:59.526 INFO:tasks.rados.rados.0.smithi160.stdout:2168: writing smithi16030472-195 from 3438716 to 3613736 tid 4 2024-09-21T18:08:59.526 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.526 INFO:tasks.rados.rados.0.smithi160.stdout:2153: finishing write tid 3 to smithi16030472-278 2024-09-21T18:08:59.552 INFO:tasks.rados.rados.0.smithi160.stdout:2140: finishing write tid 5 to smithi16030472-264 2024-09-21T18:08:59.552 INFO:tasks.rados.rados.0.smithi160.stdout:2140: oid 264 updating version 0 to 755 2024-09-21T18:08:59.552 INFO:tasks.rados.rados.0.smithi160.stdout:2140: oid 264 version 755 is already newer than 753 2024-09-21T18:08:59.552 INFO:tasks.rados.rados.0.smithi160.stdout:2140: oid 264 version 755 is already newer than 754 2024-09-21T18:08:59.552 INFO:tasks.rados.rados.0.smithi160.stdout:2140: oid 264 version 755 is already newer than 701 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 755 (ObjNum 1038 snap 0 seq_num 1038) dirty exists 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2140: left oid 264 (ObjNum 1038 snap 0 seq_num 1038) 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2143: got (0) Success 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 279 v 0 (ObjNum 536 snap 0 seq_num 536) dirty exists 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2140: done (15 left) 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2143: done (14 left) 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2169: cache_evict oid 497 snap -1 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2170: write (excl) oid 499 current snap is 0 2024-09-21T18:08:59.553 INFO:tasks.rados.rados.0.smithi160.stdout:2170: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-21T18:08:59.558 INFO:tasks.rados.rados.0.smithi160.stdout:2170: writing smithi16030472-499 from 430543 to 907105 tid 1 2024-09-21T18:08:59.566 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:08:59.563+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:08:59.567 INFO:tasks.rados.rados.0.smithi160.stdout:2170: writing smithi16030472-499 from 1385258 to 2048293 tid 2 2024-09-21T18:08:59.577 INFO:tasks.rados.rados.0.smithi160.stdout:2170: writing smithi16030472-499 from 2840837 to 3510002 tid 3 2024-09-21T18:08:59.577 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:08:59.605 INFO:tasks.rados.rados.0.smithi160.stdout:2159: got (16) Device or resource busy 2024-09-21T18:08:59.605 INFO:tasks.rados.rados.0.smithi160.stdout:2159: done (15 left) 2024-09-21T18:08:59.605 INFO:tasks.rados.rados.0.smithi160.stdout:2171: write oid 229 current snap is 0 2024-09-21T18:08:59.605 INFO:tasks.rados.rados.0.smithi160.stdout:2171: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-21T18:08:59.615 INFO:tasks.rados.rados.0.smithi160.stdout:2171: writing smithi16030472-229 from 447350 to 1115514 tid 1 2024-09-21T18:08:59.621 INFO:tasks.rados.rados.0.smithi160.stdout:2171: writing smithi16030472-229 from 1560093 to 1966786 tid 2 2024-09-21T18:08:59.628 INFO:tasks.rados.rados.0.smithi160.stdout:2171: writing smithi16030472-229 from 2584613 to 3065050 tid 3 2024-09-21T18:08:59.629 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.646 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:59.646 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:59.659 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:59.653+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:59.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout:2167: got (0) Success 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 428 v 0 (ObjNum 427 snap 0 seq_num 427) dirty exists 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout:2167: done (15 left) 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout:2172: read oid 193 snap -1 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout:2172: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:08:59.707 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.745 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:08:59.746 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:08:59.760 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:08:59.755+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:08:59.762 INFO:tasks.rados.rados.0.smithi160.stdout:2157: finishing write tid 1 to smithi16030472-108 2024-09-21T18:08:59.762 INFO:tasks.rados.rados.0.smithi160.stdout:2157: finishing write tid 2 to smithi16030472-108 2024-09-21T18:08:59.762 INFO:tasks.rados.rados.0.smithi160.stdout:2157: finishing write tid 3 to smithi16030472-108 2024-09-21T18:08:59.766 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:08:59.767 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: finishing write tid 4 to smithi16030472-278 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: finishing write tid 5 to smithi16030472-278 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: oid 278 updating version 0 to 751 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: oid 278 updating version 751 to 752 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: oid 278 updating version 752 to 754 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: oid 278 version 754 is already newer than 753 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 278 v 754 (ObjNum 1043 snap 0 seq_num 1043) dirty exists 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: left oid 278 (ObjNum 1043 snap 0 seq_num 1043) 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2153: done (15 left) 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2173: write (excl) oid 3 current snap is 0 2024-09-21T18:08:59.786 INFO:tasks.rados.rados.0.smithi160.stdout:2173: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-09-21T18:08:59.787 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:08:59.787 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:08:59.797 INFO:tasks.rados.rados.0.smithi160.stdout:2173: writing smithi16030472-3 from 464157 to 1207570 tid 1 2024-09-21T18:08:59.807 INFO:tasks.rados.rados.0.smithi160.stdout:2173: writing smithi16030472-3 from 1618575 to 2285279 tid 2 2024-09-21T18:08:59.808 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:08:59.808 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2173: writing smithi16030472-3 from 3012036 to 3420098 tid 3 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: finishing write tid 4 to smithi16030472-108 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: finishing write tid 5 to smithi16030472-108 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: oid 108 updating version 0 to 757 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: oid 108 updating version 757 to 758 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: oid 108 version 758 is already newer than 756 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:2157: oid 108 updating version 758 to 759 2024-09-21T18:08:59.819 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 108 v 759 (ObjNum 1045 snap 0 seq_num 1045) dirty exists 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2157: left oid 108 (ObjNum 1045 snap 0 seq_num 1045) 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2162: got (16) Device or resource busy 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2157: done (15 left) 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2162: done (14 left) 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2174: write (excl) oid 97 current snap is 0 2024-09-21T18:08:59.820 INFO:tasks.rados.rados.0.smithi160.stdout:2174: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-21T18:08:59.825 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:08:59.825 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:08:59.827 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:08:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:08:59.828 INFO:tasks.rados.rados.0.smithi160.stdout:2174: writing smithi16030472-97 from 480964 to 899626 tid 1 2024-09-21T18:08:59.838 INFO:tasks.rados.rados.0.smithi160.stdout:2174: writing smithi16030472-97 from 1560704 to 1971066 tid 2 2024-09-21T18:08:59.841 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:08:59.848 INFO:tasks.rados.rados.0.smithi160.stdout:2174: writing smithi16030472-97 from 2523106 to 3142440 tid 3 2024-09-21T18:08:59.848 INFO:tasks.rados.rados.0.smithi160.stdout:2175: write (excl) oid 414 current snap is 0 2024-09-21T18:08:59.848 INFO:tasks.rados.rados.0.smithi160.stdout:2175: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-21T18:08:59.850 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:08:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:08:59.856 INFO:tasks.rados.rados.0.smithi160.stdout:2175: writing smithi16030472-414 from 497771 to 991682 tid 1 2024-09-21T18:08:59.861 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:08:59.856+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:08:59.864 INFO:tasks.rados.rados.0.smithi160.stdout:2175: writing smithi16030472-414 from 1619186 to 2173206 tid 2 2024-09-21T18:08:59.872 INFO:tasks.rados.rados.0.smithi160.stdout:2175: writing smithi16030472-414 from 2950529 to 3381135 tid 3 2024-09-21T18:08:59.872 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:08:59.872 INFO:tasks.rados.rados.0.smithi160.stdout:2161: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:08:59.872 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:08:59.872 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:08:59.872 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:08:59.873 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:08:59.890 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:08:59.890 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:08:59.910 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:08:59.911 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:08:59.931 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:08:59.931 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:08:59.962 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:08:59.957+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:08:59.964 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.008 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:00.009 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:00.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:00.038 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:00.038 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:00.050 INFO:tasks.rados.rados.0.smithi160.stdout:2164: finishing copy_from to smithi16030472-82 2024-09-21T18:09:00.050 INFO:tasks.rados.rados.0.smithi160.stdout:2164: got expected ENOENT (src dne) 2024-09-21T18:09:00.050 INFO:tasks.rados.rados.0.smithi160.stdout:2164: finishing copy_from racing read to smithi16030472-82 2024-09-21T18:09:00.050 INFO:tasks.rados.rados.0.smithi160.stdout:2158: finishing write tid 1 to smithi16030472-402 2024-09-21T18:09:00.050 INFO:tasks.rados.rados.0.smithi160.stdout:2158: finishing write tid 2 to smithi16030472-402 2024-09-21T18:09:00.051 INFO:tasks.rados.rados.0.smithi160.stdout:2154: finishing write tid 2 to smithi16030472-215 2024-09-21T18:09:00.051 INFO:tasks.rados.rados.0.smithi160.stdout:2154: finishing write tid 3 to smithi16030472-215 2024-09-21T18:09:00.051 INFO:tasks.rados.rados.0.smithi160.stdout:2154: finishing write tid 4 to smithi16030472-215 2024-09-21T18:09:00.051 INFO:tasks.rados.rados.0.smithi160.stdout:2155: expect (ObjNum 488 snap 0 seq_num 488) 2024-09-21T18:09:00.063 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:00.060+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.066 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:00.066 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:00.103 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:2154: finishing write tid 5 to smithi16030472-215 2024-09-21T18:09:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:2154: oid 215 updating version 0 to 728 2024-09-21T18:09:00.111 INFO:tasks.rados.rados.0.smithi160.stdout:2154: oid 215 updating version 728 to 736 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2154: oid 215 version 736 is already newer than 734 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2154: oid 215 version 736 is already newer than 735 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 215 v 736 (ObjNum 1044 snap 0 seq_num 1044) dirty exists 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2154: left oid 215 (ObjNum 1044 snap 0 seq_num 1044) 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: finishing write tid 1 to smithi16030472-133 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2173: finishing write tid 1 to smithi16030472-3 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2173: finishing write tid 2 to smithi16030472-3 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: finishing write tid 2 to smithi16030472-133 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: finishing write tid 3 to smithi16030472-133 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: finishing write tid 4 to smithi16030472-133 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: finishing write tid 5 to smithi16030472-133 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: oid 133 updating version 0 to 737 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: oid 133 updating version 737 to 738 2024-09-21T18:09:00.112 INFO:tasks.rados.rados.0.smithi160.stdout:2163: oid 133 updating version 738 to 739 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2163: oid 133 updating version 739 to 740 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 133 v 740 (ObjNum 1047 snap 0 seq_num 1047) dirty exists 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2163: left oid 133 (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: finishing write tid 3 to smithi16030472-402 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: finishing write tid 4 to smithi16030472-402 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: oid 402 updating version 0 to 762 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: oid 402 version 762 is already newer than 761 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: oid 402 version 762 is already newer than 760 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 402 v 762 (ObjNum 1046 snap 0 seq_num 1046) dirty exists 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2158: left oid 402 (ObjNum 1046 snap 0 seq_num 1046) 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2170: finishing write tid 1 to smithi16030472-499 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2170: finishing write tid 2 to smithi16030472-499 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2170: finishing write tid 3 to smithi16030472-499 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2170: finishing write tid 4 to smithi16030472-499 2024-09-21T18:09:00.113 INFO:tasks.rados.rados.0.smithi160.stdout:2154: done (15 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2155: done (14 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2158: done (13 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2161: done (12 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2163: done (11 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2164: done (10 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2166: done (9 left) 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2176: delete oid 157 current snap is 0 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 1 to smithi16030472-195 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 2 to smithi16030472-195 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 3 to smithi16030472-195 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 4 to smithi16030472-195 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 5 to smithi16030472-195 2024-09-21T18:09:00.114 INFO:tasks.rados.rados.0.smithi160.stdout:2168: finishing write tid 6 to smithi16030472-195 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: oid 195 updating version 0 to 758 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: oid 195 version 758 is already newer than 755 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: oid 195 updating version 758 to 759 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: oid 195 version 759 is already newer than 756 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: oid 195 version 759 is already newer than 757 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 759 (ObjNum 1048 snap 0 seq_num 1048) dirty exists 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2168: left oid 195 (ObjNum 1048 snap 0 seq_num 1048) 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2169: got (0) Success 2024-09-21T18:09:00.115 INFO:tasks.rados.rados.0.smithi160.stdout:2175: finishing write tid 1 to smithi16030472-414 2024-09-21T18:09:00.129 INFO:tasks.rados.rados.0.smithi160.stdout:2170: finishing write tid 5 to smithi16030472-499 2024-09-21T18:09:00.129 INFO:tasks.rados.rados.0.smithi160.stdout:2170: oid 499 updating version 0 to 764 2024-09-21T18:09:00.130 INFO:tasks.rados.rados.0.smithi160.stdout:2170: oid 499 updating version 764 to 766 2024-09-21T18:09:00.130 INFO:tasks.rados.rados.0.smithi160.stdout:2170: oid 499 version 766 is already newer than 763 2024-09-21T18:09:00.130 INFO:tasks.rados.rados.0.smithi160.stdout:2170: oid 499 version 766 is already newer than 765 2024-09-21T18:09:00.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 766 (ObjNum 1049 snap 0 seq_num 1049) dirty exists 2024-09-21T18:09:00.130 INFO:tasks.rados.rados.0.smithi160.stdout:2170: left oid 499 (ObjNum 1049 snap 0 seq_num 1049) 2024-09-21T18:09:00.135 INFO:tasks.rados.rados.0.smithi160.stdout:2172: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:09:00.137 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:00.138 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:00.146 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:00.147 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:00.147 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:00.147 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:00.147 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:00.163 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:00.164 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:00.157+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:00.170 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1339329 2024-09-21T18:09:00.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:09:00.240 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:00.257 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:00.251+0000 7f7283507640 -1 osd.14 89 handle_osd_map injecting map crc failure 2024-09-21T18:09:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:00.262 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:00.264 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:00.260+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2173: finishing write tid 3 to smithi16030472-3 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2173: finishing write tid 4 to smithi16030472-3 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2175: finishing write tid 2 to smithi16030472-414 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2175: finishing write tid 3 to smithi16030472-414 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2175: finishing write tid 4 to smithi16030472-414 2024-09-21T18:09:00.281 INFO:tasks.rados.rados.0.smithi160.stdout:2175: finishing write tid 5 to smithi16030472-414 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2175: oid 414 updating version 0 to 763 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2175: oid 414 version 763 is already newer than 760 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2175: oid 414 version 763 is already newer than 762 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2175: oid 414 version 763 is already newer than 761 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 763 (ObjNum 1053 snap 0 seq_num 1053) dirty exists 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2175: left oid 414 (ObjNum 1053 snap 0 seq_num 1053) 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: finishing write tid 5 to smithi16030472-3 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: oid 3 updating version 0 to 725 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: oid 3 updating version 725 to 728 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: oid 3 version 728 is already newer than 727 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: oid 3 version 728 is already newer than 726 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 3 v 728 (ObjNum 1051 snap 0 seq_num 1051) dirty exists 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2173: left oid 3 (ObjNum 1051 snap 0 seq_num 1051) 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2174: finishing write tid 1 to smithi16030472-97 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2168: done (9 left) 2024-09-21T18:09:00.282 INFO:tasks.rados.rados.0.smithi160.stdout:2169: done (8 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2170: done (7 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2172: done (6 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2173: done (5 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2175: done (4 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2176: done (3 left) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2177: read oid 354 snap -1 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2177: expect (ObjNum 476 snap 0 seq_num 476) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2178: cache_try_flush oid 41 snap -1 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2174: finishing write tid 2 to smithi16030472-97 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2179: read oid 462 snap -1 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2179: expect (ObjNum 632 snap 0 seq_num 632) 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2180: read oid 73 snap -1 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2180: expect deleted 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2181: copy_from oid 363 from oid 86 current snap is 0 2024-09-21T18:09:00.283 INFO:tasks.rados.rados.0.smithi160.stdout:2174: finishing write tid 3 to smithi16030472-97 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2171: finishing write tid 1 to smithi16030472-229 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2171: finishing write tid 2 to smithi16030472-229 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2165: got (0) Success 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 398 v 0 (ObjNum 875 snap 0 seq_num 875) dirty exists 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: finishing write tid 4 to smithi16030472-97 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: finishing write tid 5 to smithi16030472-97 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: oid 97 updating version 0 to 742 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: oid 97 updating version 742 to 743 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: oid 97 version 743 is already newer than 741 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: oid 97 updating version 743 to 744 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 97 v 744 (ObjNum 1052 snap 0 seq_num 1052) dirty exists 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: left oid 97 (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2165: done (7 left) 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2174: done (6 left) 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2182: copy_from oid 460 from oid 81 current snap is 0 2024-09-21T18:09:00.284 INFO:tasks.rados.rados.0.smithi160.stdout:2183: read oid 418 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2183: expect (ObjNum 417 snap 0 seq_num 417) 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2184: cache_evict oid 241 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2185: cache_evict oid 305 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2186: copy_from oid 333 from oid 58 current snap is 0 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2187: cache_flush oid 75 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2188: read oid 128 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2188: expect deleted 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2189: cache_try_flush oid 118 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2190: read oid 446 snap -1 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2190: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:09:00.285 INFO:tasks.rados.rados.0.smithi160.stdout:2191: delete oid 44 current snap is 0 2024-09-21T18:09:00.286 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:00.286 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:00.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.444357579, 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.032715284999995, 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:09:00.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:00.295 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.029884413000005, 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.025911504999996, 2024-09-21T18:09:00.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.022794554000001, 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.303 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:09:00.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.020863550000001, 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.307 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:09:00.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.017756790999996, 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:00.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:09:00.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.014599106999995, 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:09:00.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.012366056000005, 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:09:00.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:09:00.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.010114383000001, 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:09:00.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:09:00.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:09:00.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 94.917536795000004, 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:09:00.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:09:00.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 94.140867419000003, 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.131359834999998, 2024-09-21T18:09:00.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:09:00.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:09:00.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:09:00.339 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.024529856000001, 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:00.343 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 80.691158303999998, 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:09:00.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 75.919166834999999, 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:09:00.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:09:00.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 66.037018380000006, 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:09:00.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:09:00.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 41.496893931999999, 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:09:00.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:09:00.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 24.186935471999998, 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:09:00.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:09:00.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:09:00.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.942533161, 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:00.364 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:09:00.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:09:00.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:00.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:00.368 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:00.369 INFO:tasks.rados.rados.0.smithi160.stdout:2171: finishing write tid 3 to smithi16030472-229 2024-09-21T18:09:00.369 INFO:tasks.rados.rados.0.smithi160.stdout:2171: finishing write tid 4 to smithi16030472-229 2024-09-21T18:09:00.370 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:00.370 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:00.365+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: finishing write tid 5 to smithi16030472-229 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: oid 229 updating version 0 to 769 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: oid 229 version 769 is already newer than 768 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: oid 229 version 769 is already newer than 767 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: oid 229 updating version 769 to 770 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 229 v 770 (ObjNum 1050 snap 0 seq_num 1050) dirty exists 2024-09-21T18:09:00.431 INFO:tasks.rados.rados.0.smithi160.stdout:2171: left oid 229 (ObjNum 1050 snap 0 seq_num 1050) 2024-09-21T18:09:00.470 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:00.464+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:00.483 INFO:tasks.rados.rados.0.smithi160.stdout:2187: got (0) Success 2024-09-21T18:09:00.483 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 75 v 0 (ObjNum 129 snap 0 seq_num 3126288936) clean dne 2024-09-21T18:09:00.483 INFO:tasks.rados.rados.0.smithi160.stdout:2189: got (2) No such file or directory 2024-09-21T18:09:00.486 INFO:tasks.rados.rados.0.smithi160.stdout:2179: expect (ObjNum 632 snap 0 seq_num 632) 2024-09-21T18:09:00.498 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:00.498 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:00.498 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:00.498 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:00.513 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:00.571 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:00.565+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:00.628 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1281247 2024-09-21T18:09:00.631 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:00.631 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:00.631 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:00.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:00.656 INFO:tasks.rados.rados.0.smithi160.stdout:2177: expect (ObjNum 476 snap 0 seq_num 476) 2024-09-21T18:09:00.656 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.672 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:00.666+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:00.676 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.717 INFO:tasks.rados.rados.0.smithi160.stdout:2184: got (16) Device or resource busy 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2185: got (16) Device or resource busy 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2178: got (0) Success 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3132994040) clean dne 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2186: finishing copy_from racing read to smithi16030472-333 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2182: finishing copy_from racing read to smithi16030472-460 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2181: finishing copy_from to smithi16030472-363 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2181: got expected ENOENT (src dne) 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2181: finishing copy_from racing read to smithi16030472-363 2024-09-21T18:09:00.718 INFO:tasks.rados.rados.0.smithi160.stdout:2190: expect (ObjNum 496 snap 0 seq_num 496) 2024-09-21T18:09:00.719 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.761 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:00.773 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:00.765+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:00.785 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:00.786 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:00.786 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:00.786 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:09:00.786 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:00.787 INFO:tasks.rados.rados.0.smithi160.stdout:2186: finishing copy_from to smithi16030472-333 2024-09-21T18:09:00.787 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 31 (ObjNum 57 snap 0 seq_num 57) dirty exists 2024-09-21T18:09:00.787 INFO:tasks.rados.rados.0.smithi160.stdout:2182: finishing copy_from to smithi16030472-460 2024-09-21T18:09:00.787 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 460 v 499 (ObjNum 786 snap 0 seq_num 786) dirty exists 2024-09-21T18:09:00.787 INFO:tasks.rados.rados.0.smithi160.stdout:2183: expect (ObjNum 417 snap 0 seq_num 417) 2024-09-21T18:09:00.800 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:00.885 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:00.881+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:00.890 INFO:tasks.rados.rados.0.smithi160.stdout:2171: done (15 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2177: done (14 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2178: done (13 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2179: done (12 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2180: done (11 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2181: done (10 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2182: done (9 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2183: done (8 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2184: done (7 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2185: done (6 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2186: done (5 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2187: done (4 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2188: done (3 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2189: done (2 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2190: done (1 left) 2024-09-21T18:09:00.891 INFO:tasks.rados.rados.0.smithi160.stdout:2191: done (0 left) 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2192: read oid 267 snap -1 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2192: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2193: copy_from oid 204 from oid 214 current snap is 0 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2194: read oid 296 snap -1 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2194: expect deleted 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2195: copy_from oid 23 from oid 71 current snap is 0 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2196: cache_flush oid 115 snap -1 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2197: write oid 51 current snap is 0 2024-09-21T18:09:00.892 INFO:tasks.rados.rados.0.smithi160.stdout:2197: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-21T18:09:00.901 INFO:tasks.rados.rados.0.smithi160.stdout:2197: writing smithi16030472-51 from 548192 to 1267850 tid 1 2024-09-21T18:09:00.907 INFO:tasks.rados.rados.0.smithi160.stdout:2197: writing smithi16030472-51 from 1678279 to 2095979 tid 2 2024-09-21T18:09:00.917 INFO:tasks.rados.rados.0.smithi160.stdout:2197: writing smithi16030472-51 from 2516445 to 3297220 tid 3 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2198: cache_flush oid 424 snap -1 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2199: copy_from oid 397 from oid 199 current snap is 0 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2194: done (7 left) 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2200: cache_try_flush oid 157 snap -1 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2201: cache_evict oid 443 snap -1 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2202: cache_evict oid 150 snap -1 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2203: read oid 116 snap -1 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2203: expect (ObjNum 869 snap 0 seq_num 869) 2024-09-21T18:09:00.918 INFO:tasks.rados.rados.0.smithi160.stdout:2204: delete oid 261 current snap is 0 2024-09-21T18:09:00.960 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:00.960 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:00.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.960 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6209 4.1 4:a920e1b8:::smithi16030472-115:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.890122+0000", 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.068374530000000003, 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.068422779000000003, 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6209 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890122+0000", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890122+0000", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890123+0000", 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3599999999999999e-06 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890129+0000", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9429999999999997e-06 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890145+0000", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5417000000000002e-05 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890162+0000", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7255999999999999e-05 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:00.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890793+0000", 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00063077399999999996 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.891016+0000", 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00022298000000000001 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:882 4.1 4.415310bd (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.890204+0000", 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.068292700999999997, 2024-09-21T18:09:00.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.068444163000000002, 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890204+0000", 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890204+0000", 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890206+0000", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890215+0000", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3100000000000006e-06 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890235+0000", 2024-09-21T18:09:00.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9720000000000001e-05 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.890249+0000", 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4056000000000001e-05 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6210 4.1 4.36c9f149 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.900314+0000", 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.058182794000000003, 2024-09-21T18:09:00.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.058406352000000002, 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6210 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900314+0000", 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900314+0000", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900317+0000", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8020000000000002e-06 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900865+0000", 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00054774000000000001 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900891+0000", 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6333999999999999e-05 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.900904+0000", 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3036e-05 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6211 4.1 4.36c9f149 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.970 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.907034+0000", 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.051462586999999997, 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.051757574000000001, 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6211 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.971 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907034+0000", 2024-09-21T18:09:00.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907034+0000", 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907036+0000", 2024-09-21T18:09:00.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6139999999999999e-06 2024-09-21T18:09:00.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907299+0000", 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026244099999999998 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907318+0000", 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9329000000000001e-05 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.907333+0000", 2024-09-21T18:09:00.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.465e-05 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6212 4.1 4.36c9f149 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.916770+0000", 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.041726375000000003, 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.042092698999999997, 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.976 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6212 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.916770+0000", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.916770+0000", 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.916774+0000", 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8519999999999997e-06 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917386+0000", 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00061230599999999996 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917409+0000", 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3339000000000001e-05 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917426+0000", 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6878e-05 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6213 4.1 4.36c9f149 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.917454+0000", 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.041042642999999997, 2024-09-21T18:09:00.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.041479348999999999, 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6213 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917454+0000", 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917454+0000", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917455+0000", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.514e-06 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917463+0000", 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6690000000000006e-06 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917476+0000", 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2955e-05 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917488+0000", 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1812e-05 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.982 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6214 4.1 4.36c9f149 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e90)", 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:00.917502+0000", 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.040994083000000001, 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.041512291999999999, 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6214 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917502+0000", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917502+0000", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917503+0000", 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.145e-06 2024-09-21T18:09:00.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917509+0000", 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.1510000000000003e-06 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:00.917523+0000", 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.416e-05 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 7 2024-09-21T18:09:00.985 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:00.987 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:00.987 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:00.984+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:01.088 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:01.086+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:01.093 INFO:tasks.rados.rados.0.smithi160.stdout:2202: got (16) Device or resource busy 2024-09-21T18:09:01.093 INFO:tasks.rados.rados.0.smithi160.stdout:2199: finishing copy_from to smithi16030472-397 2024-09-21T18:09:01.094 INFO:tasks.rados.rados.0.smithi160.stdout:2199: got expected ENOENT (src dne) 2024-09-21T18:09:01.094 INFO:tasks.rados.rados.0.smithi160.stdout:2199: finishing copy_from racing read to smithi16030472-397 2024-09-21T18:09:01.097 INFO:tasks.rados.rados.0.smithi160.stdout:2200: got (0) Success 2024-09-21T18:09:01.097 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:09:01.111 INFO:tasks.rados.rados.0.smithi160.stdout:2192: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:09:01.135 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:01.157 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:01.157 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:01.157 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:01.157 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:01.157 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:01.171 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:01.173 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:09:01.189 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:01.184+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:01.201 INFO:tasks.rados.rados.0.smithi160.stdout:2201: got (2) No such file or directory 2024-09-21T18:09:01.201 INFO:tasks.rados.rados.0.smithi160.stdout:2193: finishing copy_from to smithi16030472-204 2024-09-21T18:09:01.201 INFO:tasks.rados.rados.0.smithi160.stdout:2193: got expected ENOENT (src dne) 2024-09-21T18:09:01.201 INFO:tasks.rados.rados.0.smithi160.stdout:2193: finishing copy_from racing read to smithi16030472-204 2024-09-21T18:09:01.224 INFO:tasks.rados.rados.0.smithi160.stdout:2197: finishing write tid 1 to smithi16030472-51 2024-09-21T18:09:01.225 INFO:tasks.rados.rados.0.smithi160.stdout:2196: got (0) Success 2024-09-21T18:09:01.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 115 v 0 (ObjNum 895 snap 0 seq_num 895) dirty exists 2024-09-21T18:09:01.225 INFO:tasks.rados.rados.0.smithi160.stdout:2197: finishing write tid 2 to smithi16030472-51 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: finishing write tid 3 to smithi16030472-51 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: finishing write tid 4 to smithi16030472-51 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: finishing write tid 5 to smithi16030472-51 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: oid 51 updating version 0 to 729 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: oid 51 updating version 729 to 730 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: oid 51 updating version 730 to 731 2024-09-21T18:09:01.226 INFO:tasks.rados.rados.0.smithi160.stdout:2197: oid 51 updating version 731 to 732 2024-09-21T18:09:01.227 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 732 (ObjNum 1056 snap 0 seq_num 1056) dirty exists 2024-09-21T18:09:01.227 INFO:tasks.rados.rados.0.smithi160.stdout:2197: left oid 51 (ObjNum 1056 snap 0 seq_num 1056) 2024-09-21T18:09:01.227 INFO:tasks.rados.rados.0.smithi160.stdout:2203: expect (ObjNum 869 snap 0 seq_num 869) 2024-09-21T18:09:01.289 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.14213279000001, 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.290 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:09:01.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:09:01.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:01.296 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.14208648100001, 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:01.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:09:01.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:09:01.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:09:01.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:01.302 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.139990568, 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:01.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:09:01.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:09:01.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.13932745300001, 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.309 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:09:01.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:09:01.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:01.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:01.315 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.139290954, 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:09:01.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.005652782, 2024-09-21T18:09:01.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:01.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:09:01.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.004940991, 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.328 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:01.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:09:01.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:09:01.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 108.00490342800001, 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:09:01.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:09:01.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:09:01.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.207517429000006, 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:09:01.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:09:01.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.207322883000003, 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.345 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.207036494999997, 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.349 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:09:01.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:09:01.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.352 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.207020415000002, 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:09:01.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:09:01.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:09:01.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.357 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.357 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.358 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.358 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:01.358 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.178093163, 2024-09-21T18:09:01.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:01.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:09:01.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.362 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.171874087999996, 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.363 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:09:01.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:09:01.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:09:01.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.166772492999996, 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.367 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:09:01.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:09:01.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.166438545000005, 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.371 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:09:01.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:09:01.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.166418066000006, 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:09:01.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:09:01.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.378 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.080089538999999, 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:09:01.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:09:01.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.382 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.071629353000006, 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.065701648000001, 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:09:01.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.391 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.391 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.391 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.391 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:01.392 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:01.288+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:01.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2450035 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1703096 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:2195: finishing copy_from racing read to smithi16030472-23 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:2195: finishing copy_from to smithi16030472-23 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 23 v 337 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:2198: got (0) Success 2024-09-21T18:09:01.393 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:09:01.394 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:01.388+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2192: done (11 left) 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2193: done (10 left) 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2195: done (9 left) 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2196: done (8 left) 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2197: done (7 left) 2024-09-21T18:09:01.424 INFO:tasks.rados.rados.0.smithi160.stdout:2198: done (6 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2199: done (5 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2200: done (4 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2201: done (3 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2202: done (2 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2203: done (1 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2204: done (0 left) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2205: cache_evict oid 166 snap -1 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2206: cache_flush oid 164 snap -1 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2207: read oid 91 snap -1 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2207: expect (ObjNum 956 snap 0 seq_num 956) 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2208: write oid 172 current snap is 0 2024-09-21T18:09:01.425 INFO:tasks.rados.rados.0.smithi160.stdout:2208: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-21T18:09:01.439 INFO:tasks.rados.rados.0.smithi160.stdout:2208: writing smithi16030472-172 from 581806 to 1168315 tid 1 2024-09-21T18:09:01.454 INFO:tasks.rados.rados.0.smithi160.stdout:2208: writing smithi16030472-172 from 1911596 to 2332965 tid 2 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2208: writing smithi16030472-172 from 3087644 to 3490963 tid 3 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2209: read oid 409 snap -1 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2209: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2210: cache_flush oid 183 snap -1 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2211: write (excl) oid 49 current snap is 0 2024-09-21T18:09:01.470 INFO:tasks.rados.rados.0.smithi160.stdout:2211: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-09-21T18:09:01.484 INFO:tasks.rados.rados.0.smithi160.stdout:2211: writing smithi16030472-49 from 598613 to 1260371 tid 1 2024-09-21T18:09:01.493 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:01.485+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:01.495 INFO:tasks.rados.rados.0.smithi160.stdout:2211: writing smithi16030472-49 from 1970078 to 2535105 tid 2 2024-09-21T18:09:01.505 INFO:tasks.rados.rados.0.smithi160.stdout:2211: writing smithi16030472-49 from 3115067 to 3729658 tid 3 2024-09-21T18:09:01.505 INFO:tasks.rados.rados.0.smithi160.stdout:2212: cache_try_flush oid 7 snap -1 2024-09-21T18:09:01.505 INFO:tasks.rados.rados.0.smithi160.stdout:2205: got (16) Device or resource busy 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2211: finishing write tid 1 to smithi16030472-49 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2211: finishing write tid 2 to smithi16030472-49 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2205: done (7 left) 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2213: cache_evict oid 16 snap -1 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2214: copy_from oid 173 from oid 489 current snap is 0 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2215: cache_try_flush oid 381 snap -1 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2216: cache_try_flush oid 105 snap -1 2024-09-21T18:09:01.506 INFO:tasks.rados.rados.0.smithi160.stdout:2217: delete oid 37 current snap is 0 2024-09-21T18:09:01.551 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6241 4.2 4.1f2af19e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:01.504325+0000", 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.045996495999999998, 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.046038849, 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6241 2024-09-21T18:09:01.552 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:01.504325+0000", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:01.504325+0000", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:01.504327+0000", 2024-09-21T18:09:01.553 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2189999999999998e-06 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:01.504711+0000", 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038338899999999998 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:01.504725+0000", 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4100000000000001e-05 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.554 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:01.555 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 1 2024-09-21T18:09:01.555 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:01.565 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:01.594 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:01.590+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:01.605 INFO:tasks.rados.rados.0.smithi160.stdout:2210: got (0) Success 2024-09-21T18:09:01.606 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 0 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:09:01.606 INFO:tasks.rados.rados.0.smithi160.stdout:2215: got (0) Success 2024-09-21T18:09:01.606 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 0 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T18:09:01.694 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:01.689+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:01.706 INFO:tasks.rados.rados.0.smithi160.stdout:2212: got (0) Success 2024-09-21T18:09:01.706 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-21T18:09:01.706 INFO:tasks.rados.rados.0.smithi160.stdout:2211: finishing write tid 3 to smithi16030472-49 2024-09-21T18:09:01.707 INFO:tasks.rados.rados.0.smithi160.stdout:2213: got (16) Device or resource busy 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: finishing write tid 4 to smithi16030472-49 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: finishing write tid 5 to smithi16030472-49 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: oid 49 updating version 0 to 772 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: oid 49 version 772 is already newer than 771 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: oid 49 updating version 772 to 774 2024-09-21T18:09:01.713 INFO:tasks.rados.rados.0.smithi160.stdout:2211: oid 49 version 774 is already newer than 773 2024-09-21T18:09:01.714 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 774 (ObjNum 1059 snap 0 seq_num 1059) dirty exists 2024-09-21T18:09:01.714 INFO:tasks.rados.rados.0.smithi160.stdout:2211: left oid 49 (ObjNum 1059 snap 0 seq_num 1059) 2024-09-21T18:09:01.725 INFO:tasks.rados.rados.0.smithi160.stdout:2206: got (0) Success 2024-09-21T18:09:01.726 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 0 (ObjNum 801 snap 0 seq_num 801) dirty exists 2024-09-21T18:09:01.727 INFO:tasks.rados.rados.0.smithi160.stdout:2207: expect (ObjNum 956 snap 0 seq_num 956) 2024-09-21T18:09:01.728 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:01.728 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:01.728 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:01.728 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:01.728 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:01.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:01.834 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:01.829+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:01.843 INFO:tasks.rados.rados.0.smithi160.stdout:2209: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:09:01.897 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:01.894+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:01.904 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1517799 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.922 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.245509238, 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:01.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:09:01.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:09:01.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.24543887599999, 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.927 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:09:01.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:09:01.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.930 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.233637441, 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:01.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:09:01.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.22577105800001, 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.939 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.225385414, 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:01.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:09:01.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:09:01.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.943 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 103.225339965, 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.944 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.947 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:09:01.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 102.177442048, 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.949 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:01.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:09:01.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:01.953 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 102.177334406, 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:09:01.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 96.601551791000006, 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.958 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:09:01.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:09:01.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.963 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 96.601499856000004, 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.964 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:09:01.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 95.834795467999996, 2024-09-21T18:09:01.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:01.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:09:01.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 95.605692210000001, 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.976 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.977 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:09:01.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:09:01.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 95.605264219999995, 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.986 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:09:01.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:01.991 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 95.605217651000004, 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:01.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:09:01.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:09:01.995 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:09:01.996 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:01.997 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1535 4.2 4:7b415eca:::smithi16030472-342:head [write 3431711~299476] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 91.105052541000006, 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.131056219, 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1535 2024-09-21T18:09:01.998 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812782+0000", 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:01.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.812786+0000", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.817e-06 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813177+0000", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00039092899999999998 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813206+0000", 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9581999999999999e-05 2024-09-21T18:09:02.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813223+0000", 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6419999999999999e-05 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850128+0000", 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.036905822999999997 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:02.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850283+0000", 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015470600000000001 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738677+0000", 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88839409300000005 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.738839+0000", 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000161495 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.739083+0000", 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024405 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842245+0000", 2024-09-21T18:09:02.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103162504 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943517+0000", 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101271839 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943700+0000", 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018297100000000001 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.943838+0000", 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013799 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 91.104548590999997, 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:02.007 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:09:02.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:09:02.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:09:02.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:09:02.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.097016795000002, 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:09:02.014 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:09:02.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:09:02.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 84.096942475000006, 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:09:02.021 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:09:02.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.027 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 83.695139229999995, 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:02.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:09:02.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:09:02.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:09:02.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 83.592789342000003, 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:02.032 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:02.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:02.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.037 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:02.038 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:02.038 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:02.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2216: got (0) Success 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 105 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2214: finishing copy_from racing read to smithi16030472-173 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2208: finishing write tid 1 to smithi16030472-172 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2214: finishing copy_from to smithi16030472-173 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 407 (ObjNum 488 snap 0 seq_num 488) dirty exists 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2206: done (11 left) 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2207: done (10 left) 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2209: done (9 left) 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2210: done (8 left) 2024-09-21T18:09:02.041 INFO:tasks.rados.rados.0.smithi160.stdout:2211: done (7 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2212: done (6 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2213: done (5 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2214: done (4 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2215: done (3 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2216: done (2 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2217: done (1 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2218: read oid 63 snap -1 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2218: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2219: delete oid 177 current snap is 0 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2219: done (2 left) 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2220: write oid 70 current snap is 0 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2220: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2220: writing smithi16030472-70 from 649034 to 1136539 tid 1 2024-09-21T18:09:02.042 INFO:tasks.rados.rados.0.smithi160.stdout:2220: writing smithi16030472-70 from 1629171 to 2057878 tid 2 2024-09-21T18:09:02.043 INFO:tasks.rados.rados.0.smithi160.stdout:2220: writing smithi16030472-70 from 2680983 to 3245743 tid 3 2024-09-21T18:09:02.043 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:02.036+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2220: writing smithi16030472-70 from 3798768 to 3849034 tid 4 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2221: delete oid 317 current snap is 0 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: finishing write tid 2 to smithi16030472-172 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: finishing write tid 3 to smithi16030472-172 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: finishing write tid 4 to smithi16030472-172 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: finishing write tid 5 to smithi16030472-172 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: oid 172 updating version 0 to 733 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: oid 172 updating version 733 to 735 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: oid 172 updating version 735 to 736 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:2208: oid 172 version 736 is already newer than 734 2024-09-21T18:09:02.044 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 172 v 736 (ObjNum 1058 snap 0 seq_num 1058) dirty exists 2024-09-21T18:09:02.045 INFO:tasks.rados.rados.0.smithi160.stdout:2208: left oid 172 (ObjNum 1058 snap 0 seq_num 1058) 2024-09-21T18:09:02.140 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:02.136+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:02.147 INFO:tasks.rados.rados.0.smithi160.stdout:2218: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:09:02.179 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:02.179 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:02.179 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:02.179 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:02.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:02.242 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:02.239+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:02.249 INFO:tasks.rados.rados.0.smithi160.stdout:2208: done (3 left) 2024-09-21T18:09:02.249 INFO:tasks.rados.rados.0.smithi160.stdout:2218: done (2 left) 2024-09-21T18:09:02.249 INFO:tasks.rados.rados.0.smithi160.stdout:2221: done (1 left) 2024-09-21T18:09:02.250 INFO:tasks.rados.rados.0.smithi160.stdout:2222: read oid 76 snap -1 2024-09-21T18:09:02.250 INFO:tasks.rados.rados.0.smithi160.stdout:2222: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-21T18:09:02.250 INFO:tasks.rados.rados.0.smithi160.stdout:2223: read oid 238 snap -1 2024-09-21T18:09:02.250 INFO:tasks.rados.rados.0.smithi160.stdout:2223: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:02.250 INFO:tasks.rados.rados.0.smithi160.stdout:2224: delete oid 96 current snap is 0 2024-09-21T18:09:02.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:02.322 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:02.323 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:02.323 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:02.323 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:02.323 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:02.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:02.343 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:02.339+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:02.351 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:02.348+0000 7fd79a4fd640 -1 osd.1 90 handle_osd_map injecting map crc failure 2024-09-21T18:09:02.358 INFO:tasks.rados.rados.0.smithi160.stdout:2222: expect (ObjNum 357 snap 0 seq_num 357) 2024-09-21T18:09:02.363 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1000989 2024-09-21T18:09:02.374 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:02.395 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:02.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:02.417 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:02.438 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:02.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:02.444 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:02.437+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:02.450 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 1 to smithi16030472-70 2024-09-21T18:09:02.450 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 2 to smithi16030472-70 2024-09-21T18:09:02.451 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 3 to smithi16030472-70 2024-09-21T18:09:02.451 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 4 to smithi16030472-70 2024-09-21T18:09:02.456 INFO:tasks.rados.rados.0.smithi160.stdout:2223: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:02.458 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:02.458 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:02.460 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:02.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:02.483 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:02.501 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:02.501 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:02.522 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:02.522 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:02.545 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:02.540+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:02.560 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:02.561 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:02.582 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:02.583 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 5 to smithi16030472-70 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: finishing write tid 6 to smithi16030472-70 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: oid 70 updating version 0 to 695 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: oid 70 version 695 is already newer than 694 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: oid 70 version 695 is already newer than 693 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: oid 70 updating version 695 to 696 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: oid 70 version 696 is already newer than 692 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 696 (ObjNum 1062 snap 0 seq_num 1062) dirty exists 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: left oid 70 (ObjNum 1062 snap 0 seq_num 1062) 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2220: done (3 left) 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2222: done (2 left) 2024-09-21T18:09:02.596 INFO:tasks.rados.rados.0.smithi160.stdout:2223: done (1 left) 2024-09-21T18:09:02.597 INFO:tasks.rados.rados.0.smithi160.stdout:2224: done (0 left) 2024-09-21T18:09:02.597 INFO:tasks.rados.rados.0.smithi160.stdout:2225: read oid 333 snap -1 2024-09-21T18:09:02.597 INFO:tasks.rados.rados.0.smithi160.stdout:2225: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-21T18:09:02.597 INFO:tasks.rados.rados.0.smithi160.stdout:2226: delete oid 94 current snap is 0 2024-09-21T18:09:02.597 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:02.597 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:02.600 INFO:tasks.rados.rados.0.smithi160.stdout:2226: done (1 left) 2024-09-21T18:09:02.600 INFO:tasks.rados.rados.0.smithi160.stdout:2227: cache_try_flush oid 112 snap -1 2024-09-21T18:09:02.601 INFO:tasks.rados.rados.0.smithi160.stdout:2228: read oid 38 snap -1 2024-09-21T18:09:02.601 INFO:tasks.rados.rados.0.smithi160.stdout:2228: expect (ObjNum 500 snap 0 seq_num 500) 2024-09-21T18:09:02.601 INFO:tasks.rados.rados.0.smithi160.stdout:2229: delete oid 206 current snap is 0 2024-09-21T18:09:02.602 INFO:tasks.rados.rados.0.smithi160.stdout:2225: expect (ObjNum 57 snap 0 seq_num 57) 2024-09-21T18:09:02.605 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:02.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:02.645 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:02.637+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:02.706 INFO:tasks.rados.rados.0.smithi160.stdout:2228: expect (ObjNum 500 snap 0 seq_num 500) 2024-09-21T18:09:02.706 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 225085 2024-09-21T18:09:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:02.746 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:02.738+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:02.777 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:02.811 INFO:tasks.rados.rados.0.smithi160.stdout:2227: got (0) Success 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 112 v 0 (ObjNum 81 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2225: done (3 left) 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2227: done (2 left) 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2228: done (1 left) 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2229: done (0 left) 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2230: copy_from oid 355 from oid 367 current snap is 0 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2231: read oid 5 snap -1 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2231: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2232: cache_flush oid 123 snap -1 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2233: cache_evict oid 98 snap -1 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2234: cache_evict oid 126 snap -1 2024-09-21T18:09:02.812 INFO:tasks.rados.rados.0.smithi160.stdout:2235: read oid 15 snap -1 2024-09-21T18:09:02.813 INFO:tasks.rados.rados.0.smithi160.stdout:2235: expect (ObjNum 868 snap 0 seq_num 868) 2024-09-21T18:09:02.813 INFO:tasks.rados.rados.0.smithi160.stdout:2236: cache_try_flush oid 142 snap -1 2024-09-21T18:09:02.813 INFO:tasks.rados.rados.0.smithi160.stdout:2237: cache_try_flush oid 313 snap -1 2024-09-21T18:09:02.813 INFO:tasks.rados.rados.0.smithi160.stdout:2238: write (excl) oid 364 current snap is 0 2024-09-21T18:09:02.813 INFO:tasks.rados.rados.0.smithi160.stdout:2238: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-09-21T18:09:02.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:02.815 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:02.815 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:02.833 INFO:tasks.rados.rados.0.smithi160.stdout:2238: writing smithi16030472-364 from 733069 to 1313172 tid 1 2024-09-21T18:09:02.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:02.844 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:02.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:02.847 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:02.842+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:02.853 INFO:tasks.rados.rados.0.smithi160.stdout:2238: writing smithi16030472-364 from 1921581 to 2501284 tid 2 2024-09-21T18:09:02.868 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:02.871 INFO:tasks.rados.rados.0.smithi160.stdout:2238: writing smithi16030472-364 from 3218098 to 3755571 tid 3 2024-09-21T18:09:02.871 INFO:tasks.rados.rados.0.smithi160.stdout:2239: write oid 146 current snap is 0 2024-09-21T18:09:02.871 INFO:tasks.rados.rados.0.smithi160.stdout:2239: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-21T18:09:02.872 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:02.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:02.885 INFO:tasks.rados.rados.0.smithi160.stdout:2239: writing smithi16030472-146 from 749876 to 1405228 tid 1 2024-09-21T18:09:02.895 INFO:tasks.rados.rados.0.smithi160.stdout:2239: writing smithi16030472-146 from 1863710 to 2587071 tid 2 2024-09-21T18:09:02.905 INFO:tasks.rados.rados.0.smithi160.stdout:2239: writing smithi16030472-146 from 3012815 to 3761560 tid 3 2024-09-21T18:09:02.905 INFO:tasks.rados.rados.0.smithi160.stdout:2240: delete oid 234 current snap is 0 2024-09-21T18:09:02.905 INFO:tasks.rados.rados.0.smithi160.stdout:2233: got (0) Success 2024-09-21T18:09:02.905 INFO:tasks.rados.rados.0.smithi160.stdout:2230: finishing copy_from racing read to smithi16030472-355 2024-09-21T18:09:02.914 INFO:tasks.rados.rados.0.smithi160.stdout:2231: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:02.944 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 569387 2024-09-21T18:09:02.947 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:02.944+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:02.963 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1546162 2024-09-21T18:09:02.972 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:02.972 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:02.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:03.007 INFO:tasks.rados.rados.0.smithi160.stdout:2230: finishing copy_from to smithi16030472-355 2024-09-21T18:09:03.007 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 712 (ObjNum 968 snap 0 seq_num 968) dirty exists 2024-09-21T18:09:03.015 INFO:tasks.rados.rados.0.smithi160.stdout:2234: got (0) Success 2024-09-21T18:09:03.018 INFO:tasks.rados.rados.0.smithi160.stdout:2235: expect (ObjNum 868 snap 0 seq_num 868) 2024-09-21T18:09:03.018 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:03.048 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:03.042+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:03.115 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.115 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:03.137 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 833864 2024-09-21T18:09:03.149 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:03.144+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2232: got (0) Success 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 0 (ObjNum 884 snap 0 seq_num 884) dirty exists 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2236: got (0) Success 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 0 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2238: finishing write tid 1 to smithi16030472-364 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2239: finishing write tid 1 to smithi16030472-146 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2239: finishing write tid 2 to smithi16030472-146 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2239: finishing write tid 3 to smithi16030472-146 2024-09-21T18:09:03.167 INFO:tasks.rados.rados.0.smithi160.stdout:2239: finishing write tid 4 to smithi16030472-146 2024-09-21T18:09:03.185 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:03.177+0000 7f94db546640 -1 osd.15 91 handle_osd_map injecting map crc failure 2024-09-21T18:09:03.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: finishing write tid 5 to smithi16030472-146 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: oid 146 updating version 0 to 769 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: oid 146 version 769 is already newer than 766 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: oid 146 version 769 is already newer than 768 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: oid 146 version 769 is already newer than 767 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 769 (ObjNum 1068 snap 0 seq_num 1068) dirty exists 2024-09-21T18:09:03.240 INFO:tasks.rados.rados.0.smithi160.stdout:2239: left oid 146 (ObjNum 1068 snap 0 seq_num 1068) 2024-09-21T18:09:03.256 INFO:tasks.rados.rados.0.smithi160.stdout:2237: got (0) Success 2024-09-21T18:09:03.256 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 313 v 0 (ObjNum 1003 snap 0 seq_num 1003) dirty exists 2024-09-21T18:09:03.259 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:03.254+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:03.301 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.302 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:03.351 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:03.347+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:03.452 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:03.447+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:03.457 INFO:tasks.rados.rados.0.smithi160.stdout:2238: finishing write tid 2 to smithi16030472-364 2024-09-21T18:09:03.457 INFO:tasks.rados.rados.0.smithi160.stdout:2238: finishing write tid 3 to smithi16030472-364 2024-09-21T18:09:03.458 INFO:tasks.rados.rados.0.smithi160.stdout:2238: finishing write tid 4 to smithi16030472-364 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: finishing write tid 5 to smithi16030472-364 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: oid 364 updating version 0 to 777 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: oid 364 version 777 is already newer than 776 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: oid 364 updating version 777 to 779 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: oid 364 version 779 is already newer than 778 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 779 (ObjNum 1067 snap 0 seq_num 1067) dirty exists 2024-09-21T18:09:03.459 INFO:tasks.rados.rados.0.smithi160.stdout:2238: left oid 364 (ObjNum 1067 snap 0 seq_num 1067) 2024-09-21T18:09:03.460 INFO:tasks.rados.rados.0.smithi160.stdout:2230: done (10 left) 2024-09-21T18:09:03.460 INFO:tasks.rados.rados.0.smithi160.stdout:2231: done (9 left) 2024-09-21T18:09:03.460 INFO:tasks.rados.rados.0.smithi160.stdout:2232: done (8 left) 2024-09-21T18:09:03.460 INFO:tasks.rados.rados.0.smithi160.stdout:2233: done (7 left) 2024-09-21T18:09:03.460 INFO:tasks.rados.rados.0.smithi160.stdout:2234: done (6 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2235: done (5 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2236: done (4 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2237: done (3 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2238: done (2 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2239: done (1 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2240: done (0 left) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2241: read oid 275 snap -1 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2241: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2242: copy_from oid 438 from oid 321 current snap is 0 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2243: write (excl) oid 203 current snap is 0 2024-09-21T18:09:03.461 INFO:tasks.rados.rados.0.smithi160.stdout:2243: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-09-21T18:09:03.468 INFO:tasks.rados.rados.0.smithi160.stdout:2243: writing smithi16030472-203 from 783490 to 1189340 tid 1 2024-09-21T18:09:03.478 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.478 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.479 INFO:tasks.rados.rados.0.smithi160.stdout:2243: writing smithi16030472-203 from 1980674 to 2707704 tid 2 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2243: writing smithi16030472-203 from 3184014 to 3671656 tid 3 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2244: cache_try_flush oid 170 snap -1 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2245: cache_try_flush oid 156 snap -1 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2246: copy_from oid 372 from oid 391 current snap is 0 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2242: finishing copy_from to smithi16030472-438 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2242: got expected ENOENT (src dne) 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2242: finishing copy_from racing read to smithi16030472-438 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2242: done (5 left) 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2247: read oid 186 snap -1 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2247: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2248: write oid 75 current snap is 0 2024-09-21T18:09:03.487 INFO:tasks.rados.rados.0.smithi160.stdout:2248: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-21T18:09:03.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:03.493 INFO:tasks.rados.rados.0.smithi160.stdout:2248: writing smithi16030472-75 from 400297 to 881396 tid 1 2024-09-21T18:09:03.500 INFO:tasks.rados.rados.0.smithi160.stdout:2248: writing smithi16030472-75 from 1639156 to 2000297 tid 2 2024-09-21T18:09:03.500 INFO:tasks.rados.rados.0.smithi160.stdout:2249: cache_try_flush oid 142 snap -1 2024-09-21T18:09:03.500 INFO:tasks.rados.rados.0.smithi160.stdout:2250: write (excl) oid 453 current snap is 0 2024-09-21T18:09:03.501 INFO:tasks.rados.rados.0.smithi160.stdout:2250: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-09-21T18:09:03.508 INFO:tasks.rados.rados.0.smithi160.stdout:2250: writing smithi16030472-453 from 417104 to 1089805 tid 1 2024-09-21T18:09:03.515 INFO:tasks.rados.rados.0.smithi160.stdout:2250: writing smithi16030472-453 from 1813991 to 2017104 tid 2 2024-09-21T18:09:03.515 INFO:tasks.rados.rados.0.smithi160.stdout:2251: read oid 394 snap -1 2024-09-21T18:09:03.515 INFO:tasks.rados.rados.0.smithi160.stdout:2251: expect deleted 2024-09-21T18:09:03.515 INFO:tasks.rados.rados.0.smithi160.stdout:2252: copy_from oid 194 from oid 327 current snap is 0 2024-09-21T18:09:03.515 INFO:tasks.rados.rados.0.smithi160.stdout:2253: delete oid 93 current snap is 0 2024-09-21T18:09:03.553 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:03.548+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:03.558 INFO:tasks.rados.rados.0.smithi160.stdout:2249: got (0) Success 2024-09-21T18:09:03.558 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 0 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:09:03.601 INFO:tasks.rados.rados.0.smithi160.stdout:2247: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:09:03.626 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.627 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:03.653 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:03.649+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:03.669 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:03.747 INFO:tasks.rados.rados.0.smithi160.stdout:2243: finishing write tid 1 to smithi16030472-203 2024-09-21T18:09:03.747 INFO:tasks.rados.rados.0.smithi160.stdout:2244: got (0) Success 2024-09-21T18:09:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 170 v 0 (ObjNum 129 snap 0 seq_num 3131691024) clean dne 2024-09-21T18:09:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:2245: got (0) Success 2024-09-21T18:09:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 156 v 0 (ObjNum 1019 snap 0 seq_num 1019) dirty exists 2024-09-21T18:09:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:2248: finishing write tid 1 to smithi16030472-75 2024-09-21T18:09:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:2243: finishing write tid 2 to smithi16030472-203 2024-09-21T18:09:03.755 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:03.751+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:03.767 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.767 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:03.794 INFO:tasks.rados.rados.0.smithi160.stdout:2241: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:09:03.794 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:03.855 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:03.852+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:2246: finishing copy_from racing read to smithi16030472-372 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:2246: finishing copy_from to smithi16030472-372 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 283 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:2243: finishing write tid 3 to smithi16030472-203 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:2250: finishing write tid 1 to smithi16030472-453 2024-09-21T18:09:03.904 INFO:tasks.rados.rados.0.smithi160.stdout:2250: finishing write tid 2 to smithi16030472-453 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: finishing write tid 3 to smithi16030472-453 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: finishing write tid 4 to smithi16030472-453 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: oid 453 updating version 0 to 738 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: oid 453 updating version 738 to 740 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: oid 453 version 740 is already newer than 739 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 453 v 740 (ObjNum 1072 snap 0 seq_num 1072) dirty exists 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2250: left oid 453 (ObjNum 1072 snap 0 seq_num 1072) 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: finishing write tid 4 to smithi16030472-203 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: finishing write tid 5 to smithi16030472-203 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: oid 203 updating version 0 to 747 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: oid 203 version 747 is already newer than 731 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: oid 203 version 747 is already newer than 730 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:2243: oid 203 updating version 747 to 748 2024-09-21T18:09:03.905 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 748 (ObjNum 1070 snap 0 seq_num 1070) dirty exists 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2243: left oid 203 (ObjNum 1070 snap 0 seq_num 1070) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2241: done (11 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2243: done (10 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2244: done (9 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2245: done (8 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2246: done (7 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2247: done (6 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2249: done (5 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2250: done (4 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2251: done (3 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2253: done (2 left) 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2254: cache_try_flush oid 357 snap -1 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2255: cache_flush oid 390 snap -1 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2256: cache_flush oid 61 snap -1 2024-09-21T18:09:03.906 INFO:tasks.rados.rados.0.smithi160.stdout:2257: read oid 460 snap -1 2024-09-21T18:09:03.907 INFO:tasks.rados.rados.0.smithi160.stdout:2257: expect (ObjNum 786 snap 0 seq_num 786) 2024-09-21T18:09:03.907 INFO:tasks.rados.rados.0.smithi160.stdout:2258: write (excl) oid 49 current snap is 0 2024-09-21T18:09:03.907 INFO:tasks.rados.rados.0.smithi160.stdout:2258: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-21T18:09:03.910 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:03.910 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:03.913 INFO:tasks.rados.rados.0.smithi160.stdout:2258: writing smithi16030472-49 from 450718 to 873917 tid 1 2024-09-21T18:09:03.925 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:03.925 INFO:tasks.rados.rados.0.smithi160.stdout:2258: writing smithi16030472-49 from 1414602 to 2050718 tid 2 2024-09-21T18:09:03.925 INFO:tasks.rados.rados.0.smithi160.stdout:2259: write oid 248 current snap is 0 2024-09-21T18:09:03.925 INFO:tasks.rados.rados.0.smithi160.stdout:2254: got (0) Success 2024-09-21T18:09:03.926 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 357 v 0 (ObjNum 356 snap 0 seq_num 356) dirty exists 2024-09-21T18:09:03.926 INFO:tasks.rados.rados.0.smithi160.stdout:2257: expect (ObjNum 786 snap 0 seq_num 786) 2024-09-21T18:09:03.956 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:03.949+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:04.020 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:04.056 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:04.049+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:04.062 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.062 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.070 INFO:tasks.rados.rados.0.smithi160.stdout:2256: got (2) No such file or directory 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2255: got (2) No such file or directory 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2252: finishing copy_from to smithi16030472-194 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 194 v 229 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2252: finishing copy_from racing read to smithi16030472-194 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: finishing write tid 2 to smithi16030472-75 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: finishing write tid 3 to smithi16030472-75 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: finishing write tid 4 to smithi16030472-75 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: oid 75 updating version 0 to 698 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: oid 75 version 698 is already newer than 697 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: oid 75 updating version 698 to 699 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 75 v 699 (ObjNum 1071 snap 0 seq_num 1071) dirty exists 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2248: left oid 75 (ObjNum 1071 snap 0 seq_num 1071) 2024-09-21T18:09:04.071 INFO:tasks.rados.rados.0.smithi160.stdout:2259: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-21T18:09:04.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:04.085 INFO:tasks.rados.rados.0.smithi160.stdout:2259: writing smithi16030472-248 from 467525 to 965973 tid 1 2024-09-21T18:09:04.102 INFO:tasks.rados.rados.0.smithi160.stdout:2259: writing smithi16030472-248 from 1473084 to 2067463 tid 2 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2248: done (7 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2252: done (6 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2254: done (5 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2255: done (4 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2256: done (3 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2257: done (2 left) 2024-09-21T18:09:04.103 INFO:tasks.rados.rados.0.smithi160.stdout:2260: delete oid 312 current snap is 0 2024-09-21T18:09:04.157 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:04.154+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:04.193 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:09:04.194 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:04.191+0000 7f728550b640 -1 osd.14 92 handle_osd_map injecting map crc failure 2024-09-21T18:09:04.197 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.197 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.203 INFO:tasks.rados.rados.0.smithi160.stdout:2258: finishing write tid 1 to smithi16030472-49 2024-09-21T18:09:04.212 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:04.236 INFO:tasks.rados.rados.0.smithi160.stdout:2258: finishing write tid 2 to smithi16030472-49 2024-09-21T18:09:04.236 INFO:tasks.rados.rados.0.smithi160.stdout:2258: finishing write tid 3 to smithi16030472-49 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:2258: finishing write tid 4 to smithi16030472-49 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:2258: oid 49 updating version 0 to 780 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:2258: oid 49 updating version 780 to 781 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:2258: oid 49 updating version 781 to 782 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 782 (ObjNum 1074 snap 0 seq_num 1074) dirty exists 2024-09-21T18:09:04.237 INFO:tasks.rados.rados.0.smithi160.stdout:2258: left oid 49 (ObjNum 1074 snap 0 seq_num 1074) 2024-09-21T18:09:04.259 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:04.254+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:04.336 INFO:tasks.rados.rados.0.smithi160.stdout:2259: finishing write tid 1 to smithi16030472-248 2024-09-21T18:09:04.338 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.338 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.353 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:04.399 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:04.396+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:04.436 INFO:tasks.rados.rados.0.smithi160.stdout:2259: finishing write tid 2 to smithi16030472-248 2024-09-21T18:09:04.436 INFO:tasks.rados.rados.0.smithi160.stdout:2259: finishing write tid 3 to smithi16030472-248 2024-09-21T18:09:04.459 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:04.457+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:04.480 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.480 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.495 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: finishing write tid 4 to smithi16030472-248 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: oid 248 updating version 0 to 784 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: oid 248 updating version 784 to 785 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: oid 248 version 785 is already newer than 783 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 248 v 785 (ObjNum 1075 snap 0 seq_num 1075) dirty exists 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: left oid 248 (ObjNum 1075 snap 0 seq_num 1075) 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2258: done (2 left) 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2259: done (1 left) 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2260: done (0 left) 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2261: cache_try_flush oid 7 snap -1 2024-09-21T18:09:04.537 INFO:tasks.rados.rados.0.smithi160.stdout:2262: read oid 192 snap -1 2024-09-21T18:09:04.538 INFO:tasks.rados.rados.0.smithi160.stdout:2262: expect (ObjNum 970 snap 0 seq_num 970) 2024-09-21T18:09:04.538 INFO:tasks.rados.rados.0.smithi160.stdout:2263: read oid 185 snap -1 2024-09-21T18:09:04.538 INFO:tasks.rados.rados.0.smithi160.stdout:2263: expect deleted 2024-09-21T18:09:04.538 INFO:tasks.rados.rados.0.smithi160.stdout:2264: read oid 472 snap -1 2024-09-21T18:09:04.538 INFO:tasks.rados.rados.0.smithi160.stdout:2264: expect (ObjNum 741 snap 0 seq_num 741) 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:2265: cache_evict oid 333 snap -1 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:2266: copy_from oid 334 from oid 127 current snap is 0 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:2267: copy_from oid 224 from oid 118 current snap is 0 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:2268: delete oid 337 current snap is 0 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:2261: got (0) Success 2024-09-21T18:09:04.539 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 7 v 0 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-21T18:09:04.544 INFO:tasks.rados.rados.0.smithi160.stdout:2261: done (7 left) 2024-09-21T18:09:04.544 INFO:tasks.rados.rados.0.smithi160.stdout:2268: done (6 left) 2024-09-21T18:09:04.544 INFO:tasks.rados.rados.0.smithi160.stdout:2269: delete oid 482 current snap is 0 2024-09-21T18:09:04.548 INFO:tasks.rados.rados.0.smithi160.stdout:2269: done (6 left) 2024-09-21T18:09:04.548 INFO:tasks.rados.rados.0.smithi160.stdout:2270: read oid 133 snap -1 2024-09-21T18:09:04.548 INFO:tasks.rados.rados.0.smithi160.stdout:2270: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:09:04.548 INFO:tasks.rados.rados.0.smithi160.stdout:2271: write oid 15 current snap is 0 2024-09-21T18:09:04.548 INFO:tasks.rados.rados.0.smithi160.stdout:2271: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-21T18:09:04.560 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:04.558+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:04.569 INFO:tasks.rados.rados.0.smithi160.stdout:2271: writing smithi16030472-15 from 534753 to 1334197 tid 1 2024-09-21T18:09:04.582 INFO:tasks.rados.rados.0.smithi160.stdout:2271: writing smithi16030472-15 from 1990659 to 2134753 tid 2 2024-09-21T18:09:04.583 INFO:tasks.rados.rados.0.smithi160.stdout:2272: write oid 146 current snap is 0 2024-09-21T18:09:04.583 INFO:tasks.rados.rados.0.smithi160.stdout:2272: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-21T18:09:04.600 INFO:tasks.rados.rados.0.smithi160.stdout:2272: writing smithi16030472-146 from 551560 to 1142606 tid 1 2024-09-21T18:09:04.612 INFO:tasks.rados.rados.0.smithi160.stdout:2272: writing smithi16030472-146 from 1765494 to 2151560 tid 2 2024-09-21T18:09:04.612 INFO:tasks.rados.rados.0.smithi160.stdout:2273: write (excl) oid 363 current snap is 0 2024-09-21T18:09:04.612 INFO:tasks.rados.rados.0.smithi160.stdout:2273: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-09-21T18:09:04.624 INFO:tasks.rados.rados.0.smithi160.stdout:2273: writing smithi16030472-363 from 568367 to 1234662 tid 1 2024-09-21T18:09:04.632 INFO:tasks.rados.rados.0.smithi160.stdout:2273: writing smithi16030472-363 from 1707623 to 2168367 tid 2 2024-09-21T18:09:04.632 INFO:tasks.rados.rados.0.smithi160.stdout:2274: cache_flush oid 139 snap -1 2024-09-21T18:09:04.632 INFO:tasks.rados.rados.0.smithi160.stdout:2275: delete oid 397 current snap is 0 2024-09-21T18:09:04.661 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:04.656+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:04.669 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.670 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.685 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:04.733 INFO:tasks.rados.rados.0.smithi160.stdout:2272: finishing write tid 1 to smithi16030472-146 2024-09-21T18:09:04.750 INFO:tasks.rados.rados.0.smithi160.stdout:2266: finishing copy_from racing read to smithi16030472-334 2024-09-21T18:09:04.762 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:04.757+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:04.819 INFO:tasks.rados.rados.0.smithi160.stdout:2275: done (11 left) 2024-09-21T18:09:04.819 INFO:tasks.rados.rados.0.smithi160.stdout:2276: read oid 212 snap -1 2024-09-21T18:09:04.819 INFO:tasks.rados.rados.0.smithi160.stdout:2276: expect (ObjNum 509 snap 0 seq_num 509) 2024-09-21T18:09:04.819 INFO:tasks.rados.rados.0.smithi160.stdout:2277: cache_try_flush oid 414 snap -1 2024-09-21T18:09:04.819 INFO:tasks.rados.rados.0.smithi160.stdout:2278: read oid 231 snap -1 2024-09-21T18:09:04.820 INFO:tasks.rados.rados.0.smithi160.stdout:2278: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:04.820 INFO:tasks.rados.rados.0.smithi160.stdout:2279: read oid 216 snap -1 2024-09-21T18:09:04.820 INFO:tasks.rados.rados.0.smithi160.stdout:2279: expect (ObjNum 605 snap 0 seq_num 605) 2024-09-21T18:09:04.820 INFO:tasks.rados.rados.0.smithi160.stdout:2280: cache_flush oid 307 snap -1 2024-09-21T18:09:04.820 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:04.843 INFO:tasks.rados.rados.0.smithi160.stdout:2265: got (16) Device or resource busy 2024-09-21T18:09:04.843 INFO:tasks.rados.rados.0.smithi160.stdout:2265: done (15 left) 2024-09-21T18:09:04.843 INFO:tasks.rados.rados.0.smithi160.stdout:2281: read oid 376 snap -1 2024-09-21T18:09:04.843 INFO:tasks.rados.rados.0.smithi160.stdout:2281: expect (ObjNum 740 snap 0 seq_num 740) 2024-09-21T18:09:04.843 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:04.844 INFO:tasks.rados.rados.0.smithi160.stdout:2262: expect (ObjNum 970 snap 0 seq_num 970) 2024-09-21T18:09:04.849 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:04.844+0000 7f4976023640 -1 osd.5 92 handle_osd_map injecting map crc failure 2024-09-21T18:09:04.860 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:04.860 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:04.862 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:04.858+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:04.869 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1765900 2024-09-21T18:09:04.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:04.874 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:04.889 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:04.963 INFO:tasks.rados.rados.0.smithi160.stdout:2264: expect (ObjNum 741 snap 0 seq_num 741) 2024-09-21T18:09:04.963 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:04.958+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:05.004 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:05.004 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:05.020 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:05.030 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2116066 2024-09-21T18:09:05.036 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:05.036 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:05.056 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:05.056 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:05.071 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:05.068+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:05.074 INFO:tasks.rados.rados.0.smithi160.stdout:2270: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:09:05.074 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:05.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:05.100 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:05.101 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:05.122 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:05.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:05.145 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:05.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:05.147 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:05.147 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:05.162 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:05.165 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:05.157+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:05.169 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:05.169 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:05.192 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:05.192 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:05.195 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3185415 2024-09-21T18:09:05.206 INFO:tasks.rados.rados.0.smithi160.stdout:2280: got (0) Success 2024-09-21T18:09:05.206 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 0 (ObjNum 129 snap 0 seq_num 3136084568) clean dne 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: finishing write tid 2 to smithi16030472-146 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: finishing write tid 3 to smithi16030472-146 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: finishing write tid 4 to smithi16030472-146 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: oid 146 updating version 0 to 772 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: oid 146 updating version 772 to 773 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: oid 146 updating version 773 to 774 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 774 (ObjNum 1080 snap 0 seq_num 1080) dirty exists 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2272: left oid 146 (ObjNum 1080 snap 0 seq_num 1080) 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2273: finishing write tid 1 to smithi16030472-363 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2273: finishing write tid 2 to smithi16030472-363 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2262: done (15 left) 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2263: done (14 left) 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2264: done (13 left) 2024-09-21T18:09:05.207 INFO:tasks.rados.rados.0.smithi160.stdout:2270: done (12 left) 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2272: done (11 left) 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2280: done (10 left) 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2282: delete oid 125 current snap is 0 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: finishing write tid 3 to smithi16030472-363 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: finishing write tid 4 to smithi16030472-363 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: oid 363 updating version 0 to 777 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: oid 363 version 777 is already newer than 775 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: oid 363 version 777 is already newer than 776 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 777 (ObjNum 1081 snap 0 seq_num 1081) dirty exists 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2273: left oid 363 (ObjNum 1081 snap 0 seq_num 1081) 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2266: finishing copy_from to smithi16030472-334 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 334 v 229 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:09:05.208 INFO:tasks.rados.rados.0.smithi160.stdout:2281: expect (ObjNum 740 snap 0 seq_num 740) 2024-09-21T18:09:05.212 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:05.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:05.215 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1309309 2024-09-21T18:09:05.236 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:05.236 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.258 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:05.266 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:05.263+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:05.272 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.279 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:05.279 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:05.302 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:05.302 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2267: finishing copy_from to smithi16030472-224 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 224 v 649 (ObjNum 897 snap 0 seq_num 897) dirty exists 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2267: finishing copy_from racing read to smithi16030472-224 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2271: finishing write tid 1 to smithi16030472-15 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2271: finishing write tid 2 to smithi16030472-15 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2271: finishing write tid 3 to smithi16030472-15 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2271: finishing write tid 4 to smithi16030472-15 2024-09-21T18:09:05.309 INFO:tasks.rados.rados.0.smithi160.stdout:2271: oid 15 updating version 0 to 751 2024-09-21T18:09:05.310 INFO:tasks.rados.rados.0.smithi160.stdout:2271: oid 15 version 751 is already newer than 750 2024-09-21T18:09:05.310 INFO:tasks.rados.rados.0.smithi160.stdout:2271: oid 15 version 751 is already newer than 749 2024-09-21T18:09:05.310 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 751 (ObjNum 1079 snap 0 seq_num 1079) dirty exists 2024-09-21T18:09:05.310 INFO:tasks.rados.rados.0.smithi160.stdout:2271: left oid 15 (ObjNum 1079 snap 0 seq_num 1079) 2024-09-21T18:09:05.310 INFO:tasks.rados.rados.0.smithi160.stdout:2276: expect (ObjNum 509 snap 0 seq_num 509) 2024-09-21T18:09:05.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.305739994999996, 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:09:05.340 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.341 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:09:05.342 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:09:05.343 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.344 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.302129444000002, 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.345 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.346 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.347 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.348 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.299322904999997, 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.349 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.350 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:09:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:09:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.351 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.352 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.353 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 94.295574705000007, 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.354 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:09:05.355 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:09:05.356 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:09:05.357 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.674723900999993, 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:09:05.358 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:05.362 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.573869829000003, 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:09:05.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.366 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.573113548999999, 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.370 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.572721991999998, 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:05.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:09:05.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:09:05.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:09:05.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.572420867999995, 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.571697017999995, 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.379 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:09:05.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.470800178999994, 2024-09-21T18:09:05.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:09:05.387 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.470705895999998, 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:05.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:09:05.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:09:05.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.391 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.469792701000003, 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.469223013000004, 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.399 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.468495528999995, 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:09:05.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:09:05.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:09:05.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 93.402313620000001, 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:05.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:09:05.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:09:05.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:09:05.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 92.146267815000002, 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 92.145985017000001, 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 92.145906031999999, 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.416 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:09:05.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:09:05.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 92.143287701000006, 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:05.420 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:09:05.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:09:05.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:09:05.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:05.424 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:05.425 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:05.426 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.426 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.426 INFO:tasks.rados.rados.0.smithi160.stdout:2278: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:05.427 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:05.428 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:05.428 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:05.418+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:05.488 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.526 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:05.524+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:05.528 INFO:tasks.rados.rados.0.smithi160.stdout:2279: expect (ObjNum 605 snap 0 seq_num 605) 2024-09-21T18:09:05.557 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:05.557 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:05.557 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:05.571 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:05.613 INFO:tasks.rados.rados.0.smithi160.stdout:2277: got (0) Success 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 0 (ObjNum 1053 snap 0 seq_num 1053) dirty exists 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2274: got (0) Success 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 0 (ObjNum 978 snap 0 seq_num 978) dirty exists 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2266: done (10 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2267: done (9 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2271: done (8 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2273: done (7 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2274: done (6 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2276: done (5 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2277: done (4 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2278: done (3 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2279: done (2 left) 2024-09-21T18:09:05.614 INFO:tasks.rados.rados.0.smithi160.stdout:2281: done (1 left) 2024-09-21T18:09:05.615 INFO:tasks.rados.rados.0.smithi160.stdout:2282: done (0 left) 2024-09-21T18:09:05.615 INFO:tasks.rados.rados.0.smithi160.stdout:2283: read oid 103 snap -1 2024-09-21T18:09:05.615 INFO:tasks.rados.rados.0.smithi160.stdout:2283: expect (ObjNum 991 snap 0 seq_num 991) 2024-09-21T18:09:05.615 INFO:tasks.rados.rados.0.smithi160.stdout:2284: write oid 416 current snap is 0 2024-09-21T18:09:05.615 INFO:tasks.rados.rados.0.smithi160.stdout:2284: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-21T18:09:05.627 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:05.623+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:05.631 INFO:tasks.rados.rados.0.smithi160.stdout:2284: writing smithi16030472-416 from 618788 to 1110830 tid 1 2024-09-21T18:09:05.650 INFO:tasks.rados.rados.0.smithi160.stdout:2284: writing smithi16030472-416 from 1883069 to 2218788 tid 2 2024-09-21T18:09:05.650 INFO:tasks.rados.rados.0.smithi160.stdout:2285: cache_try_flush oid 46 snap -1 2024-09-21T18:09:05.650 INFO:tasks.rados.rados.0.smithi160.stdout:2284: finishing write tid 1 to smithi16030472-416 2024-09-21T18:09:05.650 INFO:tasks.rados.rados.0.smithi160.stdout:2286: copy_from oid 456 from oid 298 current snap is 0 2024-09-21T18:09:05.650 INFO:tasks.rados.rados.0.smithi160.stdout:2287: cache_try_flush oid 111 snap -1 2024-09-21T18:09:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:2288: cache_evict oid 239 snap -1 2024-09-21T18:09:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:2289: cache_flush oid 214 snap -1 2024-09-21T18:09:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:2290: write (excl) oid 264 current snap is 0 2024-09-21T18:09:05.651 INFO:tasks.rados.rados.0.smithi160.stdout:2290: seq_num 1085 ranges {635595=567291,1825198=410397} 2024-09-21T18:09:05.668 INFO:tasks.rados.rados.0.smithi160.stdout:2290: writing smithi16030472-264 from 635595 to 1202886 tid 1 2024-09-21T18:09:05.680 INFO:tasks.rados.rados.0.smithi160.stdout:2290: writing smithi16030472-264 from 1825198 to 2235595 tid 2 2024-09-21T18:09:05.681 INFO:tasks.rados.rados.0.smithi160.stdout:2291: write (excl) oid 393 current snap is 0 2024-09-21T18:09:05.681 INFO:tasks.rados.rados.0.smithi160.stdout:2285: got (0) Success 2024-09-21T18:09:05.681 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-21T18:09:05.681 INFO:tasks.rados.rados.0.smithi160.stdout:2291: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-21T18:09:05.694 INFO:tasks.rados.rados.0.smithi160.stdout:2291: writing smithi16030472-393 from 652402 to 1294942 tid 1 2024-09-21T18:09:05.704 INFO:tasks.rados.rados.0.smithi160.stdout:2291: writing smithi16030472-393 from 1883680 to 2252402 tid 2 2024-09-21T18:09:05.704 INFO:tasks.rados.rados.0.smithi160.stdout:2285: done (8 left) 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2292: read oid 460 snap -1 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2292: expect (ObjNum 786 snap 0 seq_num 786) 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2293: copy_from oid 494 from oid 180 current snap is 0 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2291: finishing write tid 1 to smithi16030472-393 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2294: copy_from oid 84 from oid 483 current snap is 0 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2295: cache_flush oid 323 snap -1 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2296: cache_flush oid 94 snap -1 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2297: cache_flush oid 455 snap -1 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2298: read oid 20 snap -1 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2298: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout:2299: copy_from oid 428 from oid 482 current snap is 0 2024-09-21T18:09:05.705 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:05.728 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:05.724+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:05.746 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:05.746 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:05.746 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:05.746 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:05.746 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:05.758 INFO:tasks.rados.rados.0.smithi160.stdout:2283: expect (ObjNum 991 snap 0 seq_num 991) 2024-09-21T18:09:05.758 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:05.762 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:05.829 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:05.823+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:05.852 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2447305 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: finishing write tid 2 to smithi16030472-393 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: finishing write tid 3 to smithi16030472-393 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: finishing write tid 4 to smithi16030472-393 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: oid 393 updating version 0 to 780 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: oid 393 version 780 is already newer than 778 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: oid 393 version 780 is already newer than 779 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 780 (ObjNum 1086 snap 0 seq_num 1086) dirty exists 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2291: left oid 393 (ObjNum 1086 snap 0 seq_num 1086) 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2295: got (0) Success 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 323 v 0 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:09:05.927 INFO:tasks.rados.rados.0.smithi160.stdout:2298: expect (ObjNum 92 snap 0 seq_num 92) 2024-09-21T18:09:05.931 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:05.931 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:05.931 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:05.931 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:09:05.931 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:05.931 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:05.927+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:05.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:06.025 INFO:tasks.rados.rados.0.smithi160.stdout:2289: got (0) Success 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 214 v 0 (ObjNum 129 snap 0 seq_num 3132570048) clean dne 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: finishing write tid 2 to smithi16030472-416 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: finishing write tid 3 to smithi16030472-416 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: finishing write tid 4 to smithi16030472-416 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: oid 416 updating version 0 to 753 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: oid 416 version 753 is already newer than 752 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: oid 416 updating version 753 to 754 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 754 (ObjNum 1084 snap 0 seq_num 1084) dirty exists 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: left oid 416 (ObjNum 1084 snap 0 seq_num 1084) 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2299: finishing copy_from to smithi16030472-428 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2299: got expected ENOENT (src dne) 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2283: done (15 left) 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2284: done (14 left) 2024-09-21T18:09:06.026 INFO:tasks.rados.rados.0.smithi160.stdout:2289: done (13 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2291: done (12 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2295: done (11 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2298: done (10 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2300: read oid 202 snap -1 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2300: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2301: cache_flush oid 49 snap -1 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2299: finishing copy_from racing read to smithi16030472-428 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2286: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2288: got (0) Success 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2288: done (11 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2299: done (10 left) 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2302: cache_flush oid 366 snap -1 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2290: finishing write tid 1 to smithi16030472-264 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2303: read oid 189 snap -1 2024-09-21T18:09:06.027 INFO:tasks.rados.rados.0.smithi160.stdout:2303: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:09:06.028 INFO:tasks.rados.rados.0.smithi160.stdout:2304: write (excl) oid 146 current snap is 0 2024-09-21T18:09:06.028 INFO:tasks.rados.rados.0.smithi160.stdout:2304: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-09-21T18:09:06.032 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:06.026+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:06.047 INFO:tasks.rados.rados.0.smithi160.stdout:2304: writing smithi16030472-146 from 669209 to 1386998 tid 1 2024-09-21T18:09:06.061 INFO:tasks.rados.rados.0.smithi160.stdout:2304: writing smithi16030472-146 from 1942162 to 2269209 tid 2 2024-09-21T18:09:06.061 INFO:tasks.rados.rados.0.smithi160.stdout:2305: delete oid 196 current snap is 0 2024-09-21T18:09:06.061 INFO:tasks.rados.rados.0.smithi160.stdout:2300: expect (ObjNum 201 snap 0 seq_num 201) 2024-09-21T18:09:06.071 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2694590 2024-09-21T18:09:06.081 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.081 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:06.081 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.081 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.11.22:2309 3.3 e93/31)", 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:06.065675+0000", 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.015477253, 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.015511316000000001, 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:06.065675+0000", 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.082 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:06.065675+0000", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:06.065680+0000", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.977e-06 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:06.065718+0000", 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8049000000000001e-05 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.083 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:06.065752+0000", 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4091999999999998e-05 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:09:06.084 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.086 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:06.097 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:06.135 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:06.130+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:06.159 INFO:tasks.rados.rados.0.smithi160.stdout:2303: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:09:06.167 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2224640 2024-09-21T18:09:06.179 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 731162 2024-09-21T18:09:06.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:09:06.218 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.219 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:06.219 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:06.219 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:06.219 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:2294: finishing copy_from racing read to smithi16030472-84 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:2294: finishing copy_from to smithi16030472-84 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 84 v 325 (ObjNum 394 snap 0 seq_num 394) dirty exists 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:2286: finishing copy_from to smithi16030472-456 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 476 (ObjNum 732 snap 0 seq_num 732) dirty exists 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:2287: got (0) Success 2024-09-21T18:09:06.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 111 v 0 (ObjNum -1164551648 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:06.232 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:06.269 INFO:tasks.rados.rados.0.smithi160.stdout:2290: finishing write tid 2 to smithi16030472-264 2024-09-21T18:09:06.270 INFO:tasks.rados.rados.0.smithi160.stdout:2296: got (0) Success 2024-09-21T18:09:06.270 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:06.270 INFO:tasks.rados.rados.0.smithi160.stdout:2292: expect (ObjNum 786 snap 0 seq_num 786) 2024-09-21T18:09:06.282 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:06.279+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:06.330 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1186946 2024-09-21T18:09:06.337 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:06.332+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 108.022885819, 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:09:06.357 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:06.362 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.183033823, 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.079313375, 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.078538632, 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.371 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:09:06.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:09:06.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.375 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.07789061299999, 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.064018918, 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.380 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:09:06.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:09:06.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:09:06.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.06107790199999, 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:09:06.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.388 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.05806105400001, 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.05584947200001, 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:09:06.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:09:06.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:09:06.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.95051238000001, 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.25909000599999, 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:09:06.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.256060006, 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.405 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:09:06.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:09:06.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:09:06.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 96.289496044000003, 2024-09-21T18:09:06.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:09:06.413 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 94.622660608999993, 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:09:06.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:09:06.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.319169528000003, 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.316872494999998, 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.425 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.314822047000007, 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:06.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:09:06.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:09:06.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:09:06.429 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.312666686, 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:09:06.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:09:06.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.311903928000007, 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:09:06.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.209998882999997, 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:06.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:06.442 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: finishing write tid 3 to smithi16030472-264 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: finishing write tid 4 to smithi16030472-264 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: oid 264 updating version 0 to 789 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: oid 264 version 789 is already newer than 788 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: oid 264 updating version 789 to 790 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 790 (ObjNum 1085 snap 0 seq_num 1085) dirty exists 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2290: left oid 264 (ObjNum 1085 snap 0 seq_num 1085) 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:2297: got (0) Success 2024-09-21T18:09:06.444 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 0 (ObjNum 454 snap 0 seq_num 454) dirty exists 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: finishing write tid 1 to smithi16030472-146 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: finishing write tid 2 to smithi16030472-146 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: finishing write tid 3 to smithi16030472-146 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: finishing write tid 4 to smithi16030472-146 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: oid 146 updating version 0 to 783 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: oid 146 version 783 is already newer than 782 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: oid 146 version 783 is already newer than 781 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 146 v 783 (ObjNum 1087 snap 0 seq_num 1087) dirty exists 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2304: left oid 146 (ObjNum 1087 snap 0 seq_num 1087) 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2293: finishing copy_from racing read to smithi16030472-494 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2301: got (0) Success 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 0 (ObjNum 1074 snap 0 seq_num 1074) dirty exists 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2293: finishing copy_from to smithi16030472-494 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 385 (ObjNum 591 snap 0 seq_num 591) dirty exists 2024-09-21T18:09:06.445 INFO:tasks.rados.rados.0.smithi160.stdout:2286: done (13 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2287: done (12 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2290: done (11 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2292: done (10 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2293: done (9 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2294: done (8 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2296: done (7 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2297: done (6 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2300: done (5 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2301: done (4 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2303: done (3 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2304: done (2 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2305: done (1 left) 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2306: cache_evict oid 264 snap -1 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2307: delete oid 240 current snap is 0 2024-09-21T18:09:06.446 INFO:tasks.rados.rados.0.smithi160.stdout:2306: got (16) Device or resource busy 2024-09-21T18:09:06.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:06.447 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:06.441+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:06.544 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:06.538+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:06.613 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.613 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:06.613 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:06.613 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.627 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:06.645 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:06.640+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:06.721 INFO:tasks.rados.rados.0.smithi160.stdout:2306: done (2 left) 2024-09-21T18:09:06.721 INFO:tasks.rados.rados.0.smithi160.stdout:2307: done (1 left) 2024-09-21T18:09:06.721 INFO:tasks.rados.rados.0.smithi160.stdout:2308: delete oid 424 current snap is 0 2024-09-21T18:09:06.746 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:06.742+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:06.751 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.751 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:06.751 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:06.751 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:06.751 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.765 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:06.795 INFO:tasks.rados.rados.0.smithi160.stdout:2302: got (0) Success 2024-09-21T18:09:06.796 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 0 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:09:06.846 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:06.838+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:06.890 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:06.890 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:06.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:06.890 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:09:06.890 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:06.899 INFO:tasks.rados.rados.0.smithi160.stdout:2302: done (1 left) 2024-09-21T18:09:06.899 INFO:tasks.rados.rados.0.smithi160.stdout:2308: done (0 left) 2024-09-21T18:09:06.899 INFO:tasks.rados.rados.0.smithi160.stdout:2309: write (excl) oid 181 current snap is 0 2024-09-21T18:09:06.899 INFO:tasks.rados.rados.0.smithi160.stdout:2309: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-21T18:09:06.904 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:06.923 INFO:tasks.rados.rados.0.smithi160.stdout:2309: writing smithi16030472-181 from 736437 to 1471575 tid 1 2024-09-21T18:09:06.938 INFO:tasks.rados.rados.0.smithi160.stdout:2309: writing smithi16030472-181 from 2176090 to 2336437 tid 2 2024-09-21T18:09:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:2310: write (excl) oid 419 current snap is 0 2024-09-21T18:09:06.939 INFO:tasks.rados.rados.0.smithi160.stdout:2310: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-21T18:09:06.949 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:06.947+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:06.954 INFO:tasks.rados.rados.0.smithi160.stdout:2310: writing smithi16030472-419 from 753244 to 1163631 tid 1 2024-09-21T18:09:06.968 INFO:tasks.rados.rados.0.smithi160.stdout:2310: writing smithi16030472-419 from 1834572 to 2353244 tid 2 2024-09-21T18:09:06.968 INFO:tasks.rados.rados.0.smithi160.stdout:2311: write (excl) oid 463 current snap is 0 2024-09-21T18:09:06.968 INFO:tasks.rados.rados.0.smithi160.stdout:2311: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-21T18:09:06.979 INFO:tasks.rados.rados.0.smithi160.stdout:2311: writing smithi16030472-463 from 770051 to 1255687 tid 1 2024-09-21T18:09:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:2311: writing smithi16030472-463 from 1776701 to 2287748 tid 2 2024-09-21T18:09:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:2312: delete oid 5 current snap is 0 2024-09-21T18:09:07.027 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.027 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:07.027 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:07.027 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:07.047 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:07.042+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:2309: finishing write tid 1 to smithi16030472-181 2024-09-21T18:09:07.143 INFO:tasks.rados.rados.0.smithi160.stdout:2310: finishing write tid 1 to smithi16030472-419 2024-09-21T18:09:07.143 INFO:tasks.rados.rados.0.smithi160.stdout:2310: finishing write tid 2 to smithi16030472-419 2024-09-21T18:09:07.143 INFO:tasks.rados.rados.0.smithi160.stdout:2311: finishing write tid 1 to smithi16030472-463 2024-09-21T18:09:07.148 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:07.142+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:07.154 INFO:tasks.rados.rados.0.smithi160.stdout:2311: finishing write tid 2 to smithi16030472-463 2024-09-21T18:09:07.170 INFO:tasks.rados.rados.0.smithi160.stdout:2309: finishing write tid 2 to smithi16030472-181 2024-09-21T18:09:07.170 INFO:tasks.rados.rados.0.smithi160.stdout:2309: finishing write tid 3 to smithi16030472-181 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:2309: finishing write tid 4 to smithi16030472-181 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:2309: oid 181 updating version 0 to 701 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:2309: oid 181 updating version 701 to 742 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:2309: oid 181 version 742 is already newer than 741 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 181 v 742 (ObjNum 1091 snap 0 seq_num 1091) dirty exists 2024-09-21T18:09:07.171 INFO:tasks.rados.rados.0.smithi160.stdout:2309: left oid 181 (ObjNum 1091 snap 0 seq_num 1091) 2024-09-21T18:09:07.184 INFO:tasks.rados.rados.0.smithi160.stdout:2310: finishing write tid 3 to smithi16030472-419 2024-09-21T18:09:07.201 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:07.202 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:07.202 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.211 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:09:07.217 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:07.249 INFO:tasks.rados.rados.0.smithi160.stdout:2310: finishing write tid 4 to smithi16030472-419 2024-09-21T18:09:07.250 INFO:tasks.rados.rados.0.smithi160.stdout:2310: oid 419 updating version 0 to 755 2024-09-21T18:09:07.250 INFO:tasks.rados.rados.0.smithi160.stdout:2310: oid 419 version 755 is already newer than 735 2024-09-21T18:09:07.250 INFO:tasks.rados.rados.0.smithi160.stdout:2310: oid 419 version 755 is already newer than 736 2024-09-21T18:09:07.250 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 419 v 755 (ObjNum 1092 snap 0 seq_num 1092) dirty exists 2024-09-21T18:09:07.251 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:07.245+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:07.251 INFO:tasks.rados.rados.0.smithi160.stdout:2310: left oid 419 (ObjNum 1092 snap 0 seq_num 1092) 2024-09-21T18:09:07.256 INFO:tasks.rados.rados.0.smithi160.stdout:2311: finishing write tid 3 to smithi16030472-463 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:2311: finishing write tid 4 to smithi16030472-463 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:2311: oid 463 updating version 0 to 704 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:2311: oid 463 version 704 is already newer than 702 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:2311: oid 463 version 704 is already newer than 703 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 463 v 704 (ObjNum 1093 snap 0 seq_num 1093) dirty exists 2024-09-21T18:09:07.257 INFO:tasks.rados.rados.0.smithi160.stdout:2311: left oid 463 (ObjNum 1093 snap 0 seq_num 1093) 2024-09-21T18:09:07.284 INFO:tasks.rados.rados.0.smithi160.stdout:2309: done (3 left) 2024-09-21T18:09:07.285 INFO:tasks.rados.rados.0.smithi160.stdout:2310: done (2 left) 2024-09-21T18:09:07.285 INFO:tasks.rados.rados.0.smithi160.stdout:2311: done (1 left) 2024-09-21T18:09:07.285 INFO:tasks.rados.rados.0.smithi160.stdout:2312: done (0 left) 2024-09-21T18:09:07.285 INFO:tasks.rados.rados.0.smithi160.stdout:2313: write oid 181 current snap is 0 2024-09-21T18:09:07.285 INFO:tasks.rados.rados.0.smithi160.stdout:2313: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-21T18:09:07.293 INFO:tasks.rados.rados.0.smithi160.stdout:2313: writing smithi16030472-181 from 403665 to 1156152 tid 1 2024-09-21T18:09:07.301 INFO:tasks.rados.rados.0.smithi160.stdout:2313: writing smithi16030472-181 from 1610018 to 2124734 tid 2 2024-09-21T18:09:07.301 INFO:tasks.rados.rados.0.smithi160.stdout:2314: copy_from oid 402 from oid 286 current snap is 0 2024-09-21T18:09:07.301 INFO:tasks.rados.rados.0.smithi160.stdout:2313: finishing write tid 1 to smithi16030472-181 2024-09-21T18:09:07.301 INFO:tasks.rados.rados.0.smithi160.stdout:2315: write oid 474 current snap is 0 2024-09-21T18:09:07.301 INFO:tasks.rados.rados.0.smithi160.stdout:2315: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-21T18:09:07.307 INFO:tasks.rados.rados.0.smithi160.stdout:2315: writing smithi16030472-474 from 420472 to 848208 tid 1 2024-09-21T18:09:07.315 INFO:tasks.rados.rados.0.smithi160.stdout:2315: writing smithi16030472-474 from 1268500 to 1926874 tid 2 2024-09-21T18:09:07.316 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:07.309+0000 7fb260c0e640 -1 osd.11 93 handle_osd_map injecting map crc failure 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2315: writing smithi16030472-474 from 2347953 to 2420472 tid 3 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2316: read oid 97 snap -1 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2316: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2317: cache_flush oid 39 snap -1 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2318: cache_evict oid 104 snap -1 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2319: write oid 76 current snap is 0 2024-09-21T18:09:07.320 INFO:tasks.rados.rados.0.smithi160.stdout:2319: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-21T18:09:07.327 INFO:tasks.rados.rados.0.smithi160.stdout:2319: writing smithi16030472-76 from 437279 to 940264 tid 1 2024-09-21T18:09:07.336 INFO:tasks.rados.rados.0.smithi160.stdout:2319: writing smithi16030472-76 from 1610629 to 2129014 tid 2 2024-09-21T18:09:07.336 INFO:tasks.rados.rados.0.smithi160.stdout:2320: copy_from oid 153 from oid 5 current snap is 0 2024-09-21T18:09:07.336 INFO:tasks.rados.rados.0.smithi160.stdout:2321: delete oid 165 current snap is 0 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.344 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.747122341999997, 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:09:07.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:09:07.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:09:07.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.317152394999994, 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:09:07.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:09:07.354 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.355 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.219632020000006, 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:09:07.356 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:07.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:09:07.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:09:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:09:07.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:09:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:09:07.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:09:07.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.21894777, 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.363 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.218114415000002, 2024-09-21T18:09:07.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:09:07.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:09:07.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.371 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.215395784999998, 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:09:07.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.375 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.119587378999995, 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:07.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:09:07.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:09:07.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:09:07.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.117282837999994, 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2900 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'245, mlcod=59'234)", 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.017002551000004, 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.22363195199999999, 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.384 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325921+0000", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325923+0000", 2024-09-21T18:09:07.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2929999999999999e-06 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325935+0000", 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1763000000000001e-05 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325952+0000", 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6722999999999999e-05 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325965+0000", 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3743e-05 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545223+0000", 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21925772700000001 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545279+0000", 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6085999999999998e-05 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549487+0000", 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.004207868 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549553+0000", 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.5748999999999997e-05 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.139654217999997, 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.392 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.038498535999999, 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:09:07.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:09:07.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:09:07.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.037702873999997, 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:09:07.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.036567054000002, 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.401 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:09:07.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:09:07.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.015849461000002, 2024-09-21T18:09:07.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:09:07.409 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.006496777000002, 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:07.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:09:07.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:09:07.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 54.001122129000002, 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.417 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 53.998573854, 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:07.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:09:07.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:09:07.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:09:07.421 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 47.770873317000003, 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.423 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:09:07.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 47.770207024000001, 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 47.770114911999997, 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.430 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.434 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:07.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:07.437 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:07.432+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:07.439 INFO:tasks.rados.rados.0.smithi160.stdout:2313: finishing write tid 2 to smithi16030472-181 2024-09-21T18:09:07.452 INFO:tasks.rados.rados.0.smithi160.stdout:2314: finishing copy_from racing read to smithi16030472-402 2024-09-21T18:09:07.537 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:07.532+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:07.554 INFO:tasks.rados.rados.0.smithi160.stdout:2314: finishing copy_from to smithi16030472-402 2024-09-21T18:09:07.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 402 v 454 (ObjNum 595 snap 0 seq_num 595) dirty exists 2024-09-21T18:09:07.558 INFO:tasks.rados.rados.0.smithi160.stdout:2313: finishing write tid 3 to smithi16030472-181 2024-09-21T18:09:07.559 INFO:tasks.rados.rados.0.smithi160.stdout:2315: finishing write tid 1 to smithi16030472-474 2024-09-21T18:09:07.566 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:07.566 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:07.566 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.580 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:07.594 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:07.595 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:07.616 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:07.629 INFO:tasks.rados.rados.0.smithi160.stdout:2313: finishing write tid 4 to smithi16030472-181 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:2313: oid 181 updating version 0 to 744 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:2313: oid 181 updating version 744 to 745 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:2313: oid 181 version 745 is already newer than 743 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 181 v 745 (ObjNum 1095 snap 0 seq_num 1095) dirty exists 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:2313: left oid 181 (ObjNum 1095 snap 0 seq_num 1095) 2024-09-21T18:09:07.630 INFO:tasks.rados.rados.0.smithi160.stdout:2315: finishing write tid 2 to smithi16030472-474 2024-09-21T18:09:07.634 INFO:tasks.rados.rados.0.smithi160.stdout:2316: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:07.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:07.637 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:07.630+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:07.656 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:07.657 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:07.674 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:07.674 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:07.693 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2312552 2024-09-21T18:09:07.695 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:07.698 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:07.698 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:07.698 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.713 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:07.716 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:07.731 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 635564 2024-09-21T18:09:07.737 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:07.737 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:07.737 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:07.730+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:07.760 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:07.760 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:07.779 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:07.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:07.803 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:07.803 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:07.822 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:07.832 INFO:tasks.rados.rados.0.smithi160.stdout:2318: got (0) Success 2024-09-21T18:09:07.832 INFO:tasks.rados.rados.0.smithi160.stdout:2315: finishing write tid 3 to smithi16030472-474 2024-09-21T18:09:07.832 INFO:tasks.rados.rados.0.smithi160.stdout:2315: finishing write tid 4 to smithi16030472-474 2024-09-21T18:09:07.832 INFO:tasks.rados.rados.0.smithi160.stdout:2315: finishing write tid 5 to smithi16030472-474 2024-09-21T18:09:07.832 INFO:tasks.rados.rados.0.smithi160.stdout:2315: oid 474 updating version 0 to 746 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2315: oid 474 updating version 746 to 748 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2315: oid 474 updating version 748 to 749 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2315: oid 474 version 749 is already newer than 747 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 474 v 749 (ObjNum 1096 snap 0 seq_num 1096) dirty exists 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2315: left oid 474 (ObjNum 1096 snap 0 seq_num 1096) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2319: finishing write tid 1 to smithi16030472-76 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2319: finishing write tid 2 to smithi16030472-76 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2313: done (8 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2314: done (7 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2315: done (6 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2316: done (5 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2318: done (4 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2321: done (3 left) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2322: read oid 345 snap -1 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2322: expect (ObjNum 1031 snap 0 seq_num 1031) 2024-09-21T18:09:07.833 INFO:tasks.rados.rados.0.smithi160.stdout:2323: write (excl) oid 166 current snap is 0 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2317: got (0) Success 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 0 (ObjNum 858 snap 0 seq_num 858) dirty exists 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2319: finishing write tid 3 to smithi16030472-76 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2320: finishing copy_from to smithi16030472-153 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2320: got expected ENOENT (src dne) 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2320: finishing copy_from racing read to smithi16030472-153 2024-09-21T18:09:07.834 INFO:tasks.rados.rados.0.smithi160.stdout:2323: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-21T18:09:07.838 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:07.835+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:07.839 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 108.999687853, 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.58804555899999, 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.844 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:09:07.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.585214687, 2024-09-21T18:09:07.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:09:07.852 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.581241779, 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:09:07.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:09:07.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:09:07.856 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.578124828, 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:09:07.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.576193824, 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.861 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:09:07.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:09:07.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.573087065, 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.865 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:09:07.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:09:07.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:09:07.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.56992938099999, 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.869 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.56769633, 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:09:07.873 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:07.877 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.565444657, 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:09:07.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:09:07.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.881 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.472867069, 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.696197693, 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.686690108999997, 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.893 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.57986013, 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:09:07.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.897 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.246488577999997, 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:07.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:09:07.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:09:07.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:09:07.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.474497108999998, 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.592348654000006, 2024-09-21T18:09:07.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 49.052224205999998, 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.915 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 31.742265746000001, 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.919 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 28.497863434999999, 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:07.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:09:07.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:09:07.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.923 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:07.924 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:07.924 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:07.924 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:07.926 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:07.927 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:07.927 INFO:tasks.rados.rados.0.smithi160.stdout:2323: writing smithi16030472-166 from 470893 to 1124376 tid 1 2024-09-21T18:09:07.927 INFO:tasks.rados.rados.0.smithi160.stdout:2323: writing smithi16030472-166 from 1727593 to 2249647 tid 2 2024-09-21T18:09:07.927 INFO:tasks.rados.rados.0.smithi160.stdout:2317: done (4 left) 2024-09-21T18:09:07.927 INFO:tasks.rados.rados.0.smithi160.stdout:2320: done (3 left) 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2324: cache_try_flush oid 80 snap -1 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: finishing write tid 4 to smithi16030472-76 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: oid 76 updating version 0 to 750 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: oid 76 updating version 750 to 752 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: oid 76 version 752 is already newer than 751 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 752 (ObjNum 1097 snap 0 seq_num 1097) dirty exists 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: left oid 76 (ObjNum 1097 snap 0 seq_num 1097) 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2319: done (3 left) 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2325: read oid 143 snap -1 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2325: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2326: cache_evict oid 371 snap -1 2024-09-21T18:09:07.928 INFO:tasks.rados.rados.0.smithi160.stdout:2322: expect (ObjNum 1031 snap 0 seq_num 1031) 2024-09-21T18:09:07.928 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:07.929 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:07.929 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:07.974 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:07.966+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2205492 2024-09-21T18:09:08.012 INFO:tasks.rados.rados.0.smithi160.stdout:2324: got (0) Success 2024-09-21T18:09:08.012 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 0 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:09:08.012 INFO:tasks.rados.rados.0.smithi160.stdout:2326: got (16) Device or resource busy 2024-09-21T18:09:08.012 INFO:tasks.rados.rados.0.smithi160.stdout:2325: expect (ObjNum 142 snap 0 seq_num 142) 2024-09-21T18:09:08.040 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:08.034+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:08.062 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.062 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.070 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 981726 2024-09-21T18:09:08.077 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:08.086 INFO:tasks.rados.rados.0.smithi160.stdout:2323: finishing write tid 1 to smithi16030472-166 2024-09-21T18:09:08.086 INFO:tasks.rados.rados.0.smithi160.stdout:2322: done (4 left) 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2324: done (3 left) 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2325: done (2 left) 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2326: done (1 left) 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2327: copy_from oid 363 from oid 62 current snap is 0 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2328: cache_flush oid 310 snap -1 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2329: write oid 49 current snap is 0 2024-09-21T18:09:08.087 INFO:tasks.rados.rados.0.smithi160.stdout:2329: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-21T18:09:08.097 INFO:tasks.rados.rados.0.smithi160.stdout:2329: writing smithi16030472-49 from 487700 to 1216432 tid 1 2024-09-21T18:09:08.109 INFO:tasks.rados.rados.0.smithi160.stdout:2329: writing smithi16030472-49 from 1786075 to 2451787 tid 2 2024-09-21T18:09:08.109 INFO:tasks.rados.rados.0.smithi160.stdout:2330: read oid 32 snap -1 2024-09-21T18:09:08.109 INFO:tasks.rados.rados.0.smithi160.stdout:2330: expect (ObjNum 846 snap 0 seq_num 846) 2024-09-21T18:09:08.109 INFO:tasks.rados.rados.0.smithi160.stdout:2331: read oid 130 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2331: expect deleted 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2332: read oid 94 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2332: expect deleted 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2333: read oid 231 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2333: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2334: read oid 370 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2334: expect (ObjNum 731 snap 0 seq_num 731) 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2335: cache_evict oid 191 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2336: read oid 194 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2336: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2337: cache_try_flush oid 179 snap -1 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2329: finishing write tid 1 to smithi16030472-49 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2338: copy_from oid 284 from oid 97 current snap is 0 2024-09-21T18:09:08.110 INFO:tasks.rados.rados.0.smithi160.stdout:2339: write oid 68 current snap is 0 2024-09-21T18:09:08.111 INFO:tasks.rados.rados.0.smithi160.stdout:2339: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-09-21T18:09:08.114 INFO:tasks.rados.rados.0.smithi160.stdout:2339: writing smithi16030472-68 from 504507 to 908488 tid 1 2024-09-21T18:09:08.119 INFO:tasks.rados.rados.0.smithi160.stdout:2339: writing smithi16030472-68 from 1328204 to 1737574 tid 2 2024-09-21T18:09:08.123 INFO:tasks.rados.rados.0.smithi160.stdout:2339: writing smithi16030472-68 from 2252362 to 2504507 tid 3 2024-09-21T18:09:08.123 INFO:tasks.rados.rados.0.smithi160.stdout:2340: write (excl) oid 78 current snap is 0 2024-09-21T18:09:08.123 INFO:tasks.rados.rados.0.smithi160.stdout:2340: seq_num 1102 ranges {521314=479230,1786686=669381} 2024-09-21T18:09:08.129 INFO:tasks.rados.rados.0.smithi160.stdout:2340: writing smithi16030472-78 from 521314 to 1000544 tid 1 2024-09-21T18:09:08.137 INFO:tasks.rados.rados.0.smithi160.stdout:2340: writing smithi16030472-78 from 1786686 to 2456067 tid 2 2024-09-21T18:09:08.137 INFO:tasks.rados.rados.0.smithi160.stdout:2341: cache_evict oid 381 snap -1 2024-09-21T18:09:08.137 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:08.141 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:08.136+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:08.195 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.196 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.210 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:08.210 INFO:tasks.rados.rados.0.smithi160.stdout:2333: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:08.210 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.242 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:08.234+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:08.272 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3033288 2024-09-21T18:09:08.318 INFO:tasks.rados.rados.0.smithi160.stdout:2335: got (2) No such file or directory 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: finishing write tid 2 to smithi16030472-49 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: finishing write tid 3 to smithi16030472-49 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: finishing write tid 4 to smithi16030472-49 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: oid 49 updating version 0 to 793 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: oid 49 version 793 is already newer than 792 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: oid 49 version 793 is already newer than 791 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 793 (ObjNum 1100 snap 0 seq_num 1100) dirty exists 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2329: left oid 49 (ObjNum 1100 snap 0 seq_num 1100) 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:2330: expect (ObjNum 846 snap 0 seq_num 846) 2024-09-21T18:09:08.319 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.329 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.329 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.343 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:08.344 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:08.342+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:08.400 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.443 INFO:tasks.rados.rados.0.smithi160.stdout:2336: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:08.444 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 552217 2024-09-21T18:09:08.444 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:08.440+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:08.468 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.469 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.484 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:08.544 INFO:tasks.rados.rados.0.smithi160.stdout:2323: finishing write tid 2 to smithi16030472-166 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: finishing write tid 3 to smithi16030472-166 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: finishing write tid 4 to smithi16030472-166 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: oid 166 updating version 0 to 786 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: oid 166 version 786 is already newer than 784 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: oid 166 version 786 is already newer than 785 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 166 v 786 (ObjNum 1099 snap 0 seq_num 1099) dirty exists 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2323: left oid 166 (ObjNum 1099 snap 0 seq_num 1099) 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2328: got (0) Success 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 310 v 0 (ObjNum 145 snap 0 seq_num 145) dirty exists 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2339: finishing write tid 1 to smithi16030472-68 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2339: finishing write tid 2 to smithi16030472-68 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2339: finishing write tid 3 to smithi16030472-68 2024-09-21T18:09:08.545 INFO:tasks.rados.rados.0.smithi160.stdout:2339: finishing write tid 4 to smithi16030472-68 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: finishing write tid 5 to smithi16030472-68 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: oid 68 updating version 0 to 797 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: oid 68 version 797 is already newer than 795 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: oid 68 version 797 is already newer than 794 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: oid 68 version 797 is already newer than 796 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 68 v 797 (ObjNum 1101 snap 0 seq_num 1101) dirty exists 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2339: left oid 68 (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2323: done (15 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2328: done (14 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2329: done (13 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2330: done (12 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2332: done (11 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2333: done (10 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2335: done (9 left) 2024-09-21T18:09:08.546 INFO:tasks.rados.rados.0.smithi160.stdout:2336: done (8 left) 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2339: done (7 left) 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2342: cache_try_flush oid 442 snap -1 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2343: read oid 91 snap -1 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2343: expect (ObjNum 956 snap 0 seq_num 956) 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2344: read oid 400 snap -1 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2344: expect (ObjNum 533 snap 0 seq_num 533) 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2345: cache_try_flush oid 37 snap -1 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2346: write oid 88 current snap is 0 2024-09-21T18:09:08.547 INFO:tasks.rados.rados.0.smithi160.stdout:2346: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-21T18:09:08.548 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:08.539+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:08.553 INFO:tasks.rados.rados.0.smithi160.stdout:2346: writing smithi16030472-88 from 538121 to 1208953 tid 1 2024-09-21T18:09:08.568 INFO:tasks.rados.rados.0.smithi160.stdout:2346: writing smithi16030472-88 from 1961521 to 2374560 tid 2 2024-09-21T18:09:08.568 INFO:tasks.rados.rados.0.smithi160.stdout:2347: cache_try_flush oid 405 snap -1 2024-09-21T18:09:08.568 INFO:tasks.rados.rados.0.smithi160.stdout:2343: expect (ObjNum 956 snap 0 seq_num 956) 2024-09-21T18:09:08.591 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.601 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.601 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.614 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:08.628 INFO:tasks.rados.rados.0.smithi160.stdout:2347: got (0) Success 2024-09-21T18:09:08.628 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 405 v 0 (ObjNum 129 snap 0 seq_num 3130056984) clean dne 2024-09-21T18:09:08.629 INFO:tasks.rados.rados.0.smithi160.stdout:2334: expect (ObjNum 731 snap 0 seq_num 731) 2024-09-21T18:09:08.645 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:08.638+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:08.691 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.723 INFO:tasks.rados.rados.0.smithi160.stdout:2337: got (0) Success 2024-09-21T18:09:08.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 0 (ObjNum 178 snap 0 seq_num 178) dirty exists 2024-09-21T18:09:08.723 INFO:tasks.rados.rados.0.smithi160.stdout:2327: finishing copy_from racing read to smithi16030472-363 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2345: got (0) Success 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 37 v 0 (ObjNum 129 snap 0 seq_num 3127277400) clean dne 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2334: done (12 left) 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2337: done (11 left) 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2343: done (10 left) 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2345: done (9 left) 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2347: done (8 left) 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2348: delete oid 246 current snap is 0 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:2327: finishing copy_from to smithi16030472-363 2024-09-21T18:09:08.724 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 656 (ObjNum 887 snap 0 seq_num 887) dirty exists 2024-09-21T18:09:08.732 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.732 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.745 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:08.746 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:08.740+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:08.757 INFO:tasks.rados.rados.0.smithi160.stdout:2346: finishing write tid 1 to smithi16030472-88 2024-09-21T18:09:08.757 INFO:tasks.rados.rados.0.smithi160.stdout:2346: finishing write tid 2 to smithi16030472-88 2024-09-21T18:09:08.764 INFO:tasks.rados.rados.0.smithi160.stdout:2342: got (0) Success 2024-09-21T18:09:08.764 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 0 (ObjNum 629 snap 0 seq_num 629) dirty exists 2024-09-21T18:09:08.803 INFO:tasks.rados.rados.0.smithi160.stdout:2327: done (8 left) 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2331: done (7 left) 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2342: done (6 left) 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2348: done (5 left) 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2349: read oid 378 snap -1 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2349: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2350: cache_flush oid 358 snap -1 2024-09-21T18:09:08.804 INFO:tasks.rados.rados.0.smithi160.stdout:2351: delete oid 58 current snap is 0 2024-09-21T18:09:08.809 INFO:tasks.rados.rados.0.smithi160.stdout:2351: done (7 left) 2024-09-21T18:09:08.809 INFO:tasks.rados.rados.0.smithi160.stdout:2352: read oid 148 snap -1 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2352: expect (ObjNum 851 snap 0 seq_num 851) 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2353: read oid 357 snap -1 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2353: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2354: cache_evict oid 383 snap -1 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2355: copy_from oid 172 from oid 323 current snap is 0 2024-09-21T18:09:08.810 INFO:tasks.rados.rados.0.smithi160.stdout:2356: delete oid 461 current snap is 0 2024-09-21T18:09:08.814 INFO:tasks.rados.rados.0.smithi160.stdout:2356: done (11 left) 2024-09-21T18:09:08.814 INFO:tasks.rados.rados.0.smithi160.stdout:2357: delete oid 191 current snap is 0 2024-09-21T18:09:08.817 INFO:tasks.rados.rados.0.smithi160.stdout:2352: expect (ObjNum 851 snap 0 seq_num 851) 2024-09-21T18:09:08.847 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:08.841+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:08.878 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:08.878 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:08.891 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:08.910 INFO:tasks.rados.rados.0.smithi160.stdout:2338: finishing copy_from racing read to smithi16030472-284 2024-09-21T18:09:08.910 INFO:tasks.rados.rados.0.smithi160.stdout:2340: finishing write tid 1 to smithi16030472-78 2024-09-21T18:09:08.910 INFO:tasks.rados.rados.0.smithi160.stdout:2344: expect (ObjNum 533 snap 0 seq_num 533) 2024-09-21T18:09:08.910 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:08.953 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:08.949+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:08.957 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:09.004 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:09.024 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.024 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.037 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:09.049 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:09.046+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:09.050 INFO:tasks.rados.rados.0.smithi160.stdout:2349: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:09:09.050 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 265655 2024-09-21T18:09:09.098 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: finishing write tid 3 to smithi16030472-88 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: finishing write tid 4 to smithi16030472-88 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: oid 88 updating version 0 to 758 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: oid 88 updating version 758 to 759 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: oid 88 version 759 is already newer than 757 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 759 (ObjNum 1103 snap 0 seq_num 1103) dirty exists 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: left oid 88 (ObjNum 1103 snap 0 seq_num 1103) 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2350: got (0) Success 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 358 v 0 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2344: done (11 left) 2024-09-21T18:09:09.123 INFO:tasks.rados.rados.0.smithi160.stdout:2346: done (10 left) 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2349: done (9 left) 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2350: done (8 left) 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2352: done (7 left) 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2357: done (6 left) 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2358: copy_from oid 161 from oid 167 current snap is 0 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2340: finishing write tid 2 to smithi16030472-78 2024-09-21T18:09:09.124 INFO:tasks.rados.rados.0.smithi160.stdout:2353: expect (ObjNum 356 snap 0 seq_num 356) 2024-09-21T18:09:09.126 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1828385 2024-09-21T18:09:09.150 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:09.142+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:09.175 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.175 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2354: got (16) Device or resource busy 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2355: finishing copy_from racing read to smithi16030472-172 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2353: done (6 left) 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2354: done (5 left) 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2359: cache_flush oid 34 snap -1 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2340: finishing write tid 3 to smithi16030472-78 2024-09-21T18:09:09.228 INFO:tasks.rados.rados.0.smithi160.stdout:2340: finishing write tid 4 to smithi16030472-78 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2340: oid 78 updating version 0 to 787 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2340: oid 78 updating version 787 to 788 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2340: oid 78 updating version 788 to 789 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 789 (ObjNum 1102 snap 0 seq_num 1102) dirty exists 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2340: left oid 78 (ObjNum 1102 snap 0 seq_num 1102) 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2340: done (5 left) 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2360: write (excl) oid 332 current snap is 0 2024-09-21T18:09:09.229 INFO:tasks.rados.rados.0.smithi160.stdout:2360: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-21T18:09:09.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:09:09.248 INFO:tasks.rados.rados.0.smithi160.stdout:2360: writing smithi16030472-332 from 622156 to 1269233 tid 1 2024-09-21T18:09:09.250 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:09.247+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:09.267 INFO:tasks.rados.rados.0.smithi160.stdout:2360: writing smithi16030472-332 from 1737578 to 2417966 tid 2 2024-09-21T18:09:09.267 INFO:tasks.rados.rados.0.smithi160.stdout:2361: write oid 30 current snap is 0 2024-09-21T18:09:09.267 INFO:tasks.rados.rados.0.smithi160.stdout:2361: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-21T18:09:09.286 INFO:tasks.rados.rados.0.smithi160.stdout:2361: writing smithi16030472-30 from 638963 to 1361289 tid 1 2024-09-21T18:09:09.298 INFO:tasks.rados.rados.0.smithi160.stdout:2361: writing smithi16030472-30 from 2079707 to 2503753 tid 2 2024-09-21T18:09:09.298 INFO:tasks.rados.rados.0.smithi160.stdout:2362: write oid 273 current snap is 0 2024-09-21T18:09:09.298 INFO:tasks.rados.rados.0.smithi160.stdout:2362: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-09-21T18:09:09.308 INFO:tasks.rados.rados.0.smithi160.stdout:2362: writing smithi16030472-273 from 655770 to 1169698 tid 1 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2362: writing smithi16030472-273 from 1854542 to 2538599 tid 2 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2363: cache_evict oid 186 snap -1 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2364: cache_flush oid 219 snap -1 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2365: cache_flush oid 89 snap -1 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2338: finishing copy_from to smithi16030472-284 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 284 v 744 (ObjNum 1052 snap 0 seq_num 1052) dirty exists 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2341: got (0) Success 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2355: finishing copy_from to smithi16030472-172 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 172 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2338: done (10 left) 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2341: done (9 left) 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2355: done (8 left) 2024-09-21T18:09:09.320 INFO:tasks.rados.rados.0.smithi160.stdout:2366: read oid 482 snap -1 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2366: expect deleted 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2367: read oid 261 snap -1 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2367: expect deleted 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2368: cache_flush oid 127 snap -1 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2358: finishing copy_from to smithi16030472-161 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2358: got expected ENOENT (src dne) 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2358: finishing copy_from racing read to smithi16030472-161 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2359: got (0) Success 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 0 (ObjNum 860 snap 0 seq_num 860) dirty exists 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2358: done (10 left) 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2359: done (9 left) 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2369: write oid 218 current snap is 0 2024-09-21T18:09:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:2369: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-21T18:09:09.327 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.328 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.329 INFO:tasks.rados.rados.0.smithi160.stdout:2369: writing smithi16030472-218 from 672577 to 1261754 tid 1 2024-09-21T18:09:09.339 INFO:tasks.rados.rados.0.smithi160.stdout:2369: writing smithi16030472-218 from 1913024 to 2340739 tid 2 2024-09-21T18:09:09.339 INFO:tasks.rados.rados.0.smithi160.stdout:2370: write (excl) oid 461 current snap is 0 2024-09-21T18:09:09.339 INFO:tasks.rados.rados.0.smithi160.stdout:2370: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-21T18:09:09.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:09.350 INFO:tasks.rados.rados.0.smithi160.stdout:2370: writing smithi16030472-461 from 689384 to 1353810 tid 1 2024-09-21T18:09:09.351 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:09.347+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:09.361 INFO:tasks.rados.rados.0.smithi160.stdout:2370: writing smithi16030472-461 from 1971506 to 2659232 tid 2 2024-09-21T18:09:09.361 INFO:tasks.rados.rados.0.smithi160.stdout:2371: read oid 63 snap -1 2024-09-21T18:09:09.361 INFO:tasks.rados.rados.0.smithi160.stdout:2371: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:09:09.361 INFO:tasks.rados.rados.0.smithi160.stdout:2372: write oid 428 current snap is 0 2024-09-21T18:09:09.361 INFO:tasks.rados.rados.0.smithi160.stdout:2372: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-21T18:09:09.371 INFO:tasks.rados.rados.0.smithi160.stdout:2372: writing smithi16030472-428 from 706191 to 1445866 tid 1 2024-09-21T18:09:09.377 INFO:tasks.rados.rados.0.smithi160.stdout:2372: writing smithi16030472-428 from 1913635 to 2345019 tid 2 2024-09-21T18:09:09.377 INFO:tasks.rados.rados.0.smithi160.stdout:2373: read oid 145 snap -1 2024-09-21T18:09:09.377 INFO:tasks.rados.rados.0.smithi160.stdout:2373: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:09.378 INFO:tasks.rados.rados.0.smithi160.stdout:2374: cache_evict oid 140 snap -1 2024-09-21T18:09:09.378 INFO:tasks.rados.rados.0.smithi160.stdout:2375: write oid 34 current snap is 0 2024-09-21T18:09:09.378 INFO:tasks.rados.rados.0.smithi160.stdout:2375: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-21T18:09:09.384 INFO:tasks.rados.rados.0.smithi160.stdout:2375: writing smithi16030472-34 from 722998 to 1137922 tid 1 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout:2375: writing smithi16030472-34 from 1572117 to 2147159 tid 2 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout:2366: done (15 left) 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout:2376: read oid 85 snap -1 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout:2376: expect (ObjNum 963 snap 0 seq_num 963) 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:09.391 INFO:tasks.rados.rados.0.smithi160.stdout:2360: finishing write tid 1 to smithi16030472-332 2024-09-21T18:09:09.392 INFO:tasks.rados.rados.0.smithi160.stdout:2368: got (0) Success 2024-09-21T18:09:09.392 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 127 v 0 (ObjNum 326 snap 0 seq_num 326) dirty exists 2024-09-21T18:09:09.392 INFO:tasks.rados.rados.0.smithi160.stdout:2368: done (15 left) 2024-09-21T18:09:09.392 INFO:tasks.rados.rados.0.smithi160.stdout:2377: read oid 84 snap -1 2024-09-21T18:09:09.392 INFO:tasks.rados.rados.0.smithi160.stdout:2377: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-21T18:09:09.393 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:09.421 INFO:tasks.rados.rados.0.smithi160.stdout:2367: done (15 left) 2024-09-21T18:09:09.421 INFO:tasks.rados.rados.0.smithi160.stdout:2378: write oid 53 current snap is 0 2024-09-21T18:09:09.421 INFO:tasks.rados.rados.0.smithi160.stdout:2378: seq_num 1115 ranges {739805=490173,1630599=435053,2636284=103521} 2024-09-21T18:09:09.429 INFO:tasks.rados.rados.0.smithi160.stdout:2378: writing smithi16030472-53 from 739805 to 1229978 tid 1 2024-09-21T18:09:09.435 INFO:tasks.rados.rados.0.smithi160.stdout:2378: writing smithi16030472-53 from 1630599 to 2065652 tid 2 2024-09-21T18:09:09.439 INFO:tasks.rados.rados.0.smithi160.stdout:2378: writing smithi16030472-53 from 2636284 to 2739805 tid 3 2024-09-21T18:09:09.439 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:2374: got (16) Device or resource busy 2024-09-21T18:09:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:2374: done (15 left) 2024-09-21T18:09:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:2379: read oid 134 snap -1 2024-09-21T18:09:09.444 INFO:tasks.rados.rados.0.smithi160.stdout:2379: expect (ObjNum 650 snap 0 seq_num 650) 2024-09-21T18:09:09.444 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:09.452 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:09.446+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:09.474 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.474 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:09.553 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:09.548+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:09.604 INFO:tasks.rados.rados.0.smithi160.stdout:2371: expect (ObjNum 415 snap 0 seq_num 415) 2024-09-21T18:09:09.653 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.653 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.653 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:09.649+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:09.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:09.669 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1642354 2024-09-21T18:09:09.707 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1045037 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: finishing write tid 1 to smithi16030472-428 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: finishing write tid 2 to smithi16030472-428 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: finishing write tid 3 to smithi16030472-428 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: finishing write tid 4 to smithi16030472-428 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: oid 428 updating version 0 to 753 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: oid 428 updating version 753 to 754 2024-09-21T18:09:09.729 INFO:tasks.rados.rados.0.smithi160.stdout:2372: oid 428 updating version 754 to 755 2024-09-21T18:09:09.730 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 428 v 755 (ObjNum 1113 snap 0 seq_num 1113) dirty exists 2024-09-21T18:09:09.730 INFO:tasks.rados.rados.0.smithi160.stdout:2372: left oid 428 (ObjNum 1113 snap 0 seq_num 1113) 2024-09-21T18:09:09.730 INFO:tasks.rados.rados.0.smithi160.stdout:2373: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:09.734 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 931884 2024-09-21T18:09:09.754 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:09.751+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:09.757 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:09.795 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.795 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:09.820 INFO:tasks.rados.rados.0.smithi160.stdout:2370: finishing write tid 1 to smithi16030472-461 2024-09-21T18:09:09.820 INFO:tasks.rados.rados.0.smithi160.stdout:2365: got (0) Success 2024-09-21T18:09:09.820 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 89 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:09.821 INFO:tasks.rados.rados.0.smithi160.stdout:2377: expect (ObjNum 394 snap 0 seq_num 394) 2024-09-21T18:09:09.855 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:09.850+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:09.894 INFO:tasks.rados.rados.0.smithi160.stdout:2360: finishing write tid 2 to smithi16030472-332 2024-09-21T18:09:09.894 INFO:tasks.rados.rados.0.smithi160.stdout:2360: finishing write tid 3 to smithi16030472-332 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2360: finishing write tid 4 to smithi16030472-332 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2360: oid 332 updating version 0 to 791 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2360: oid 332 updating version 791 to 792 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2360: oid 332 version 792 is already newer than 790 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 792 (ObjNum 1108 snap 0 seq_num 1108) dirty exists 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2360: left oid 332 (ObjNum 1108 snap 0 seq_num 1108) 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2363: got (0) Success 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: finishing write tid 1 to smithi16030472-218 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: finishing write tid 2 to smithi16030472-218 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: finishing write tid 3 to smithi16030472-218 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: finishing write tid 4 to smithi16030472-218 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: oid 218 updating version 0 to 794 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: oid 218 version 794 is already newer than 793 2024-09-21T18:09:09.895 INFO:tasks.rados.rados.0.smithi160.stdout:2369: oid 218 updating version 794 to 795 2024-09-21T18:09:09.896 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 218 v 795 (ObjNum 1111 snap 0 seq_num 1111) dirty exists 2024-09-21T18:09:09.896 INFO:tasks.rados.rados.0.smithi160.stdout:2369: left oid 218 (ObjNum 1111 snap 0 seq_num 1111) 2024-09-21T18:09:09.896 INFO:tasks.rados.rados.0.smithi160.stdout:2379: expect (ObjNum 650 snap 0 seq_num 650) 2024-09-21T18:09:09.920 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1994083 2024-09-21T18:09:09.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:09.939 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:09.939 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:09.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:09.956 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:09.949+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:09.961 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1464555 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2361: finishing write tid 1 to smithi16030472-30 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2361: finishing write tid 2 to smithi16030472-30 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2361: finishing write tid 3 to smithi16030472-30 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2362: finishing write tid 1 to smithi16030472-273 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2362: finishing write tid 2 to smithi16030472-273 2024-09-21T18:09:10.041 INFO:tasks.rados.rados.0.smithi160.stdout:2375: finishing write tid 1 to smithi16030472-34 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: finishing write tid 2 to smithi16030472-34 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: finishing write tid 2 to smithi16030472-461 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: finishing write tid 3 to smithi16030472-461 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: finishing write tid 3 to smithi16030472-34 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: finishing write tid 4 to smithi16030472-34 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: oid 34 updating version 0 to 758 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: oid 34 version 758 is already newer than 756 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: oid 34 version 758 is already newer than 757 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 758 (ObjNum 1114 snap 0 seq_num 1114) dirty exists 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2375: left oid 34 (ObjNum 1114 snap 0 seq_num 1114) 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: finishing write tid 4 to smithi16030472-461 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: oid 461 updating version 0 to 801 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: oid 461 updating version 801 to 802 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:2370: oid 461 version 802 is already newer than 800 2024-09-21T18:09:10.042 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 461 v 802 (ObjNum 1112 snap 0 seq_num 1112) dirty exists 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2370: left oid 461 (ObjNum 1112 snap 0 seq_num 1112) 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2361: finishing write tid 4 to smithi16030472-30 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2361: oid 30 updating version 0 to 740 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2361: oid 30 updating version 740 to 741 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2361: oid 30 updating version 741 to 742 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 30 v 742 (ObjNum 1109 snap 0 seq_num 1109) dirty exists 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2361: left oid 30 (ObjNum 1109 snap 0 seq_num 1109) 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: finishing write tid 3 to smithi16030472-273 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: finishing write tid 4 to smithi16030472-273 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: oid 273 updating version 0 to 743 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: oid 273 updating version 743 to 744 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: oid 273 updating version 744 to 760 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 273 v 760 (ObjNum 1110 snap 0 seq_num 1110) dirty exists 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2362: left oid 273 (ObjNum 1110 snap 0 seq_num 1110) 2024-09-21T18:09:10.043 INFO:tasks.rados.rados.0.smithi160.stdout:2364: got (0) Success 2024-09-21T18:09:10.044 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 219 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:09:10.044 INFO:tasks.rados.rados.0.smithi160.stdout:2376: expect (ObjNum 963 snap 0 seq_num 963) 2024-09-21T18:09:10.044 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:10.056 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:10.051+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:10.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:10.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:10.100 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:10.100 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:10.121 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:10.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:10.132 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:10.132 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:10.138 INFO:tasks.rados.rados.0.smithi160.stdout:2378: finishing write tid 1 to smithi16030472-53 2024-09-21T18:09:10.138 INFO:tasks.rados.rados.0.smithi160.stdout:2360: done (15 left) 2024-09-21T18:09:10.138 INFO:tasks.rados.rados.0.smithi160.stdout:2361: done (14 left) 2024-09-21T18:09:10.138 INFO:tasks.rados.rados.0.smithi160.stdout:2362: done (13 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2363: done (12 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2364: done (11 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2365: done (10 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2369: done (9 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2370: done (8 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2371: done (7 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2372: done (6 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2373: done (5 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2375: done (4 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2376: done (3 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2377: done (2 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2379: done (1 left) 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2380: cache_evict oid 194 snap -1 2024-09-21T18:09:10.139 INFO:tasks.rados.rados.0.smithi160.stdout:2381: read oid 416 snap -1 2024-09-21T18:09:10.140 INFO:tasks.rados.rados.0.smithi160.stdout:2381: expect (ObjNum 1084 snap 0 seq_num 1084) 2024-09-21T18:09:10.140 INFO:tasks.rados.rados.0.smithi160.stdout:2382: write oid 437 current snap is 0 2024-09-21T18:09:10.140 INFO:tasks.rados.rados.0.smithi160.stdout:2382: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-21T18:09:10.141 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:10.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:10.157 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:10.152+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:10.159 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:2382: writing smithi16030472-437 from 756612 to 1322034 tid 1 2024-09-21T18:09:10.179 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:10.182 INFO:tasks.rados.rados.0.smithi160.stdout:2382: writing smithi16030472-437 from 2089081 to 2667792 tid 2 2024-09-21T18:09:10.182 INFO:tasks.rados.rados.0.smithi160.stdout:2383: delete oid 329 current snap is 0 2024-09-21T18:09:10.200 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:10.200 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:10.217 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:09:10.240 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:10.241 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:10.241 INFO:tasks.rados.rados.0.smithi160.stdout:2378: finishing write tid 2 to smithi16030472-53 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: finishing write tid 3 to smithi16030472-53 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: finishing write tid 4 to smithi16030472-53 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: finishing write tid 5 to smithi16030472-53 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: oid 53 updating version 0 to 760 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: oid 53 updating version 760 to 761 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: oid 53 updating version 761 to 762 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: oid 53 version 762 is already newer than 759 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 53 v 762 (ObjNum 1115 snap 0 seq_num 1115) dirty exists 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2378: left oid 53 (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:2381: expect (ObjNum 1084 snap 0 seq_num 1084) 2024-09-21T18:09:10.242 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:10.258 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:10.255+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:10.261 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:10.267 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:10.267 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:10.268 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:10.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:10.283 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:10.284 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:10.294 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1742786 2024-09-21T18:09:10.301 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:10.301 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:10.322 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:10.342 INFO:tasks.rados.rados.0.smithi160.stdout:2380: got (0) Success 2024-09-21T18:09:10.354 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:10.358 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:10.353+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:10.371 INFO:tasks.rados.rados.0.smithi160.stdout:2382: finishing write tid 1 to smithi16030472-437 2024-09-21T18:09:10.371 INFO:tasks.rados.rados.0.smithi160.stdout:2382: finishing write tid 2 to smithi16030472-437 2024-09-21T18:09:10.376 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:10.383 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:10.386 INFO:tasks.rados.rados.0.smithi160.stdout:2378: done (4 left) 2024-09-21T18:09:10.386 INFO:tasks.rados.rados.0.smithi160.stdout:2380: done (3 left) 2024-09-21T18:09:10.386 INFO:tasks.rados.rados.0.smithi160.stdout:2381: done (2 left) 2024-09-21T18:09:10.386 INFO:tasks.rados.rados.0.smithi160.stdout:2383: done (1 left) 2024-09-21T18:09:10.386 INFO:tasks.rados.rados.0.smithi160.stdout:2384: cache_flush oid 94 snap -1 2024-09-21T18:09:10.387 INFO:tasks.rados.rados.0.smithi160.stdout:2385: cache_evict oid 163 snap -1 2024-09-21T18:09:10.387 INFO:tasks.rados.rados.0.smithi160.stdout:2386: delete oid 321 current snap is 0 2024-09-21T18:09:10.387 INFO:tasks.rados.rados.0.smithi160.stdout:2384: got (0) Success 2024-09-21T18:09:10.387 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:10.387 INFO:tasks.rados.rados.0.smithi160.stdout:2385: got (2) No such file or directory 2024-09-21T18:09:10.400 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:10.401 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:10.401 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:10.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:10.459 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:10.455+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:10.490 INFO:tasks.rados.rados.0.smithi160.stdout:2384: done (3 left) 2024-09-21T18:09:10.491 INFO:tasks.rados.rados.0.smithi160.stdout:2385: done (2 left) 2024-09-21T18:09:10.491 INFO:tasks.rados.rados.0.smithi160.stdout:2386: done (1 left) 2024-09-21T18:09:10.491 INFO:tasks.rados.rados.0.smithi160.stdout:2387: write oid 421 current snap is 0 2024-09-21T18:09:10.491 INFO:tasks.rados.rados.0.smithi160.stdout:2387: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2024-09-21T18:09:10.505 INFO:tasks.rados.rados.0.smithi160.stdout:2387: writing smithi16030472-421 from 407033 to 914555 tid 1 2024-09-21T18:09:10.521 INFO:tasks.rados.rados.0.smithi160.stdout:2387: writing smithi16030472-421 from 1464527 to 1906918 tid 2 2024-09-21T18:09:10.536 INFO:tasks.rados.rados.0.smithi160.stdout:2387: writing smithi16030472-421 from 2345976 to 2807033 tid 3 2024-09-21T18:09:10.536 INFO:tasks.rados.rados.0.smithi160.stdout:2388: copy_from oid 38 from oid 255 current snap is 0 2024-09-21T18:09:10.536 INFO:tasks.rados.rados.0.smithi160.stdout:2389: copy_from oid 309 from oid 276 current snap is 0 2024-09-21T18:09:10.536 INFO:tasks.rados.rados.0.smithi160.stdout:2390: write oid 59 current snap is 0 2024-09-21T18:09:10.537 INFO:tasks.rados.rados.0.smithi160.stdout:2390: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-09-21T18:09:10.549 INFO:tasks.rados.rados.0.smithi160.stdout:2390: writing smithi16030472-59 from 423840 to 1006611 tid 1 2024-09-21T18:09:10.560 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:10.554+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:10.561 INFO:tasks.rados.rados.0.smithi160.stdout:2390: writing smithi16030472-59 from 1523009 to 2109058 tid 2 2024-09-21T18:09:10.568 INFO:tasks.rados.rados.0.smithi160.stdout:2390: writing smithi16030472-59 from 2657046 to 2823840 tid 3 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2391: copy_from oid 232 from oid 409 current snap is 0 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2392: copy_from oid 284 from oid 133 current snap is 0 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2393: copy_from oid 413 from oid 243 current snap is 0 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2394: read oid 373 snap -1 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2394: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2395: cache_try_flush oid 125 snap -1 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2396: write (excl) oid 174 current snap is 0 2024-09-21T18:09:10.569 INFO:tasks.rados.rados.0.smithi160.stdout:2396: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-21T18:09:10.573 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:10.566+0000 7ff40b999640 -1 osd.4 94 handle_osd_map injecting map crc failure 2024-09-21T18:09:10.577 INFO:tasks.rados.rados.0.smithi160.stdout:2396: writing smithi16030472-174 from 440647 to 1098667 tid 1 2024-09-21T18:09:10.584 INFO:tasks.rados.rados.0.smithi160.stdout:2396: writing smithi16030472-174 from 1581491 to 2027551 tid 2 2024-09-21T18:09:10.589 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:10.589 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:10.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:10.590 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:09:10.590 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:10.590 INFO:tasks.rados.rados.0.smithi160.stdout:2396: writing smithi16030472-174 from 2800822 to 2840647 tid 3 2024-09-21T18:09:10.590 INFO:tasks.rados.rados.0.smithi160.stdout:2397: cache_flush oid 466 snap -1 2024-09-21T18:09:10.590 INFO:tasks.rados.rados.0.smithi160.stdout:2398: cache_try_flush oid 233 snap -1 2024-09-21T18:09:10.591 INFO:tasks.rados.rados.0.smithi160.stdout:2399: delete oid 67 current snap is 0 2024-09-21T18:09:10.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:10.624 INFO:tasks.rados.rados.0.smithi160.stdout:2395: got (0) Success 2024-09-21T18:09:10.625 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 125 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:10.625 INFO:tasks.rados.rados.0.smithi160.stdout:2398: got (0) Success 2024-09-21T18:09:10.625 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 233 v 0 (ObjNum 232 snap 0 seq_num 232) dirty exists 2024-09-21T18:09:10.628 INFO:tasks.rados.rados.0.smithi160.stdout:2387: finishing write tid 1 to smithi16030472-421 2024-09-21T18:09:10.669 INFO:tasks.rados.rados.0.smithi160.stdout:2382: finishing write tid 3 to smithi16030472-437 2024-09-21T18:09:10.669 INFO:tasks.rados.rados.0.smithi160.stdout:2382: finishing write tid 4 to smithi16030472-437 2024-09-21T18:09:10.669 INFO:tasks.rados.rados.0.smithi160.stdout:2382: oid 437 updating version 0 to 798 2024-09-21T18:09:10.670 INFO:tasks.rados.rados.0.smithi160.stdout:2382: oid 437 version 798 is already newer than 797 2024-09-21T18:09:10.670 INFO:tasks.rados.rados.0.smithi160.stdout:2382: oid 437 version 798 is already newer than 796 2024-09-21T18:09:10.670 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 437 v 798 (ObjNum 1116 snap 0 seq_num 1116) dirty exists 2024-09-21T18:09:10.670 INFO:tasks.rados.rados.0.smithi160.stdout:2382: left oid 437 (ObjNum 1116 snap 0 seq_num 1116) 2024-09-21T18:09:10.676 INFO:tasks.rados.rados.0.smithi160.stdout:2394: expect (ObjNum 372 snap 0 seq_num 372) 2024-09-21T18:09:10.683 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 856537 2024-09-21T18:09:10.691 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:10.686+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:10.710 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1315011 2024-09-21T18:09:10.752 INFO:tasks.rados.rados.0.smithi160.stdout:2389: finishing copy_from racing read to smithi16030472-309 2024-09-21T18:09:10.752 INFO:tasks.rados.rados.0.smithi160.stdout:2393: finishing copy_from racing read to smithi16030472-413 2024-09-21T18:09:10.752 INFO:tasks.rados.rados.0.smithi160.stdout:2393: finishing copy_from to smithi16030472-413 2024-09-21T18:09:10.752 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 545 (ObjNum 762 snap 0 seq_num 762) dirty exists 2024-09-21T18:09:10.762 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:10.754+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:10.777 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:10.777 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:10.777 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:10.777 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:10.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:10.843 INFO:tasks.rados.rados.0.smithi160.stdout:2387: finishing write tid 2 to smithi16030472-421 2024-09-21T18:09:10.843 INFO:tasks.rados.rados.0.smithi160.stdout:2387: finishing write tid 3 to smithi16030472-421 2024-09-21T18:09:10.843 INFO:tasks.rados.rados.0.smithi160.stdout:2387: finishing write tid 4 to smithi16030472-421 2024-09-21T18:09:10.883 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:10.878+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:10.888 INFO:tasks.rados.rados.0.smithi160.stdout:2389: finishing copy_from to smithi16030472-309 2024-09-21T18:09:10.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 309 v 537 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:09:10.943 INFO:tasks.rados.rados.0.smithi160.stdout:2387: finishing write tid 5 to smithi16030472-421 2024-09-21T18:09:10.943 INFO:tasks.rados.rados.0.smithi160.stdout:2387: oid 421 updating version 0 to 763 2024-09-21T18:09:10.943 INFO:tasks.rados.rados.0.smithi160.stdout:2387: oid 421 updating version 763 to 764 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2387: oid 421 version 764 is already newer than 762 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2387: oid 421 version 764 is already newer than 761 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 421 v 764 (ObjNum 1119 snap 0 seq_num 1119) dirty exists 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2387: left oid 421 (ObjNum 1119 snap 0 seq_num 1119) 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2396: finishing write tid 1 to smithi16030472-174 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2396: finishing write tid 2 to smithi16030472-174 2024-09-21T18:09:10.944 INFO:tasks.rados.rados.0.smithi160.stdout:2396: finishing write tid 3 to smithi16030472-174 2024-09-21T18:09:10.948 INFO:tasks.rados.rados.0.smithi160.stdout:2388: finishing copy_from racing read to smithi16030472-38 2024-09-21T18:09:10.952 INFO:tasks.rados.rados.0.smithi160.stdout:2390: finishing write tid 1 to smithi16030472-59 2024-09-21T18:09:10.952 INFO:tasks.rados.rados.0.smithi160.stdout:2390: finishing write tid 2 to smithi16030472-59 2024-09-21T18:09:10.953 INFO:tasks.rados.rados.0.smithi160.stdout:2388: finishing copy_from to smithi16030472-38 2024-09-21T18:09:10.953 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 459 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:09:10.963 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:10.959+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:10.967 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:10.967 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:10.967 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:10.967 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:10.967 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:10.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:11.057 INFO:tasks.rados.rados.0.smithi160.stdout:2390: finishing write tid 3 to smithi16030472-59 2024-09-21T18:09:11.058 INFO:tasks.rados.rados.0.smithi160.stdout:2390: finishing write tid 4 to smithi16030472-59 2024-09-21T18:09:11.059 INFO:tasks.rados.rados.0.smithi160.stdout:2390: finishing write tid 5 to smithi16030472-59 2024-09-21T18:09:11.059 INFO:tasks.rados.rados.0.smithi160.stdout:2390: oid 59 updating version 0 to 803 2024-09-21T18:09:11.059 INFO:tasks.rados.rados.0.smithi160.stdout:2390: oid 59 updating version 803 to 805 2024-09-21T18:09:11.059 INFO:tasks.rados.rados.0.smithi160.stdout:2390: oid 59 version 805 is already newer than 804 2024-09-21T18:09:11.059 INFO:tasks.rados.rados.0.smithi160.stdout:2390: oid 59 updating version 805 to 806 2024-09-21T18:09:11.060 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 806 (ObjNum 1120 snap 0 seq_num 1120) dirty exists 2024-09-21T18:09:11.060 INFO:tasks.rados.rados.0.smithi160.stdout:2390: left oid 59 (ObjNum 1120 snap 0 seq_num 1120) 2024-09-21T18:09:11.064 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:11.060+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:11.067 INFO:tasks.rados.rados.0.smithi160.stdout:2382: done (13 left) 2024-09-21T18:09:11.067 INFO:tasks.rados.rados.0.smithi160.stdout:2387: done (12 left) 2024-09-21T18:09:11.067 INFO:tasks.rados.rados.0.smithi160.stdout:2388: done (11 left) 2024-09-21T18:09:11.067 INFO:tasks.rados.rados.0.smithi160.stdout:2389: done (10 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2390: done (9 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2393: done (8 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2394: done (7 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2395: done (6 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2398: done (5 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2399: done (4 left) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2400: read oid 29 snap -1 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2400: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2401: cache_flush oid 3 snap -1 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2402: cache_evict oid 91 snap -1 2024-09-21T18:09:11.068 INFO:tasks.rados.rados.0.smithi160.stdout:2403: cache_flush oid 414 snap -1 2024-09-21T18:09:11.069 INFO:tasks.rados.rados.0.smithi160.stdout:2404: write oid 452 current snap is 0 2024-09-21T18:09:11.069 INFO:tasks.rados.rados.0.smithi160.stdout:2404: seq_num 1123 ranges {474261=408518,1582102=449729,2739315=134946} 2024-09-21T18:09:11.073 INFO:tasks.rados.rados.0.smithi160.stdout:2404: writing smithi16030472-452 from 474261 to 882779 tid 1 2024-09-21T18:09:11.081 INFO:tasks.rados.rados.0.smithi160.stdout:2404: writing smithi16030472-452 from 1582102 to 2031831 tid 2 2024-09-21T18:09:11.087 INFO:tasks.rados.rados.0.smithi160.stdout:2404: writing smithi16030472-452 from 2739315 to 2874261 tid 3 2024-09-21T18:09:11.087 INFO:tasks.rados.rados.0.smithi160.stdout:2405: write oid 94 current snap is 0 2024-09-21T18:09:11.088 INFO:tasks.rados.rados.0.smithi160.stdout:2392: finishing copy_from racing read to smithi16030472-284 2024-09-21T18:09:11.088 INFO:tasks.rados.rados.0.smithi160.stdout:2403: got (0) Success 2024-09-21T18:09:11.088 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 0 (ObjNum 1053 snap 0 seq_num 1053) dirty exists 2024-09-21T18:09:11.088 INFO:tasks.rados.rados.0.smithi160.stdout:2404: finishing write tid 1 to smithi16030472-452 2024-09-21T18:09:11.088 INFO:tasks.rados.rados.0.smithi160.stdout:2405: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-21T18:09:11.095 INFO:tasks.rados.rados.0.smithi160.stdout:2405: writing smithi16030472-94 from 491068 to 974835 tid 1 2024-09-21T18:09:11.104 INFO:tasks.rados.rados.0.smithi160.stdout:2405: writing smithi16030472-94 from 1640584 to 2233971 tid 2 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2405: writing smithi16030472-94 from 2650385 to 2891068 tid 3 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2403: done (9 left) 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2406: read oid 178 snap -1 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2406: expect (ObjNum 1011 snap 0 seq_num 1011) 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2407: cache_try_flush oid 437 snap -1 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2408: cache_try_flush oid 406 snap -1 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2409: copy_from oid 296 from oid 33 current snap is 0 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2410: write (excl) oid 62 current snap is 0 2024-09-21T18:09:11.110 INFO:tasks.rados.rados.0.smithi160.stdout:2410: seq_num 1125 ranges {507875=559016,1699066=737045} 2024-09-21T18:09:11.117 INFO:tasks.rados.rados.0.smithi160.stdout:2410: writing smithi16030472-62 from 507875 to 1066891 tid 1 2024-09-21T18:09:11.127 INFO:tasks.rados.rados.0.smithi160.stdout:2410: writing smithi16030472-62 from 1699066 to 2436111 tid 2 2024-09-21T18:09:11.128 INFO:tasks.rados.rados.0.smithi160.stdout:2411: write oid 158 current snap is 0 2024-09-21T18:09:11.128 INFO:tasks.rados.rados.0.smithi160.stdout:2411: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-21T18:09:11.136 INFO:tasks.rados.rados.0.smithi160.stdout:2411: writing smithi16030472-158 from 524682 to 1275300 tid 1 2024-09-21T18:09:11.142 INFO:tasks.rados.rados.0.smithi160.stdout:2411: writing smithi16030472-158 from 1757548 to 2354604 tid 2 2024-09-21T18:09:11.143 INFO:tasks.rados.rados.0.smithi160.stdout:2412: cache_flush oid 169 snap -1 2024-09-21T18:09:11.143 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.143 INFO:tasks.rados.rados.0.smithi160.stdout:2406: expect (ObjNum 1011 snap 0 seq_num 1011) 2024-09-21T18:09:11.143 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 145.26143596599999, 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:11.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.161 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 144.176043577, 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:11.163 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:11.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:09:11.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:09:11.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:11.167 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:11.168 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:11.169 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:11.164+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:11.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:11.243 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:11.269 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:11.264+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:11.311 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:11.311 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:11.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.311 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:908 3.0 3:25486741:::smithi16030472-437:head [copy-from ver 798 in=56b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:09:11.311 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:11.115831+0000", 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.19388145200000001, 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.193932036, 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115831+0000", 2024-09-21T18:09:11.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115831+0000", 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115836+0000", 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1969999999999998e-06 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115850+0000", 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3936e-05 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115881+0000", 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1053999999999998e-05 2024-09-21T18:09:11.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.115897+0000", 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6653000000000001e-05 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.116068+0000", 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00017067499999999999 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.116298+0000", 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000230034 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.126511+0000", 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010213033999999999 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.137506+0000", 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.010995370000000001 2024-09-21T18:09:11.319 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop_reply(osd.4.21:908 3.0 e95/31)", 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:11.233153+0000", 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.076560012999999996, 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.076710985999999995, 2024-09-21T18:09:11.320 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233153+0000", 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233153+0000", 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233156+0000", 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0800000000000002e-06 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233167+0000", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1297e-05 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233189+0000", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1699e-05 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:11.233202+0000", 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2774e-05 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.323 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:09:11.324 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 2 2024-09-21T18:09:11.324 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:11.325 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:2407: got (0) Success 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 437 v 0 (ObjNum 1116 snap 0 seq_num 1116) dirty exists 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:2406: done (15 left) 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:2407: done (14 left) 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:2413: write oid 332 current snap is 0 2024-09-21T18:09:11.336 INFO:tasks.rados.rados.0.smithi160.stdout:2413: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-21T18:09:11.342 INFO:tasks.rados.rados.0.smithi160.stdout:2413: writing smithi16030472-332 from 541489 to 967356 tid 1 2024-09-21T18:09:11.349 INFO:tasks.rados.rados.0.smithi160.stdout:2413: writing smithi16030472-332 from 1416030 to 2156744 tid 2 2024-09-21T18:09:11.354 INFO:tasks.rados.rados.0.smithi160.stdout:2413: writing smithi16030472-332 from 2732654 to 2941489 tid 3 2024-09-21T18:09:11.354 INFO:tasks.rados.rados.0.smithi160.stdout:2414: cache_evict oid 76 snap -1 2024-09-21T18:09:11.354 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.370 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:11.367+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:11.372 INFO:tasks.rados.rados.0.smithi160.stdout:2391: finishing copy_from racing read to smithi16030472-232 2024-09-21T18:09:11.372 INFO:tasks.rados.rados.0.smithi160.stdout:2396: finishing write tid 4 to smithi16030472-174 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: finishing write tid 5 to smithi16030472-174 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: oid 174 updating version 0 to 765 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: oid 174 updating version 765 to 766 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: oid 174 updating version 766 to 767 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: oid 174 updating version 767 to 768 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 174 v 768 (ObjNum 1121 snap 0 seq_num 1121) dirty exists 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: left oid 174 (ObjNum 1121 snap 0 seq_num 1121) 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2396: done (15 left) 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2415: read oid 86 snap -1 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout:2415: expect deleted 2024-09-21T18:09:11.373 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.374 INFO:tasks.rados.rados.0.smithi160.stdout:2400: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-21T18:09:11.386 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2096835 2024-09-21T18:09:11.428 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1973546 2024-09-21T18:09:11.433 INFO:tasks.rados.rados.0.smithi160.stdout:2402: got (16) Device or resource busy 2024-09-21T18:09:11.433 INFO:tasks.rados.rados.0.smithi160.stdout:2391: finishing copy_from to smithi16030472-232 2024-09-21T18:09:11.433 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 232 v 659 (ObjNum 1021 snap 0 seq_num 1021) dirty exists 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2412: got (2) No such file or directory 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2405: finishing write tid 1 to smithi16030472-94 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2405: finishing write tid 2 to smithi16030472-94 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2405: finishing write tid 3 to smithi16030472-94 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2391: done (15 left) 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2400: done (14 left) 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2402: done (13 left) 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2412: done (12 left) 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2415: done (11 left) 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2416: read oid 302 snap -1 2024-09-21T18:09:11.434 INFO:tasks.rados.rados.0.smithi160.stdout:2416: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2417: read oid 155 snap -1 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2417: expect deleted 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2418: read oid 336 snap -1 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2418: expect (ObjNum 335 snap 0 seq_num 335) 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2419: write oid 431 current snap is 0 2024-09-21T18:09:11.435 INFO:tasks.rados.rados.0.smithi160.stdout:2419: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-09-21T18:09:11.441 INFO:tasks.rados.rados.0.smithi160.stdout:2419: writing smithi16030472-431 from 558296 to 1059412 tid 1 2024-09-21T18:09:11.448 INFO:tasks.rados.rados.0.smithi160.stdout:2419: writing smithi16030472-431 from 1474512 to 2075237 tid 2 2024-09-21T18:09:11.454 INFO:tasks.rados.rados.0.smithi160.stdout:2419: writing smithi16030472-431 from 2760077 to 2958296 tid 3 2024-09-21T18:09:11.454 INFO:tasks.rados.rados.0.smithi160.stdout:2420: read oid 419 snap -1 2024-09-21T18:09:11.454 INFO:tasks.rados.rados.0.smithi160.stdout:2420: expect (ObjNum 1092 snap 0 seq_num 1092) 2024-09-21T18:09:11.454 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.471 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:11.465+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:11.486 INFO:tasks.rados.rados.0.smithi160.stdout:2404: finishing write tid 2 to smithi16030472-452 2024-09-21T18:09:11.486 INFO:tasks.rados.rados.0.smithi160.stdout:2404: finishing write tid 3 to smithi16030472-452 2024-09-21T18:09:11.493 INFO:tasks.rados.rados.0.smithi160.stdout:2404: finishing write tid 4 to smithi16030472-452 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: finishing write tid 5 to smithi16030472-452 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: oid 452 updating version 0 to 811 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: oid 452 version 811 is already newer than 809 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: oid 452 version 811 is already newer than 808 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: oid 452 version 811 is already newer than 810 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 452 v 811 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: left oid 452 (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2404: done (15 left) 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout:2421: copy_from oid 362 from oid 82 current snap is 0 2024-09-21T18:09:11.494 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:11.497 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:11.497 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:11.511 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:11.551 INFO:tasks.rados.rados.0.smithi160.stdout:2392: finishing copy_from to smithi16030472-284 2024-09-21T18:09:11.552 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 284 v 740 (ObjNum 1047 snap 0 seq_num 1047) dirty exists 2024-09-21T18:09:11.552 INFO:tasks.rados.rados.0.smithi160.stdout:2392: done (15 left) 2024-09-21T18:09:11.552 INFO:tasks.rados.rados.0.smithi160.stdout:2422: write oid 251 current snap is 0 2024-09-21T18:09:11.552 INFO:tasks.rados.rados.0.smithi160.stdout:2422: seq_num 1129 ranges {575103=576365,1932994=744383} 2024-09-21T18:09:11.563 INFO:tasks.rados.rados.0.smithi160.stdout:2422: writing smithi16030472-251 from 575103 to 1151468 tid 1 2024-09-21T18:09:11.572 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:11.566+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:11.577 INFO:tasks.rados.rados.0.smithi160.stdout:2422: writing smithi16030472-251 from 1932994 to 2677377 tid 2 2024-09-21T18:09:11.577 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.593 INFO:tasks.rados.rados.0.smithi160.stdout:2410: finishing write tid 1 to smithi16030472-62 2024-09-21T18:09:11.594 INFO:tasks.rados.rados.0.smithi160.stdout:2410: finishing write tid 2 to smithi16030472-62 2024-09-21T18:09:11.594 INFO:tasks.rados.rados.0.smithi160.stdout:2410: finishing write tid 3 to smithi16030472-62 2024-09-21T18:09:11.595 INFO:tasks.rados.rados.0.smithi160.stdout:2410: finishing write tid 4 to smithi16030472-62 2024-09-21T18:09:11.595 INFO:tasks.rados.rados.0.smithi160.stdout:2410: oid 62 updating version 0 to 814 2024-09-21T18:09:11.595 INFO:tasks.rados.rados.0.smithi160.stdout:2410: oid 62 version 814 is already newer than 813 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:2410: oid 62 version 814 is already newer than 812 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 62 v 814 (ObjNum 1125 snap 0 seq_num 1125) dirty exists 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:2410: left oid 62 (ObjNum 1125 snap 0 seq_num 1125) 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:2410: done (15 left) 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:2423: cache_flush oid 111 snap -1 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.596 INFO:tasks.rados.rados.0.smithi160.stdout:2423: got (0) Success 2024-09-21T18:09:11.597 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 111 v 0 (ObjNum -1164551648 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:11.597 INFO:tasks.rados.rados.0.smithi160.stdout:2423: done (15 left) 2024-09-21T18:09:11.597 INFO:tasks.rados.rados.0.smithi160.stdout:2424: read oid 252 snap -1 2024-09-21T18:09:11.597 INFO:tasks.rados.rados.0.smithi160.stdout:2424: expect (ObjNum 893 snap 0 seq_num 893) 2024-09-21T18:09:11.597 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:11.642 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.600750924, 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:09:11.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:09:11.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.60051129999999, 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:09:11.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:09:11.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.600475893, 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:09:11.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:09:11.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.60044272, 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.657 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:09:11.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:09:11.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:11.661 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.60041316, 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:09:11.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:09:11.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:09:11.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:09:11.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.60038350400001, 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:09:11.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.674 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.599897248, 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:09:11.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.59986365500001, 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:09:11.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:09:11.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.49964596300001, 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:09:11.685 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:09:11.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.49904705900001, 2024-09-21T18:09:11.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:11.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:09:11.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:09:11.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.49833244200001, 2024-09-21T18:09:11.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.703 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.48907971, 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:09:11.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:09:11.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.709 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.48881500900001, 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:09:11.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:09:11.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.714 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.48872032, 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.715 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:09:11.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:09:11.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.48867903, 2024-09-21T18:09:11.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.721 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:09:11.722 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.723 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:09:11.724 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.725 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.477409127, 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.726 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.727 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.728 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:09:11.729 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:09:11.730 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.731 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.732 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.468292894, 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.733 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:09:11.734 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.735 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.736 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:09:11.737 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.738 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.467229527, 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:09:11.739 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:11.740 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.741 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.742 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:09:11.743 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.744 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.467189217, 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:09:11.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:09:11.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 99.704551374999994, 2024-09-21T18:09:11.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:11.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:09:11.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:09:11.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:09:11.755 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:11.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:11.756 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:11.758 INFO:tasks.rados.rados.0.smithi160.stdout:2411: finishing write tid 1 to smithi16030472-158 2024-09-21T18:09:11.758 INFO:tasks.rados.rados.0.smithi160.stdout:2411: finishing write tid 2 to smithi16030472-158 2024-09-21T18:09:11.758 INFO:tasks.rados.rados.0.smithi160.stdout:2424: expect (ObjNum 893 snap 0 seq_num 893) 2024-09-21T18:09:11.758 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:11.758 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:11.753+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:11.759 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:11.858 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:11.853+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:11.887 INFO:tasks.rados.rados.0.smithi160.stdout:2411: finishing write tid 3 to smithi16030472-158 2024-09-21T18:09:11.887 INFO:tasks.rados.rados.0.smithi160.stdout:2411: finishing write tid 4 to smithi16030472-158 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2411: oid 158 updating version 0 to 799 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2411: oid 158 updating version 799 to 801 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2411: oid 158 version 801 is already newer than 800 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 158 v 801 (ObjNum 1126 snap 0 seq_num 1126) dirty exists 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2411: left oid 158 (ObjNum 1126 snap 0 seq_num 1126) 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2401: got (0) Success 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 3 v 0 (ObjNum 1051 snap 0 seq_num 1051) dirty exists 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2414: got (16) Device or resource busy 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2405: finishing write tid 4 to smithi16030472-94 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2405: finishing write tid 5 to smithi16030472-94 2024-09-21T18:09:11.888 INFO:tasks.rados.rados.0.smithi160.stdout:2405: oid 94 updating version 0 to 769 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2405: oid 94 updating version 769 to 772 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2405: oid 94 version 772 is already newer than 770 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2405: oid 94 version 772 is already newer than 771 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 772 (ObjNum 1124 snap 0 seq_num 1124) dirty exists 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2405: left oid 94 (ObjNum 1124 snap 0 seq_num 1124) 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2419: finishing write tid 1 to smithi16030472-431 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2419: finishing write tid 2 to smithi16030472-431 2024-09-21T18:09:11.889 INFO:tasks.rados.rados.0.smithi160.stdout:2416: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:09:11.890 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:11.890 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:11.890 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:11.891 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:11.905 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:11.952 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1746778 2024-09-21T18:09:11.959 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:11.955+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:12.016 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2413733 2024-09-21T18:09:12.033 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:12.033 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:12.033 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:12.033 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:12.033 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:12.048 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:12.050 INFO:tasks.rados.rados.0.smithi160.stdout:2420: expect (ObjNum 1092 snap 0 seq_num 1092) 2024-09-21T18:09:12.060 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:12.055+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:12.101 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 393257 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: finishing write tid 3 to smithi16030472-431 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: finishing write tid 4 to smithi16030472-431 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: finishing write tid 5 to smithi16030472-431 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: oid 431 updating version 0 to 765 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: oid 431 version 765 is already newer than 764 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: oid 431 updating version 765 to 766 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: oid 431 version 766 is already newer than 763 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 431 v 766 (ObjNum 1128 snap 0 seq_num 1128) dirty exists 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2419: left oid 431 (ObjNum 1128 snap 0 seq_num 1128) 2024-09-21T18:09:12.128 INFO:tasks.rados.rados.0.smithi160.stdout:2418: expect (ObjNum 335 snap 0 seq_num 335) 2024-09-21T18:09:12.160 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:12.156+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:12.180 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 147.87813832500001, 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:12.181 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:09:12.182 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.183 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.184 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 147.876924266, 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.185 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.186 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:09:12.187 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:09:12.188 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.189 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 147.87557334600001, 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:12.190 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.191 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.192 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.193 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 141.58799270899999, 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.194 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.195 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.196 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:09:12.197 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.198 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 119.48940007900001, 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:12.199 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:09:12.200 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:09:12.201 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 119.48892286900001, 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.203 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:09:12.204 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.205 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.206 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:67 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'24, mlcod=42'9)", 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 119.488487916, 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21108489, 2024-09-21T18:09:12.207 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688702+0000", 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.208 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688703+0000", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8000000000000004e-07 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688715+0000", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416e-05 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688725+0000", 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.082e-05 2024-09-21T18:09:12.209 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.788853+0000", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012726600000001 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794619+0000", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0057662 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.794662+0000", 2024-09-21T18:09:12.210 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3334999999999998e-05 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899755+0000", 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105092903 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899787+0000", 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2070000000000003e-05 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.211 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.18875346900001, 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.212 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:09:12.213 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.214 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:09:12.215 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.08818407699999, 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:09:12.216 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.217 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.218 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:09:12.219 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.220 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.08782601, 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.221 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:09:12.222 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.223 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.224 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.087789177, 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.225 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.226 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.227 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.228 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:12.229 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.08732944499999, 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:12.230 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.231 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:09:12.232 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:09:12.233 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.08689791800001, 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.234 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.235 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.236 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.237 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.085961303, 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:09:12.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:09:12.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:09:12.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.08592262000001, 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:09:12.242 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.243 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:09:12.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.084635968, 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:09:12.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:09:12.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:09:12.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.982491504, 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:09:12.253 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:09:12.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.981767711, 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.255 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.259 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 92.856057472000003, 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:12.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:09:12.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:09:12.264 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 68.216262985, 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.265 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:09:12.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:09:12.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.269 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.270 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:12.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:09:12.273 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:12.273 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3429326 2024-09-21T18:09:12.273 INFO:tasks.rados.rados.0.smithi160.stdout:2422: finishing write tid 1 to smithi16030472-251 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: finishing write tid 1 to smithi16030472-332 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: finishing write tid 2 to smithi16030472-332 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: finishing write tid 3 to smithi16030472-332 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: finishing write tid 4 to smithi16030472-332 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2422: finishing write tid 2 to smithi16030472-251 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2422: finishing write tid 3 to smithi16030472-251 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: finishing write tid 5 to smithi16030472-332 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: oid 332 updating version 0 to 802 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: oid 332 updating version 802 to 805 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: oid 332 version 805 is already newer than 803 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: oid 332 version 805 is already newer than 804 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 805 (ObjNum 1127 snap 0 seq_num 1127) dirty exists 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2413: left oid 332 (ObjNum 1127 snap 0 seq_num 1127) 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2409: finishing copy_from racing read to smithi16030472-296 2024-09-21T18:09:12.274 INFO:tasks.rados.rados.0.smithi160.stdout:2421: finishing copy_from to smithi16030472-362 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 362 v 690 (ObjNum 964 snap 0 seq_num 964) dirty exists 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2421: finishing copy_from racing read to smithi16030472-362 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2408: got (0) Success 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 406 v 0 (ObjNum 172 snap 0 seq_num 172) dirty exists 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2397: got (0) Success 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 466 v 0 (ObjNum 1020 snap 0 seq_num 1020) dirty exists 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2409: finishing copy_from to smithi16030472-296 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 296 v 459 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2422: finishing write tid 4 to smithi16030472-251 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2422: oid 251 updating version 0 to 746 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2422: oid 251 updating version 746 to 748 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2422: oid 251 version 748 is already newer than 747 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 251 v 748 (ObjNum 1129 snap 0 seq_num 1129) dirty exists 2024-09-21T18:09:12.275 INFO:tasks.rados.rados.0.smithi160.stdout:2422: left oid 251 (ObjNum 1129 snap 0 seq_num 1129) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2397: done (15 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2401: done (14 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2405: done (13 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2408: done (12 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2409: done (11 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2411: done (10 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2413: done (9 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2414: done (8 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2416: done (7 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2418: done (6 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2419: done (5 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2420: done (4 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2421: done (3 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2422: done (2 left) 2024-09-21T18:09:12.276 INFO:tasks.rados.rados.0.smithi160.stdout:2424: done (1 left) 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2425: cache_evict oid 66 snap -1 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2426: read oid 488 snap -1 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2426: expect (ObjNum 487 snap 0 seq_num 487) 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2427: write (excl) oid 500 current snap is 0 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2427: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2427: writing smithi16030472-500 from 591910 to 1243524 tid 1 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2427: writing smithi16030472-500 from 1875123 to 2479517 tid 2 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2428: delete oid 81 current snap is 0 2024-09-21T18:09:12.277 INFO:tasks.rados.rados.0.smithi160.stdout:2425: got (0) Success 2024-09-21T18:09:12.278 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:12.267+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:12.373 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:12.367+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:12.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:12.444 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:12.445 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:12.456 INFO:tasks.rados.rados.0.smithi160.stdout:2417: done (4 left) 2024-09-21T18:09:12.456 INFO:tasks.rados.rados.0.smithi160.stdout:2425: done (3 left) 2024-09-21T18:09:12.456 INFO:tasks.rados.rados.0.smithi160.stdout:2428: done (2 left) 2024-09-21T18:09:12.456 INFO:tasks.rados.rados.0.smithi160.stdout:2429: cache_try_flush oid 13 snap -1 2024-09-21T18:09:12.456 INFO:tasks.rados.rados.0.smithi160.stdout:2430: delete oid 443 current snap is 0 2024-09-21T18:09:12.460 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:12.461 INFO:tasks.rados.rados.0.smithi160.stdout:2426: expect (ObjNum 487 snap 0 seq_num 487) 2024-09-21T18:09:12.470 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1608880 2024-09-21T18:09:12.473 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:12.467+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:12.486 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2109292 2024-09-21T18:09:12.487 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:12.548 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:12.549 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:12.573 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:12.573 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:12.573 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:12.571+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:12.593 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:12.594 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:12.596 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:12.597 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:12.597 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:12.597 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:12.597 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:12.611 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:12.614 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:12.614 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:12.641 INFO:tasks.rados.rados.0.smithi160.stdout:2427: finishing write tid 1 to smithi16030472-500 2024-09-21T18:09:12.659 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:12.659 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:12.674 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:12.671+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:12.676 INFO:tasks.rados.rados.0.smithi160.stdout:2429: got (0) Success 2024-09-21T18:09:12.676 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 13 v 0 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T18:09:12.682 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:12.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:12.724 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:12.725 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:12.742 INFO:tasks.rados.rados.0.smithi160.stdout:2426: done (3 left) 2024-09-21T18:09:12.742 INFO:tasks.rados.rados.0.smithi160.stdout:2429: done (2 left) 2024-09-21T18:09:12.742 INFO:tasks.rados.rados.0.smithi160.stdout:2430: done (1 left) 2024-09-21T18:09:12.742 INFO:tasks.rados.rados.0.smithi160.stdout:2431: cache_flush oid 126 snap -1 2024-09-21T18:09:12.743 INFO:tasks.rados.rados.0.smithi160.stdout:2432: cache_flush oid 190 snap -1 2024-09-21T18:09:12.743 INFO:tasks.rados.rados.0.smithi160.stdout:2433: copy_from oid 157 from oid 162 current snap is 0 2024-09-21T18:09:12.743 INFO:tasks.rados.rados.0.smithi160.stdout:2434: copy_from oid 158 from oid 289 current snap is 0 2024-09-21T18:09:12.743 INFO:tasks.rados.rados.0.smithi160.stdout:2435: write oid 36 current snap is 0 2024-09-21T18:09:12.743 INFO:tasks.rados.rados.0.smithi160.stdout:2435: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-09-21T18:09:12.743 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.709296423, 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.744 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:09:12.745 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:09:12.746 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.747 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.705685872, 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:09:12.748 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.749 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:09:12.750 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:09:12.751 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.752 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.702879333, 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.753 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.754 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.755 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.756 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.69913113299999, 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.757 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.758 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.759 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.760 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 101.07828032899999, 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.761 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:09:12.762 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:09:12.763 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:09:12.764 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.977426257, 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.765 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.766 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.767 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.768 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.976669977, 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.769 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:09:12.770 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.771 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.772 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.97627842, 2024-09-21T18:09:12.773 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.774 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:09:12.775 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:09:12.776 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.777 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.975977296, 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.778 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.779 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.780 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.781 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.975253446, 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.783 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:09:12.784 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:09:12.785 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.874356607, 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.786 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.787 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.788 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.789 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.874262324, 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.790 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:09:12.791 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:09:12.792 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:09:12.793 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.873349129, 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.794 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.795 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.796 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.797 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.87277944100001, 2024-09-21T18:09:12.798 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.799 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.800 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.801 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.802 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.872051957, 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.803 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.804 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.805 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.806 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 100.805870048, 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.807 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.808 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.809 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:09:12.810 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 99.549824243000003, 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.811 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.812 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.813 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.814 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 99.549541445000003, 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.815 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:09:12.816 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.817 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:09:12.818 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 99.549462460000001, 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.819 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.820 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.821 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.822 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:09:12.823 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 99.546844128999993, 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:12.824 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:09:12.825 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:09:12.826 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:12.827 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:12.829 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:12.830 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:12.831 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:12.824+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2435: writing smithi16030472-36 from 642331 to 1236045 tid 1 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2435: writing smithi16030472-36 from 1766922 to 2518643 tid 2 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2436: cache_try_flush oid 217 snap -1 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2431: got (0) Success 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 126 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2432: got (2) No such file or directory 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2427: finishing write tid 2 to smithi16030472-500 2024-09-21T18:09:12.832 INFO:tasks.rados.rados.0.smithi160.stdout:2434: finishing copy_from racing read to smithi16030472-158 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2433: finishing copy_from racing read to smithi16030472-157 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: finishing write tid 3 to smithi16030472-500 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: finishing write tid 4 to smithi16030472-500 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: oid 500 updating version 0 to 768 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: oid 500 version 768 is already newer than 729 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: oid 500 updating version 768 to 769 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 500 v 769 (ObjNum 1130 snap 0 seq_num 1130) dirty exists 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: left oid 500 (ObjNum 1130 snap 0 seq_num 1130) 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2435: finishing write tid 1 to smithi16030472-36 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2427: done (6 left) 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2431: done (5 left) 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2432: done (4 left) 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2437: copy_from oid 478 from oid 28 current snap is 0 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2438: read oid 124 snap -1 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2438: expect (ObjNum 898 snap 0 seq_num 898) 2024-09-21T18:09:12.833 INFO:tasks.rados.rados.0.smithi160.stdout:2439: write (excl) oid 417 current snap is 0 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2439: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2439: writing smithi16030472-417 from 659138 to 1328101 tid 1 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2439: writing smithi16030472-417 from 2109051 to 2720783 tid 2 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2440: cache_evict oid 16 snap -1 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2436: got (2) No such file or directory 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2436: done (7 left) 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2441: read oid 74 snap -1 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2441: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2442: read oid 467 snap -1 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2442: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:12.834 INFO:tasks.rados.rados.0.smithi160.stdout:2443: delete oid 355 current snap is 0 2024-09-21T18:09:12.867 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:12.887 INFO:tasks.rados.rados.0.smithi160.stdout:2433: finishing copy_from to smithi16030472-157 2024-09-21T18:09:12.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 695 (ObjNum 946 snap 0 seq_num 946) dirty exists 2024-09-21T18:09:12.887 INFO:tasks.rados.rados.0.smithi160.stdout:2434: finishing copy_from to smithi16030472-158 2024-09-21T18:09:12.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 158 v 177 (ObjNum 288 snap 0 seq_num 288) dirty exists 2024-09-21T18:09:12.899 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:12.925 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:12.929 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:12.922+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:12.930 INFO:tasks.rados.rados.0.smithi160.stdout:2435: finishing write tid 2 to smithi16030472-36 2024-09-21T18:09:12.930 INFO:tasks.rados.rados.0.smithi160.stdout:2435: finishing write tid 3 to smithi16030472-36 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2435: finishing write tid 4 to smithi16030472-36 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2435: oid 36 updating version 0 to 774 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2435: oid 36 updating version 774 to 775 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2435: oid 36 version 775 is already newer than 773 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 775 (ObjNum 1133 snap 0 seq_num 1133) dirty exists 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2435: left oid 36 (ObjNum 1133 snap 0 seq_num 1133) 2024-09-21T18:09:12.931 INFO:tasks.rados.rados.0.smithi160.stdout:2442: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:12.950 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:12.997 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:12.997 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.011 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:13.030 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:13.031 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:13.022+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:13.031 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:13.061 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:13.061 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:13.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:13.106 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1980278 2024-09-21T18:09:13.126 INFO:tasks.rados.rados.0.smithi160.stdout:2438: expect (ObjNum 898 snap 0 seq_num 898) 2024-09-21T18:09:13.128 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.129 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.131 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:13.128+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:13.143 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:13.182 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:13.231 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:13.228+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:13.236 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:09:13.251 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 715301 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:2437: finishing copy_from racing read to smithi16030472-478 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:2437: finishing copy_from to smithi16030472-478 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 478 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:2439: finishing write tid 1 to smithi16030472-417 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:2440: got (16) Device or resource busy 2024-09-21T18:09:13.301 INFO:tasks.rados.rados.0.smithi160.stdout:2439: finishing write tid 2 to smithi16030472-417 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: finishing write tid 3 to smithi16030472-417 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: finishing write tid 4 to smithi16030472-417 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: oid 417 updating version 0 to 817 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: oid 417 version 817 is already newer than 815 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: oid 417 version 817 is already newer than 816 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 817 (ObjNum 1134 snap 0 seq_num 1134) dirty exists 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2439: left oid 417 (ObjNum 1134 snap 0 seq_num 1134) 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:2441: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:09:13.302 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:13.314 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.314 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.329 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:13.332 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:13.326+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:13.399 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:13.433 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:13.427+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:13.456 INFO:tasks.rados.rados.0.smithi160.stdout:2433: done (9 left) 2024-09-21T18:09:13.456 INFO:tasks.rados.rados.0.smithi160.stdout:2434: done (8 left) 2024-09-21T18:09:13.456 INFO:tasks.rados.rados.0.smithi160.stdout:2435: done (7 left) 2024-09-21T18:09:13.456 INFO:tasks.rados.rados.0.smithi160.stdout:2437: done (6 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2438: done (5 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2439: done (4 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2440: done (3 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2441: done (2 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2442: done (1 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2443: done (0 left) 2024-09-21T18:09:13.457 INFO:tasks.rados.rados.0.smithi160.stdout:2444: delete oid 175 current snap is 0 2024-09-21T18:09:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:2444: done (0 left) 2024-09-21T18:09:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:2445: copy_from oid 266 from oid 345 current snap is 0 2024-09-21T18:09:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:2446: copy_from oid 247 from oid 315 current snap is 0 2024-09-21T18:09:13.461 INFO:tasks.rados.rados.0.smithi160.stdout:2447: delete oid 9 current snap is 0 2024-09-21T18:09:13.463 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.463 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.477 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:13.534 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:13.530+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:13.634 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:13.628+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:13.651 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.651 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.665 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:13.682 INFO:tasks.rados.rados.0.smithi160.stdout:2446: finishing copy_from racing read to smithi16030472-247 2024-09-21T18:09:13.735 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:13.733+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:13.786 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.786 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.801 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:13.850 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:13.847+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:13.881 INFO:tasks.rados.rados.0.smithi160.stdout:2445: finishing copy_from racing read to smithi16030472-266 2024-09-21T18:09:13.881 INFO:tasks.rados.rados.0.smithi160.stdout:2447: done (2 left) 2024-09-21T18:09:13.881 INFO:tasks.rados.rados.0.smithi160.stdout:2448: cache_evict oid 472 snap -1 2024-09-21T18:09:13.881 INFO:tasks.rados.rados.0.smithi160.stdout:2449: write (excl) oid 371 current snap is 0 2024-09-21T18:09:13.881 INFO:tasks.rados.rados.0.smithi160.stdout:2449: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-09-21T18:09:13.902 INFO:tasks.rados.rados.0.smithi160.stdout:2449: writing smithi16030472-371 from 726366 to 1296325 tid 1 2024-09-21T18:09:13.919 INFO:tasks.rados.rados.0.smithi160.stdout:2449: writing smithi16030472-371 from 1826626 to 2329343 tid 2 2024-09-21T18:09:13.923 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:13.923 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:13.931 INFO:tasks.rados.rados.0.smithi160.stdout:2449: writing smithi16030472-371 from 2801601 to 3126366 tid 3 2024-09-21T18:09:13.931 INFO:tasks.rados.rados.0.smithi160.stdout:2450: write (excl) oid 86 current snap is 0 2024-09-21T18:09:13.932 INFO:tasks.rados.rados.0.smithi160.stdout:2450: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-21T18:09:13.936 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:13.929+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:13.938 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:13.948 INFO:tasks.rados.rados.0.smithi160.stdout:2450: writing smithi16030472-86 from 743173 to 1388381 tid 1 2024-09-21T18:09:13.960 INFO:tasks.rados.rados.0.smithi160.stdout:2450: writing smithi16030472-86 from 1885108 to 2647836 tid 2 2024-09-21T18:09:13.961 INFO:tasks.rados.rados.0.smithi160.stdout:2451: write (excl) oid 188 current snap is 0 2024-09-21T18:09:13.961 INFO:tasks.rados.rados.0.smithi160.stdout:2451: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-09-21T18:09:13.972 INFO:tasks.rados.rados.0.smithi160.stdout:2451: writing smithi16030472-188 from 759980 to 1480437 tid 1 2024-09-21T18:09:13.980 INFO:tasks.rados.rados.0.smithi160.stdout:2451: writing smithi16030472-188 from 1943590 to 2449976 tid 2 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2451: writing smithi16030472-188 from 2856447 to 3159980 tid 3 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2452: copy_from oid 434 from oid 326 current snap is 0 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2446: finishing copy_from to smithi16030472-247 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 481 (ObjNum 737 snap 0 seq_num 737) dirty exists 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2446: done (6 left) 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2453: read oid 261 snap -1 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2453: expect deleted 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2454: write (excl) oid 393 current snap is 0 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2448: got (16) Device or resource busy 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:2445: finishing copy_from to smithi16030472-266 2024-09-21T18:09:13.985 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 744 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:09:13.986 INFO:tasks.rados.rados.0.smithi160.stdout:2454: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-21T18:09:13.994 INFO:tasks.rados.rados.0.smithi160.stdout:2454: writing smithi16030472-393 from 776787 to 1288846 tid 1 2024-09-21T18:09:14.003 INFO:tasks.rados.rados.0.smithi160.stdout:2454: writing smithi16030472-393 from 1718425 to 2484822 tid 2 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2454: writing smithi16030472-393 from 3000223 to 3176787 tid 3 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2445: done (7 left) 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2448: done (6 left) 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2455: write (excl) oid 494 current snap is 0 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2449: finishing write tid 1 to smithi16030472-371 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2449: finishing write tid 2 to smithi16030472-371 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2454: finishing write tid 1 to smithi16030472-393 2024-09-21T18:09:14.009 INFO:tasks.rados.rados.0.smithi160.stdout:2455: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-21T18:09:14.020 INFO:tasks.rados.rados.0.smithi160.stdout:2455: writing smithi16030472-494 from 793594 to 1380902 tid 1 2024-09-21T18:09:14.030 INFO:tasks.rados.rados.0.smithi160.stdout:2455: writing smithi16030472-494 from 2060554 to 2570609 tid 2 2024-09-21T18:09:14.030 INFO:tasks.rados.rados.0.smithi160.stdout:2456: delete oid 346 current snap is 0 2024-09-21T18:09:14.037 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:14.029+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:14.089 INFO:tasks.rados.rados.0.smithi160.stdout:2451: finishing write tid 1 to smithi16030472-188 2024-09-21T18:09:14.090 INFO:tasks.rados.rados.0.smithi160.stdout:2451: finishing write tid 2 to smithi16030472-188 2024-09-21T18:09:14.107 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.107 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.111 INFO:tasks.rados.rados.0.smithi160.stdout:2454: finishing write tid 2 to smithi16030472-393 2024-09-21T18:09:14.122 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:14.131 INFO:tasks.rados.rados.0.smithi160.stdout:2452: finishing copy_from racing read to smithi16030472-434 2024-09-21T18:09:14.138 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:14.132+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:14.232 INFO:tasks.rados.rados.0.smithi160.stdout:2449: finishing write tid 3 to smithi16030472-371 2024-09-21T18:09:14.232 INFO:tasks.rados.rados.0.smithi160.stdout:2452: finishing copy_from to smithi16030472-434 2024-09-21T18:09:14.232 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 434 v 274 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:09:14.232 INFO:tasks.rados.rados.0.smithi160.stdout:2449: finishing write tid 4 to smithi16030472-371 2024-09-21T18:09:14.239 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:14.234+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:14.245 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.245 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.261 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:14.264 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:14.261+0000 7f6a3cca9640 -1 osd.3 95 handle_osd_map injecting map crc failure 2024-09-21T18:09:14.288 INFO:tasks.rados.rados.0.smithi160.stdout:2451: finishing write tid 3 to smithi16030472-188 2024-09-21T18:09:14.289 INFO:tasks.rados.rados.0.smithi160.stdout:2454: finishing write tid 3 to smithi16030472-393 2024-09-21T18:09:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:2454: finishing write tid 4 to smithi16030472-393 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: finishing write tid 5 to smithi16030472-393 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: oid 393 updating version 0 to 811 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: oid 393 version 811 is already newer than 810 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: oid 393 updating version 811 to 812 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: oid 393 version 812 is already newer than 809 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 812 (ObjNum 1141 snap 0 seq_num 1141) dirty exists 2024-09-21T18:09:14.312 INFO:tasks.rados.rados.0.smithi160.stdout:2454: left oid 393 (ObjNum 1141 snap 0 seq_num 1141) 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: finishing write tid 5 to smithi16030472-371 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: oid 371 updating version 0 to 779 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: oid 371 version 779 is already newer than 777 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: oid 371 version 779 is already newer than 778 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: oid 371 version 779 is already newer than 776 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 779 (ObjNum 1138 snap 0 seq_num 1138) dirty exists 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2449: left oid 371 (ObjNum 1138 snap 0 seq_num 1138) 2024-09-21T18:09:14.333 INFO:tasks.rados.rados.0.smithi160.stdout:2450: finishing write tid 1 to smithi16030472-86 2024-09-21T18:09:14.334 INFO:tasks.rados.rados.0.smithi160.stdout:2450: finishing write tid 2 to smithi16030472-86 2024-09-21T18:09:14.334 INFO:tasks.rados.rados.0.smithi160.stdout:2450: finishing write tid 3 to smithi16030472-86 2024-09-21T18:09:14.339 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:14.335+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:14.378 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.378 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.392 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:14.434 INFO:tasks.rados.rados.0.smithi160.stdout:2450: finishing write tid 4 to smithi16030472-86 2024-09-21T18:09:14.434 INFO:tasks.rados.rados.0.smithi160.stdout:2450: oid 86 updating version 0 to 781 2024-09-21T18:09:14.434 INFO:tasks.rados.rados.0.smithi160.stdout:2450: oid 86 version 781 is already newer than 780 2024-09-21T18:09:14.435 INFO:tasks.rados.rados.0.smithi160.stdout:2450: oid 86 updating version 781 to 782 2024-09-21T18:09:14.435 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 86 v 782 (ObjNum 1139 snap 0 seq_num 1139) dirty exists 2024-09-21T18:09:14.435 INFO:tasks.rados.rados.0.smithi160.stdout:2450: left oid 86 (ObjNum 1139 snap 0 seq_num 1139) 2024-09-21T18:09:14.435 INFO:tasks.rados.rados.0.smithi160.stdout:2455: finishing write tid 1 to smithi16030472-494 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2455: finishing write tid 2 to smithi16030472-494 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2449: done (7 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2450: done (6 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2452: done (5 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2453: done (4 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2454: done (3 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2456: done (2 left) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2457: read oid 420 snap -1 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2457: expect (ObjNum 419 snap 0 seq_num 419) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2458: cache_evict oid 384 snap -1 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2459: read oid 233 snap -1 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2459: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2460: cache_evict oid 71 snap -1 2024-09-21T18:09:14.437 INFO:tasks.rados.rados.0.smithi160.stdout:2461: cache_try_flush oid 224 snap -1 2024-09-21T18:09:14.438 INFO:tasks.rados.rados.0.smithi160.stdout:2462: write oid 264 current snap is 0 2024-09-21T18:09:14.438 INFO:tasks.rados.rados.0.smithi160.stdout:2462: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2024-09-21T18:09:14.440 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:14.434+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:14.446 INFO:tasks.rados.rados.0.smithi160.stdout:2462: writing smithi16030472-264 from 427208 to 1165014 tid 1 2024-09-21T18:09:14.455 INFO:tasks.rados.rados.0.smithi160.stdout:2462: writing smithi16030472-264 from 1777518 to 2291242 tid 2 2024-09-21T18:09:14.462 INFO:tasks.rados.rados.0.smithi160.stdout:2462: writing smithi16030472-264 from 2966139 to 3227208 tid 3 2024-09-21T18:09:14.462 INFO:tasks.rados.rados.0.smithi160.stdout:2463: cache_try_flush oid 446 snap -1 2024-09-21T18:09:14.462 INFO:tasks.rados.rados.0.smithi160.stdout:2455: finishing write tid 3 to smithi16030472-494 2024-09-21T18:09:14.462 INFO:tasks.rados.rados.0.smithi160.stdout:2458: got (16) Device or resource busy 2024-09-21T18:09:14.462 INFO:tasks.rados.rados.0.smithi160.stdout:2455: finishing write tid 4 to smithi16030472-494 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2455: oid 494 updating version 0 to 783 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2455: oid 494 updating version 783 to 784 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2455: oid 494 updating version 784 to 785 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 785 (ObjNum 1142 snap 0 seq_num 1142) dirty exists 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2455: left oid 494 (ObjNum 1142 snap 0 seq_num 1142) 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2455: done (8 left) 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2458: done (7 left) 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2464: cache_flush oid 103 snap -1 2024-09-21T18:09:14.463 INFO:tasks.rados.rados.0.smithi160.stdout:2459: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-21T18:09:14.541 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:14.535+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:14.560 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.560 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.575 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:14.642 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:14.634+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2461: got (0) Success 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 224 v 0 (ObjNum 897 snap 0 seq_num 897) dirty exists 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2460: got (16) Device or resource busy 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2464: got (0) Success 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 0 (ObjNum 991 snap 0 seq_num 991) dirty exists 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2451: finishing write tid 4 to smithi16030472-188 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2451: finishing write tid 5 to smithi16030472-188 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2451: oid 188 updating version 0 to 820 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2451: oid 188 version 820 is already newer than 819 2024-09-21T18:09:14.646 INFO:tasks.rados.rados.0.smithi160.stdout:2451: oid 188 updating version 820 to 821 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2451: oid 188 version 821 is already newer than 818 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 188 v 821 (ObjNum 1140 snap 0 seq_num 1140) dirty exists 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2451: left oid 188 (ObjNum 1140 snap 0 seq_num 1140) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2462: finishing write tid 1 to smithi16030472-264 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2462: finishing write tid 2 to smithi16030472-264 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2451: done (7 left) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2459: done (6 left) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2460: done (5 left) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2461: done (4 left) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2464: done (3 left) 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2465: read oid 405 snap -1 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2465: expect deleted 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2466: cache_try_flush oid 81 snap -1 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2467: copy_from oid 178 from oid 2 current snap is 0 2024-09-21T18:09:14.647 INFO:tasks.rados.rados.0.smithi160.stdout:2468: copy_from oid 47 from oid 71 current snap is 0 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2469: copy_from oid 8 from oid 53 current snap is 0 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2470: cache_try_flush oid 350 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2471: read oid 265 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2471: expect (ObjNum 817 snap 0 seq_num 817) 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2472: cache_flush oid 374 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2473: cache_flush oid 308 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2474: cache_try_flush oid 14 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2475: read oid 153 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2475: expect deleted 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2476: read oid 219 snap -1 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2476: expect deleted 2024-09-21T18:09:14.648 INFO:tasks.rados.rados.0.smithi160.stdout:2477: delete oid 290 current snap is 0 2024-09-21T18:09:14.649 INFO:tasks.rados.rados.0.smithi160.stdout:2457: expect (ObjNum 419 snap 0 seq_num 419) 2024-09-21T18:09:14.678 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:14.711 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:14.738 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.738 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.743 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:14.740+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:14.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:14.767 INFO:tasks.rados.rados.0.smithi160.stdout:2463: got (0) Success 2024-09-21T18:09:14.767 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 0 (ObjNum 496 snap 0 seq_num 496) dirty exists 2024-09-21T18:09:14.767 INFO:tasks.rados.rados.0.smithi160.stdout:2467: finishing copy_from racing read to smithi16030472-178 2024-09-21T18:09:14.768 INFO:tasks.rados.rados.0.smithi160.stdout:2467: finishing copy_from to smithi16030472-178 2024-09-21T18:09:14.768 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 178 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T18:09:14.844 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:14.842+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:2462: finishing write tid 3 to smithi16030472-264 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:2462: finishing write tid 4 to smithi16030472-264 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:2466: got (0) Success 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 81 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:2472: got (2) No such file or directory 2024-09-21T18:09:14.847 INFO:tasks.rados.rados.0.smithi160.stdout:2470: got (2) No such file or directory 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: finishing write tid 5 to smithi16030472-264 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: oid 264 updating version 0 to 822 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: oid 264 updating version 822 to 824 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: oid 264 updating version 824 to 825 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: oid 264 version 825 is already newer than 823 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 825 (ObjNum 1144 snap 0 seq_num 1144) dirty exists 2024-09-21T18:09:14.848 INFO:tasks.rados.rados.0.smithi160.stdout:2462: left oid 264 (ObjNum 1144 snap 0 seq_num 1144) 2024-09-21T18:09:14.861 INFO:tasks.rados.rados.0.smithi160.stdout:2471: expect (ObjNum 817 snap 0 seq_num 817) 2024-09-21T18:09:14.875 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:14.875 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:14.889 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:14.945 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:14.937+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:14.969 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:15.012 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:15.012 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:15.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:15.046 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:15.042+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:15.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2469: finishing copy_from racing read to smithi16030472-8 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2474: got (0) Success 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 0 (ObjNum 984 snap 0 seq_num 984) dirty exists 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2469: finishing copy_from to smithi16030472-8 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 8 v 762 (ObjNum 1115 snap 0 seq_num 1115) dirty exists 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2468: finishing copy_from racing read to smithi16030472-47 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2473: got (0) Success 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 0 (ObjNum 567 snap 0 seq_num 567) dirty exists 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2457: done (15 left) 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2462: done (14 left) 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2463: done (13 left) 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2465: done (12 left) 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2466: done (11 left) 2024-09-21T18:09:15.085 INFO:tasks.rados.rados.0.smithi160.stdout:2467: done (10 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2469: done (9 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2470: done (8 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2471: done (7 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2472: done (6 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2473: done (5 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2474: done (4 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2475: done (3 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2476: done (2 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2477: done (1 left) 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2478: cache_evict oid 383 snap -1 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2479: write (excl) oid 405 current snap is 0 2024-09-21T18:09:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:2479: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2024-09-21T18:09:15.100 INFO:tasks.rados.rados.0.smithi160.stdout:2479: writing smithi16030472-405 from 460822 to 949126 tid 1 2024-09-21T18:09:15.115 INFO:tasks.rados.rados.0.smithi160.stdout:2479: writing smithi16030472-405 from 1378129 to 1895522 tid 2 2024-09-21T18:09:15.132 INFO:tasks.rados.rados.0.smithi160.stdout:2479: writing smithi16030472-405 from 2504632 to 2910594 tid 3 2024-09-21T18:09:15.132 INFO:tasks.rados.rados.0.smithi160.stdout:2480: cache_evict oid 219 snap -1 2024-09-21T18:09:15.132 INFO:tasks.rados.rados.0.smithi160.stdout:2481: write oid 308 current snap is 0 2024-09-21T18:09:15.132 INFO:tasks.rados.rados.0.smithi160.stdout:2481: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-21T18:09:15.146 INFO:tasks.rados.rados.0.smithi160.stdout:2481: writing smithi16030472-308 from 477629 to 1041182 tid 1 2024-09-21T18:09:15.146 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:15.138+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:15.150 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:15.150 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:15.161 INFO:tasks.rados.rados.0.smithi160.stdout:2481: writing smithi16030472-308 from 1836611 to 2614015 tid 2 2024-09-21T18:09:15.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:15.167 INFO:tasks.rados.rados.0.smithi160.stdout:2481: writing smithi16030472-308 from 3048408 to 3277629 tid 3 2024-09-21T18:09:15.167 INFO:tasks.rados.rados.0.smithi160.stdout:2482: read oid 263 snap -1 2024-09-21T18:09:15.167 INFO:tasks.rados.rados.0.smithi160.stdout:2482: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:09:15.167 INFO:tasks.rados.rados.0.smithi160.stdout:2483: cache_try_flush oid 338 snap -1 2024-09-21T18:09:15.168 INFO:tasks.rados.rados.0.smithi160.stdout:2484: delete oid 440 current snap is 0 2024-09-21T18:09:15.186 INFO:tasks.rados.rados.0.smithi160.stdout:2478: got (16) Device or resource busy 2024-09-21T18:09:15.225 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:15.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:15.235 INFO:tasks.rados.rados.0.smithi160.stdout:2480: got (0) Success 2024-09-21T18:09:15.246 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:15.247 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:15.247 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:15.242+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:15.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:09:15.268 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:15.268 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:15.287 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:15.287 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:15.292 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:15.292 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:15.292 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:15.292 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:15.304 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:15.304 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:15.306 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:15.325 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:15.325 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:15.346 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:15.347 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:15.342+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:15.363 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:15.364 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:15.372 INFO:tasks.rados.rados.0.smithi160.stdout:2483: got (0) Success 2024-09-21T18:09:15.373 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 338 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:09:15.384 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2468: finishing copy_from to smithi16030472-47 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 337 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: finishing write tid 1 to smithi16030472-405 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: finishing write tid 2 to smithi16030472-405 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: finishing write tid 3 to smithi16030472-405 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: finishing write tid 4 to smithi16030472-405 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: finishing write tid 5 to smithi16030472-405 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: oid 405 updating version 0 to 827 2024-09-21T18:09:15.435 INFO:tasks.rados.rados.0.smithi160.stdout:2479: oid 405 updating version 827 to 829 2024-09-21T18:09:15.436 INFO:tasks.rados.rados.0.smithi160.stdout:2479: oid 405 version 829 is already newer than 826 2024-09-21T18:09:15.436 INFO:tasks.rados.rados.0.smithi160.stdout:2479: oid 405 version 829 is already newer than 828 2024-09-21T18:09:15.436 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 405 v 829 (ObjNum 1146 snap 0 seq_num 1146) dirty exists 2024-09-21T18:09:15.436 INFO:tasks.rados.rados.0.smithi160.stdout:2479: left oid 405 (ObjNum 1146 snap 0 seq_num 1146) 2024-09-21T18:09:15.436 INFO:tasks.rados.rados.0.smithi160.stdout:2481: finishing write tid 1 to smithi16030472-308 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2468: done (7 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2478: done (6 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2479: done (5 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2480: done (4 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2483: done (3 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2484: done (2 left) 2024-09-21T18:09:15.438 INFO:tasks.rados.rados.0.smithi160.stdout:2485: cache_evict oid 433 snap -1 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2486: read oid 259 snap -1 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2486: expect deleted 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2487: cache_try_flush oid 230 snap -1 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2488: read oid 319 snap -1 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2488: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2489: cache_try_flush oid 499 snap -1 2024-09-21T18:09:15.439 INFO:tasks.rados.rados.0.smithi160.stdout:2490: delete oid 128 current snap is 0 2024-09-21T18:09:15.448 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:15.441+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:15.479 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:15.479 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:15.479 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:15.479 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:15.479 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:15.493 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:15.506 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:15.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:15.539 INFO:tasks.rados.rados.0.smithi160.stdout:2485: got (16) Device or resource busy 2024-09-21T18:09:15.540 INFO:tasks.rados.rados.0.smithi160.stdout:2482: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:09:15.549 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:15.542+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:15.567 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:15.599 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:15.599 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:15.629 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:15.633 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:15.633 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:15.650 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:15.645+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:15.657 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:15.657 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:15.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:15.657 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:09:15.657 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:15.662 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:15.675 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:15.677 INFO:tasks.rados.rados.0.smithi160.stdout:2481: finishing write tid 2 to smithi16030472-308 2024-09-21T18:09:15.677 INFO:tasks.rados.rados.0.smithi160.stdout:2488: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-21T18:09:15.687 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3069673 2024-09-21T18:09:15.693 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:15.693 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:15.697 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:15.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:15.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:15.733 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:15.752 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:15.748+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:15.769 INFO:tasks.rados.rados.0.smithi160.stdout:2481: finishing write tid 3 to smithi16030472-308 2024-09-21T18:09:15.769 INFO:tasks.rados.rados.0.smithi160.stdout:2481: finishing write tid 4 to smithi16030472-308 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: finishing write tid 5 to smithi16030472-308 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: oid 308 updating version 0 to 831 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: oid 308 updating version 831 to 832 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: oid 308 updating version 832 to 833 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: oid 308 version 833 is already newer than 830 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 833 (ObjNum 1147 snap 0 seq_num 1147) dirty exists 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: left oid 308 (ObjNum 1147 snap 0 seq_num 1147) 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2487: got (0) Success 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 0 (ObjNum 796 snap 0 seq_num 796) dirty exists 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2489: got (0) Success 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 0 (ObjNum 1049 snap 0 seq_num 1049) dirty exists 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2481: done (7 left) 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2482: done (6 left) 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2485: done (5 left) 2024-09-21T18:09:15.770 INFO:tasks.rados.rados.0.smithi160.stdout:2486: done (4 left) 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2487: done (3 left) 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2488: done (2 left) 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2489: done (1 left) 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2490: done (0 left) 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2491: write (excl) oid 190 current snap is 0 2024-09-21T18:09:15.771 INFO:tasks.rados.rados.0.smithi160.stdout:2491: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2024-09-21T18:09:15.777 INFO:tasks.rados.rados.0.smithi160.stdout:2491: writing smithi16030472-190 from 528050 to 1033703 tid 1 2024-09-21T18:09:15.786 INFO:tasks.rados.rados.0.smithi160.stdout:2491: writing smithi16030472-190 from 1612057 to 2136788 tid 2 2024-09-21T18:09:15.792 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:15.792 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:15.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.792 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6883 4.1 4.fac98349 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:15.777233+0000", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.013853726, 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.013893189, 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6883 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:15.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777233+0000", 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777233+0000", 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777237+0000", 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1400000000000002e-06 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777763+0000", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00052557899999999998 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777795+0000", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.222e-05 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.777813+0000", 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7683e-05 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:15.795 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6884 4.1 4.fac98349 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:15.786158+0000", 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.0049286130000000001, 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0050568159999999996, 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:15.796 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6884 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.786158+0000", 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.786158+0000", 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:15.797 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.786161+0000", 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1299999999999999e-06 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.786470+0000", 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030966000000000001 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:15.786486+0000", 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6286e-05 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:15.798 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:15.799 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:15.799 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:15.799 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 2 2024-09-21T18:09:15.799 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:15.799 INFO:tasks.rados.rados.0.smithi160.stdout:2491: writing smithi16030472-190 from 2614324 to 3181727 tid 3 2024-09-21T18:09:15.799 INFO:tasks.rados.rados.0.smithi160.stdout:2492: write (excl) oid 196 current snap is 0 2024-09-21T18:09:15.799 INFO:tasks.rados.rados.0.smithi160.stdout:2492: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-21T18:09:15.805 INFO:tasks.rados.rados.0.smithi160.stdout:2492: writing smithi16030472-196 from 544857 to 1125759 tid 1 2024-09-21T18:09:15.807 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:15.817 INFO:tasks.rados.rados.0.smithi160.stdout:2492: writing smithi16030472-196 from 1670539 to 2338928 tid 2 2024-09-21T18:09:15.826 INFO:tasks.rados.rados.0.smithi160.stdout:2492: writing smithi16030472-196 from 3041747 to 3344857 tid 3 2024-09-21T18:09:15.826 INFO:tasks.rados.rados.0.smithi160.stdout:2493: write (excl) oid 227 current snap is 0 2024-09-21T18:09:15.826 INFO:tasks.rados.rados.0.smithi160.stdout:2493: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-21T18:09:15.835 INFO:tasks.rados.rados.0.smithi160.stdout:2493: writing smithi16030472-227 from 561664 to 1217815 tid 1 2024-09-21T18:09:15.842 INFO:tasks.rados.rados.0.smithi160.stdout:2493: writing smithi16030472-227 from 1729021 to 2257421 tid 2 2024-09-21T18:09:15.849 INFO:tasks.rados.rados.0.smithi160.stdout:2493: writing smithi16030472-227 from 2669170 to 3361664 tid 3 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2494: read oid 12 snap -1 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2494: expect (ObjNum 772 snap 0 seq_num 772) 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2495: read oid 168 snap -1 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2495: expect deleted 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2496: cache_try_flush oid 413 snap -1 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2493: finishing write tid 1 to smithi16030472-227 2024-09-21T18:09:15.850 INFO:tasks.rados.rados.0.smithi160.stdout:2497: delete oid 118 current snap is 0 2024-09-21T18:09:15.852 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:15.848+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:15.936 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:15.936 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:15.936 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:15.936 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:15.936 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:15.950 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:15.953 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:15.947+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:16.054 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:16.050+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:16.117 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.96974295299999, 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.118 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.119 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.120 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.121 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.122 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.123 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:16.124 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.969696644, 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.125 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:09:16.126 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.127 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.128 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.129 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.967600731, 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.130 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.131 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.132 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.133 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.134 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.135 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:16.136 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.966937616, 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:16.137 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.138 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.139 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:09:16.140 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:09:16.141 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.142 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.96690111700001, 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.143 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:16.144 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:09:16.145 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:09:16.146 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.147 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.148 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.833262945, 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.149 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:09:16.150 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.151 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.152 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:09:16.153 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:09:16.154 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:16.155 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.832551154, 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.156 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.157 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.158 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.159 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.160 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.161 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 122.83251359099999, 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:09:16.162 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.163 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.164 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.165 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:09:16.166 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.167 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 99.035127591999995, 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.168 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.169 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.170 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.171 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 99.034933046000006, 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.172 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:09:16.173 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:09:16.174 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.175 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 99.034646658, 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.176 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:16.177 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:09:16.178 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:09:16.179 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 99.034630578000005, 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.180 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:16.181 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.182 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:09:16.183 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 99.005703326000003, 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.184 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.185 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.186 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:09:16.187 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.999484250999998, 2024-09-21T18:09:16.188 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.189 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:09:16.190 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:09:16.191 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.192 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.994382655999999, 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.193 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.194 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:09:16.195 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:09:16.196 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.994048707999994, 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.197 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.198 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:09:16.199 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:09:16.200 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.994028228999994, 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:09:16.201 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.202 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:09:16.203 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:09:16.204 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.907699702000002, 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:09:16.205 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.206 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:09:16.207 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.208 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.209 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.899239515999994, 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.210 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:09:16.211 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:09:16.212 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.213 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.893311811000004, 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.214 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.215 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:09:16.216 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.217 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.218 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:16.220 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: finishing write tid 2 to smithi16030472-227 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: finishing write tid 3 to smithi16030472-227 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: finishing write tid 4 to smithi16030472-227 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: finishing write tid 5 to smithi16030472-227 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: oid 227 updating version 0 to 835 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: oid 227 updating version 835 to 836 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: oid 227 updating version 836 to 837 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: oid 227 version 837 is already newer than 834 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 227 v 837 (ObjNum 1152 snap 0 seq_num 1152) dirty exists 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2493: left oid 227 (ObjNum 1152 snap 0 seq_num 1152) 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2492: finishing write tid 1 to smithi16030472-196 2024-09-21T18:09:16.221 INFO:tasks.rados.rados.0.smithi160.stdout:2491: finishing write tid 1 to smithi16030472-190 2024-09-21T18:09:16.222 INFO:tasks.rados.rados.0.smithi160.stdout:2494: expect (ObjNum 772 snap 0 seq_num 772) 2024-09-21T18:09:16.222 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:16.222 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:16.218+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:16.255 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:09:16.320 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:16.316+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:16.341 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:16.342 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:16.342 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:16.342 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:16.356 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2491: finishing write tid 2 to smithi16030472-190 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2492: finishing write tid 2 to smithi16030472-196 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2491: finishing write tid 3 to smithi16030472-190 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2491: finishing write tid 4 to smithi16030472-190 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2491: finishing write tid 5 to smithi16030472-190 2024-09-21T18:09:16.368 INFO:tasks.rados.rados.0.smithi160.stdout:2491: oid 190 updating version 0 to 772 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2491: oid 190 version 772 is already newer than 771 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2491: oid 190 version 772 is already newer than 770 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2491: oid 190 version 772 is already newer than 733 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 190 v 772 (ObjNum 1150 snap 0 seq_num 1150) dirty exists 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2491: left oid 190 (ObjNum 1150 snap 0 seq_num 1150) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2491: done (6 left) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2493: done (5 left) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2494: done (4 left) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2495: done (3 left) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2497: done (2 left) 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2498: cache_try_flush oid 258 snap -1 2024-09-21T18:09:16.369 INFO:tasks.rados.rados.0.smithi160.stdout:2499: delete oid 239 current snap is 0 2024-09-21T18:09:16.400 INFO:tasks.rados.rados.0.smithi160.stdout:2492: finishing write tid 3 to smithi16030472-196 2024-09-21T18:09:16.400 INFO:tasks.rados.rados.0.smithi160.stdout:2492: finishing write tid 4 to smithi16030472-196 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: finishing write tid 5 to smithi16030472-196 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: oid 196 updating version 0 to 751 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: oid 196 updating version 751 to 787 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: oid 196 updating version 787 to 788 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: oid 196 version 788 is already newer than 786 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 196 v 788 (ObjNum 1151 snap 0 seq_num 1151) dirty exists 2024-09-21T18:09:16.401 INFO:tasks.rados.rados.0.smithi160.stdout:2492: left oid 196 (ObjNum 1151 snap 0 seq_num 1151) 2024-09-21T18:09:16.420 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:16.416+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:16.440 INFO:tasks.rados.rados.0.smithi160.stdout:2498: got (0) Success 2024-09-21T18:09:16.441 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 0 (ObjNum 936 snap 0 seq_num 936) dirty exists 2024-09-21T18:09:16.498 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:16.498 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:16.498 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:16.498 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:16.498 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:16.512 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:16.521 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:16.516+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:16.575 INFO:tasks.rados.rados.0.smithi160.stdout:2496: got (0) Success 2024-09-21T18:09:16.575 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 0 (ObjNum 762 snap 0 seq_num 762) dirty exists 2024-09-21T18:09:16.576 INFO:tasks.rados.rados.0.smithi160.stdout:2492: done (3 left) 2024-09-21T18:09:16.576 INFO:tasks.rados.rados.0.smithi160.stdout:2496: done (2 left) 2024-09-21T18:09:16.576 INFO:tasks.rados.rados.0.smithi160.stdout:2498: done (1 left) 2024-09-21T18:09:16.576 INFO:tasks.rados.rados.0.smithi160.stdout:2499: done (0 left) 2024-09-21T18:09:16.576 INFO:tasks.rados.rados.0.smithi160.stdout:2500: delete oid 12 current snap is 0 2024-09-21T18:09:16.622 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:16.620+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.681 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 118.00435083799999, 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:09:16.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:09:16.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:09:16.685 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 118.00428047600001, 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.686 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:09:16.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:09:16.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 117.992479041, 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.690 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:09:16.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:09:16.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 117.984612658, 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:09:16.694 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.695 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:09:16.696 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:09:16.697 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.698 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 117.984227014, 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.699 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.700 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.701 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.702 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 117.984181565, 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.703 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.704 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:09:16.705 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:09:16.706 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:09:16.707 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.708 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.936283648, 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:16.709 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:09:16.710 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.711 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.712 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 116.936176006, 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.713 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:09:16.714 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:09:16.715 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 111.360393391, 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.716 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:16.717 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:09:16.718 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:09:16.719 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.720 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.721 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 111.360341456, 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.722 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.723 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.724 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:09:16.725 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:09:16.726 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 110.59363706800001, 2024-09-21T18:09:16.727 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.728 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.729 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.730 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:16.731 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:09:16.732 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 110.36453381, 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.733 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.734 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.735 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.736 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:09:16.737 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:09:16.738 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 110.36410582000001, 2024-09-21T18:09:16.739 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.740 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.741 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.742 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.743 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.744 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:16.745 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 110.364059251, 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:16.746 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:09:16.747 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:09:16.748 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:09:16.749 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:09:16.750 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 105.86339019099999, 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.751 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.752 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:09:16.753 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.754 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.755 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.756 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.855858394999998, 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.757 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.758 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.759 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.760 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:09:16.761 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.762 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.855784075000003, 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.763 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:09:16.764 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:09:16.765 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.766 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:09:16.767 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.453980830000006, 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.768 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.769 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.770 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.771 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 98.351630942, 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.772 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.773 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.774 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.775 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.776 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6670 4.2 4:574ca510:::smithi16030472-466:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 6.0390025439999997, 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4286385400000001, 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:16.777 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6670 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.778 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637674+0000", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637680+0000", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6629999999999998e-06 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637691+0000", 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1097000000000001e-05 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.779 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637700+0000", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054141+0000", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41644096200000003 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054299+0000", 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157572 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.780 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.065938+0000", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.011639553 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066192+0000", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253222 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066312+0000", 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120285 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.781 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:16.782 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:16.782 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:16.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:16.784 INFO:tasks.rados.rados.0.smithi160.stdout:2500: done (0 left) 2024-09-21T18:09:16.784 INFO:tasks.rados.rados.0.smithi160.stdout:2501: write (excl) oid 261 current snap is 0 2024-09-21T18:09:16.784 INFO:tasks.rados.rados.0.smithi160.stdout:2501: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2024-09-21T18:09:16.784 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:16.777+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:16.798 INFO:tasks.rados.rados.0.smithi160.stdout:2501: writing smithi16030472-261 from 628892 to 1302392 tid 1 2024-09-21T18:09:16.814 INFO:tasks.rados.rados.0.smithi160.stdout:2501: writing smithi16030472-261 from 1962949 to 2498687 tid 2 2024-09-21T18:09:16.827 INFO:tasks.rados.rados.0.smithi160.stdout:2501: writing smithi16030472-261 from 3178862 to 3428892 tid 3 2024-09-21T18:09:16.827 INFO:tasks.rados.rados.0.smithi160.stdout:2502: write (excl) oid 169 current snap is 0 2024-09-21T18:09:16.827 INFO:tasks.rados.rados.0.smithi160.stdout:2502: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-09-21T18:09:16.843 INFO:tasks.rados.rados.0.smithi160.stdout:2502: writing smithi16030472-169 from 645699 to 1394448 tid 1 2024-09-21T18:09:16.855 INFO:tasks.rados.rados.0.smithi160.stdout:2502: writing smithi16030472-169 from 2021431 to 2700827 tid 2 2024-09-21T18:09:16.861 INFO:tasks.rados.rados.0.smithi160.stdout:2502: writing smithi16030472-169 from 3206285 to 3445699 tid 3 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2503: copy_from oid 484 from oid 258 current snap is 0 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2501: finishing write tid 1 to smithi16030472-261 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2501: finishing write tid 2 to smithi16030472-261 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2504: cache_try_flush oid 253 snap -1 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2505: cache_flush oid 28 snap -1 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2506: cache_try_flush oid 402 snap -1 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2507: cache_try_flush oid 418 snap -1 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2508: write oid 70 current snap is 0 2024-09-21T18:09:16.862 INFO:tasks.rados.rados.0.smithi160.stdout:2508: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-21T18:09:16.869 INFO:tasks.rados.rados.0.smithi160.stdout:2508: writing smithi16030472-70 from 662506 to 1086504 tid 1 2024-09-21T18:09:16.876 INFO:tasks.rados.rados.0.smithi160.stdout:2508: writing smithi16030472-70 from 1563560 to 2102967 tid 2 2024-09-21T18:09:16.883 INFO:tasks.rados.rados.0.smithi160.stdout:2508: writing smithi16030472-70 from 2717355 to 3207640 tid 3 2024-09-21T18:09:16.883 INFO:tasks.rados.rados.0.smithi160.stdout:2509: delete oid 432 current snap is 0 2024-09-21T18:09:16.883 INFO:tasks.rados.rados.0.smithi160.stdout:2507: got (0) Success 2024-09-21T18:09:16.884 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 418 v 0 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-21T18:09:16.884 INFO:tasks.rados.rados.0.smithi160.stdout:2505: got (0) Success 2024-09-21T18:09:16.884 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:16.884 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:16.877+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:16.922 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:16.922 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:16.923 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:16.923 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:16.928 INFO:tasks.rados.rados.0.smithi160.stdout:2501: finishing write tid 3 to smithi16030472-261 2024-09-21T18:09:16.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:16.985 INFO:tasks.rados.rados.0.smithi160.stdout:2508: finishing write tid 1 to smithi16030472-70 2024-09-21T18:09:16.987 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:16.982+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:17.065 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.065 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:17.065 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:17.066 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:17.066 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:17.085 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:17.077+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:17.105 INFO:tasks.rados.rados.0.smithi160.stdout:2508: finishing write tid 2 to smithi16030472-70 2024-09-21T18:09:17.105 INFO:tasks.rados.rados.0.smithi160.stdout:2502: finishing write tid 1 to smithi16030472-169 2024-09-21T18:09:17.110 INFO:tasks.rados.rados.0.smithi160.stdout:2508: finishing write tid 3 to smithi16030472-70 2024-09-21T18:09:17.113 INFO:tasks.rados.rados.0.smithi160.stdout:2505: done (8 left) 2024-09-21T18:09:17.114 INFO:tasks.rados.rados.0.smithi160.stdout:2507: done (7 left) 2024-09-21T18:09:17.114 INFO:tasks.rados.rados.0.smithi160.stdout:2509: done (6 left) 2024-09-21T18:09:17.114 INFO:tasks.rados.rados.0.smithi160.stdout:2510: delete oid 295 current snap is 0 2024-09-21T18:09:17.186 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:17.182+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:17.204 INFO:tasks.rados.rados.0.smithi160.stdout:2503: finishing copy_from racing read to smithi16030472-484 2024-09-21T18:09:17.204 INFO:tasks.rados.rados.0.smithi160.stdout:2501: finishing write tid 4 to smithi16030472-261 2024-09-21T18:09:17.206 INFO:tasks.rados.rados.0.smithi160.stdout:2506: got (0) Success 2024-09-21T18:09:17.206 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 402 v 0 (ObjNum 595 snap 0 seq_num 595) dirty exists 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: finishing write tid 5 to smithi16030472-261 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: oid 261 updating version 0 to 841 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: oid 261 version 841 is already newer than 840 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: oid 261 version 841 is already newer than 838 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: oid 261 version 841 is already newer than 839 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 261 v 841 (ObjNum 1156 snap 0 seq_num 1156) dirty exists 2024-09-21T18:09:17.207 INFO:tasks.rados.rados.0.smithi160.stdout:2501: left oid 261 (ObjNum 1156 snap 0 seq_num 1156) 2024-09-21T18:09:17.252 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.252 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:17.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:17.252 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:09:17.252 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:17.286 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:17.281+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:17.307 INFO:tasks.rados.rados.0.smithi160.stdout:2503: finishing copy_from to smithi16030472-484 2024-09-21T18:09:17.307 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 670 (ObjNum 936 snap 0 seq_num 936) dirty exists 2024-09-21T18:09:17.311 INFO:tasks.rados.rados.0.smithi160.stdout:2508: finishing write tid 4 to smithi16030472-70 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: finishing write tid 5 to smithi16030472-70 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: oid 70 updating version 0 to 774 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: oid 70 version 774 is already newer than 773 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: oid 70 updating version 774 to 775 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: oid 70 version 775 is already newer than 735 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 775 (ObjNum 1158 snap 0 seq_num 1158) dirty exists 2024-09-21T18:09:17.312 INFO:tasks.rados.rados.0.smithi160.stdout:2508: left oid 70 (ObjNum 1158 snap 0 seq_num 1158) 2024-09-21T18:09:17.318 INFO:tasks.rados.rados.0.smithi160.stdout:2501: done (6 left) 2024-09-21T18:09:17.318 INFO:tasks.rados.rados.0.smithi160.stdout:2503: done (5 left) 2024-09-21T18:09:17.318 INFO:tasks.rados.rados.0.smithi160.stdout:2506: done (4 left) 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2508: done (3 left) 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2510: done (2 left) 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2511: cache_flush oid 80 snap -1 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2512: read oid 230 snap -1 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2512: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2513: copy_from oid 365 from oid 55 current snap is 0 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2514: copy_from oid 366 from oid 28 current snap is 0 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2515: write (excl) oid 60 current snap is 0 2024-09-21T18:09:17.319 INFO:tasks.rados.rados.0.smithi160.stdout:2515: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-09-21T18:09:17.329 INFO:tasks.rados.rados.0.smithi160.stdout:2515: writing smithi16030472-60 from 712927 to 1362672 tid 1 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2515: writing smithi16030472-60 from 2139006 to 2825740 tid 2 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2516: read oid 194 snap -1 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2516: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2517: read oid 208 snap -1 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2517: expect (ObjNum 207 snap 0 seq_num 207) 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2518: copy_from oid 180 from oid 183 current snap is 0 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2511: got (0) Success 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 0 (ObjNum 198 snap 0 seq_num 198) dirty exists 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2511: done (9 left) 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2519: copy_from oid 385 from oid 452 current snap is 0 2024-09-21T18:09:17.341 INFO:tasks.rados.rados.0.smithi160.stdout:2512: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:09:17.387 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:17.381+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:17.438 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:17.438 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:17.470 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:17.488 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:17.484+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:17.525 INFO:tasks.rados.rados.0.smithi160.stdout:2504: got (0) Success 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 253 v 0 (ObjNum -1166666496 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: finishing write tid 2 to smithi16030472-169 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: finishing write tid 3 to smithi16030472-169 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: finishing write tid 4 to smithi16030472-169 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: finishing write tid 5 to smithi16030472-169 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: oid 169 updating version 0 to 752 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: oid 169 updating version 752 to 792 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: oid 169 version 792 is already newer than 791 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: oid 169 version 792 is already newer than 790 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 169 v 792 (ObjNum 1157 snap 0 seq_num 1157) dirty exists 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2502: left oid 169 (ObjNum 1157 snap 0 seq_num 1157) 2024-09-21T18:09:17.526 INFO:tasks.rados.rados.0.smithi160.stdout:2516: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:17.527 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:17.562 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:17.579 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.580 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:17.580 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:17.580 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:17.580 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.589 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:17.587+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:17.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2502: done (9 left) 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2504: done (8 left) 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2512: done (7 left) 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2516: done (6 left) 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2520: cache_flush oid 131 snap -1 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2519: finishing copy_from racing read to smithi16030472-385 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2514: finishing copy_from racing read to smithi16030472-366 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2515: finishing write tid 1 to smithi16030472-60 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2515: finishing write tid 2 to smithi16030472-60 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:2514: finishing copy_from to smithi16030472-366 2024-09-21T18:09:17.638 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2514: done (6 left) 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2521: read oid 324 snap -1 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2521: expect (ObjNum 119 snap 0 seq_num 119) 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2522: read oid 70 snap -1 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2522: expect (ObjNum 1158 snap 0 seq_num 1158) 2024-09-21T18:09:17.639 INFO:tasks.rados.rados.0.smithi160.stdout:2523: delete oid 286 current snap is 0 2024-09-21T18:09:17.691 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:17.685+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:17.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:17.727 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:17.727 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 124.51314512, 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.728 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:09:17.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:09:17.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:09:17.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:09:17.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.969207571, 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.734 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:09:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:09:17.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:09:17.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.739 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.80828410700001, 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:09:17.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:09:17.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:09:17.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.807609512, 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.748 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.807582974, 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.749 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:09:17.752 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.797749232, 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.753 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.78533744400001, 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.757 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.776447374, 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.763 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.775635357, 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.769 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:09:17.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:09:17.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.775527145, 2024-09-21T18:09:17.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:17.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:09:17.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:09:17.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:09:17.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.675255999, 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.67435974, 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.785 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:09:17.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:17.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.67387812699999, 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:09:17.789 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.673425702, 2024-09-21T18:09:17.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.797 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.673375376, 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.801 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.547642596, 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:17.808 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.543364539, 2024-09-21T18:09:17.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:09:17.809 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.809 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:09:17.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:09:17.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:09:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.542994299, 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:09:17.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:09:17.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 106.54292456899999, 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.822 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.826 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 105.951347579, 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:09:17.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:17.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:09:17.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:09:17.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:17.832 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2518: finishing copy_from racing read to smithi16030472-180 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2513: finishing copy_from racing read to smithi16030472-365 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2515: finishing write tid 3 to smithi16030472-60 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2515: finishing write tid 4 to smithi16030472-60 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2515: oid 60 updating version 0 to 843 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2515: oid 60 version 843 is already newer than 842 2024-09-21T18:09:17.834 INFO:tasks.rados.rados.0.smithi160.stdout:2515: oid 60 updating version 843 to 844 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 844 (ObjNum 1161 snap 0 seq_num 1161) dirty exists 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:2515: left oid 60 (ObjNum 1161 snap 0 seq_num 1161) 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:2520: got (2) No such file or directory 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:2513: finishing copy_from to smithi16030472-365 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 365 v 704 (ObjNum 975 snap 0 seq_num 975) dirty exists 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:2519: finishing copy_from to smithi16030472-385 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 811 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:09:17.835 INFO:tasks.rados.rados.0.smithi160.stdout:2522: expect (ObjNum 1158 snap 0 seq_num 1158) 2024-09-21T18:09:17.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:17.836 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:17.828+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:17.854 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:17.874 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:17.875 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:17.884 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:17.895 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:17.916 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:17.932 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:17.933 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:17.934 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:17.930+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:17.954 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:17.970 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:17.964+0000 7fd79a4fd640 -1 osd.1 97 handle_osd_map injecting map crc failure 2024-09-21T18:09:17.974 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:17.975 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:17.975 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:17.975 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:17.975 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:17.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:17.995 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:17.996 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:18.018 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:18.018 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:18.035 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:18.029+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:18.060 INFO:tasks.rados.rados.0.smithi160.stdout:2521: expect (ObjNum 119 snap 0 seq_num 119) 2024-09-21T18:09:18.063 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 605742 2024-09-21T18:09:18.087 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1843618 2024-09-21T18:09:18.095 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:18.110 INFO:tasks.rados.rados.0.smithi160.stdout:2518: finishing copy_from to smithi16030472-180 2024-09-21T18:09:18.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 42 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:09:18.110 INFO:tasks.rados.rados.0.smithi160.stdout:2517: expect (ObjNum 207 snap 0 seq_num 207) 2024-09-21T18:09:18.110 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 444860 2024-09-21T18:09:18.115 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.115 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.123 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:18.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:18.136 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:18.131+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:18.147 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2084270 2024-09-21T18:09:18.149 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:18.149 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:18.174 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:18.174 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:18.198 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:18.206 INFO:tasks.rados.rados.0.smithi160.stdout:2513: done (8 left) 2024-09-21T18:09:18.206 INFO:tasks.rados.rados.0.smithi160.stdout:2515: done (7 left) 2024-09-21T18:09:18.206 INFO:tasks.rados.rados.0.smithi160.stdout:2517: done (6 left) 2024-09-21T18:09:18.206 INFO:tasks.rados.rados.0.smithi160.stdout:2518: done (5 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2519: done (4 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2520: done (3 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2521: done (2 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2522: done (1 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2523: done (0 left) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2524: read oid 13 snap -1 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2524: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-21T18:09:18.207 INFO:tasks.rados.rados.0.smithi160.stdout:2525: delete oid 479 current snap is 0 2024-09-21T18:09:18.218 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:18.223 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:18.267 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:18.262+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:18.277 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.277 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:09:18.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:18.337 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:18.335+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:18.414 INFO:tasks.rados.rados.0.smithi160.stdout:2524: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-21T18:09:18.438 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:18.433+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:18.457 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.458 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.462 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:18.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:18.488 INFO:tasks.rados.rados.0.smithi160.stdout:2524: done (1 left) 2024-09-21T18:09:18.488 INFO:tasks.rados.rados.0.smithi160.stdout:2525: done (0 left) 2024-09-21T18:09:18.488 INFO:tasks.rados.rados.0.smithi160.stdout:2526: cache_evict oid 166 snap -1 2024-09-21T18:09:18.488 INFO:tasks.rados.rados.0.smithi160.stdout:2527: delete oid 96 current snap is 0 2024-09-21T18:09:18.489 INFO:tasks.rados.rados.0.smithi160.stdout:2526: got (16) Device or resource busy 2024-09-21T18:09:18.539 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:18.535+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.593 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:18.640 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:18.633+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2526: done (1 left) 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2527: done (0 left) 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2528: cache_evict oid 16 snap -1 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2529: copy_from oid 24 from oid 352 current snap is 0 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2530: read oid 296 snap -1 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2530: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2531: copy_from oid 426 from oid 128 current snap is 0 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2532: write oid 123 current snap is 0 2024-09-21T18:09:18.697 INFO:tasks.rados.rados.0.smithi160.stdout:2532: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-21T18:09:18.716 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:18.712+0000 7fd79a4fd640 -1 osd.1 98 handle_osd_map injecting map crc failure 2024-09-21T18:09:18.720 INFO:tasks.rados.rados.0.smithi160.stdout:2532: writing smithi16030472-123 from 780155 to 1447249 tid 1 2024-09-21T18:09:18.728 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.728 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.740 INFO:tasks.rados.rados.0.smithi160.stdout:2532: writing smithi16030472-123 from 1972934 to 2667006 tid 2 2024-09-21T18:09:18.741 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:18.735+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:18.742 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2532: writing smithi16030472-123 from 3309316 to 3580155 tid 3 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2533: read oid 263 snap -1 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2533: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2534: cache_try_flush oid 77 snap -1 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2528: got (16) Device or resource busy 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2528: done (6 left) 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2535: write (excl) oid 351 current snap is 0 2024-09-21T18:09:18.754 INFO:tasks.rados.rados.0.smithi160.stdout:2535: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-09-21T18:09:18.769 INFO:tasks.rados.rados.0.smithi160.stdout:2535: writing smithi16030472-351 from 796962 to 1539305 tid 1 2024-09-21T18:09:18.780 INFO:tasks.rados.rados.0.smithi160.stdout:2535: writing smithi16030472-351 from 2031416 to 2469146 tid 2 2024-09-21T18:09:18.790 INFO:tasks.rados.rados.0.smithi160.stdout:2535: writing smithi16030472-351 from 2936739 to 3349906 tid 3 2024-09-21T18:09:18.790 INFO:tasks.rados.rados.0.smithi160.stdout:2536: cache_flush oid 408 snap -1 2024-09-21T18:09:18.790 INFO:tasks.rados.rados.0.smithi160.stdout:2534: got (2) No such file or directory 2024-09-21T18:09:18.790 INFO:tasks.rados.rados.0.smithi160.stdout:2534: done (7 left) 2024-09-21T18:09:18.790 INFO:tasks.rados.rados.0.smithi160.stdout:2537: read oid 113 snap -1 2024-09-21T18:09:18.791 INFO:tasks.rados.rados.0.smithi160.stdout:2537: expect (ObjNum 707 snap 0 seq_num 707) 2024-09-21T18:09:18.791 INFO:tasks.rados.rados.0.smithi160.stdout:2538: read oid 157 snap -1 2024-09-21T18:09:18.791 INFO:tasks.rados.rados.0.smithi160.stdout:2538: expect (ObjNum 946 snap 0 seq_num 946) 2024-09-21T18:09:18.791 INFO:tasks.rados.rados.0.smithi160.stdout:2539: write oid 297 current snap is 0 2024-09-21T18:09:18.791 INFO:tasks.rados.rados.0.smithi160.stdout:2539: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2024-09-21T18:09:18.798 INFO:tasks.rados.rados.0.smithi160.stdout:2539: writing smithi16030472-297 from 413769 to 831361 tid 1 2024-09-21T18:09:18.811 INFO:tasks.rados.rados.0.smithi160.stdout:2539: writing smithi16030472-297 from 1573545 to 2271286 tid 2 2024-09-21T18:09:18.822 INFO:tasks.rados.rados.0.smithi160.stdout:2539: writing smithi16030472-297 from 2847809 to 3472248 tid 3 2024-09-21T18:09:18.822 INFO:tasks.rados.rados.0.smithi160.stdout:2540: cache_evict oid 474 snap -1 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2541: cache_flush oid 116 snap -1 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2542: cache_flush oid 57 snap -1 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2543: read oid 479 snap -1 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2543: expect deleted 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2544: write (excl) oid 464 current snap is 0 2024-09-21T18:09:18.823 INFO:tasks.rados.rados.0.smithi160.stdout:2544: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-21T18:09:18.829 INFO:tasks.rados.rados.0.smithi160.stdout:2544: writing smithi16030472-464 from 430576 to 923417 tid 1 2024-09-21T18:09:18.837 INFO:tasks.rados.rados.0.smithi160.stdout:2544: writing smithi16030472-464 from 1632027 to 2073426 tid 2 2024-09-21T18:09:18.842 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:18.839+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:18.843 INFO:tasks.rados.rados.0.smithi160.stdout:2544: writing smithi16030472-464 from 2475232 to 2910943 tid 3 2024-09-21T18:09:18.848 INFO:tasks.rados.rados.0.smithi160.stdout:2544: writing smithi16030472-464 from 3365397 to 3630576 tid 4 2024-09-21T18:09:18.848 INFO:tasks.rados.rados.0.smithi160.stdout:2545: write (excl) oid 462 current snap is 0 2024-09-21T18:09:18.848 INFO:tasks.rados.rados.0.smithi160.stdout:2545: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-09-21T18:09:18.854 INFO:tasks.rados.rados.0.smithi160.stdout:2545: writing smithi16030472-462 from 447383 to 1015473 tid 1 2024-09-21T18:09:18.863 INFO:tasks.rados.rados.0.smithi160.stdout:2545: writing smithi16030472-462 from 1690509 to 2391919 tid 2 2024-09-21T18:09:18.868 INFO:tasks.rados.rados.0.smithi160.stdout:2545: writing smithi16030472-462 from 2902655 to 3549638 tid 3 2024-09-21T18:09:18.869 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:18.869 INFO:tasks.rados.rados.0.smithi160.stdout:2532: finishing write tid 1 to smithi16030472-123 2024-09-21T18:09:18.880 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:18.880 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:18.894 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:18.895 INFO:tasks.rados.rados.0.smithi160.stdout:2538: expect (ObjNum 946 snap 0 seq_num 946) 2024-09-21T18:09:18.943 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:18.939+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:19.014 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.014 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.030 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:19.044 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:19.037+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:19.095 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3810254 2024-09-21T18:09:19.144 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:19.139+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:19.153 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.153 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.168 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:19.172 INFO:tasks.rados.rados.0.smithi160.stdout:2532: finishing write tid 2 to smithi16030472-123 2024-09-21T18:09:19.172 INFO:tasks.rados.rados.0.smithi160.stdout:2532: finishing write tid 3 to smithi16030472-123 2024-09-21T18:09:19.172 INFO:tasks.rados.rados.0.smithi160.stdout:2533: expect (ObjNum 907 snap 0 seq_num 907) 2024-09-21T18:09:19.246 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:19.242+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:19.287 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:09:19.295 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.296 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:19.312 INFO:tasks.rados.rados.0.smithi160.stdout:2532: finishing write tid 4 to smithi16030472-123 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: finishing write tid 5 to smithi16030472-123 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: oid 123 updating version 0 to 845 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: oid 123 updating version 845 to 848 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: oid 123 version 848 is already newer than 847 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: oid 123 version 848 is already newer than 846 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 848 (ObjNum 1165 snap 0 seq_num 1165) dirty exists 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2532: left oid 123 (ObjNum 1165 snap 0 seq_num 1165) 2024-09-21T18:09:19.313 INFO:tasks.rados.rados.0.smithi160.stdout:2530: expect (ObjNum 623 snap 0 seq_num 623) 2024-09-21T18:09:19.318 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 998130 2024-09-21T18:09:19.343 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1700539 2024-09-21T18:09:19.346 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:19.343+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2529: finishing copy_from to smithi16030472-24 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2529: got expected ENOENT (src dne) 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2529: finishing copy_from racing read to smithi16030472-24 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2531: finishing copy_from to smithi16030472-426 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2531: got expected ENOENT (src dne) 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2531: finishing copy_from racing read to smithi16030472-426 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2539: finishing write tid 1 to smithi16030472-297 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2540: got (16) Device or resource busy 2024-09-21T18:09:19.375 INFO:tasks.rados.rados.0.smithi160.stdout:2541: got (0) Success 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 0 (ObjNum 869 snap 0 seq_num 869) dirty exists 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:2535: finishing write tid 1 to smithi16030472-351 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:2535: finishing write tid 2 to smithi16030472-351 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:2542: got (0) Success 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 57 v 0 (ObjNum 980 snap 0 seq_num 980) dirty exists 2024-09-21T18:09:19.376 INFO:tasks.rados.rados.0.smithi160.stdout:2537: expect (ObjNum 707 snap 0 seq_num 707) 2024-09-21T18:09:19.436 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.436 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.447 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:19.443+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:19.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:19.511 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:19.505+0000 7fb25ec0a640 -1 osd.11 99 handle_osd_map injecting map crc failure 2024-09-21T18:09:19.522 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3699444 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: finishing write tid 3 to smithi16030472-351 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: finishing write tid 4 to smithi16030472-351 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: finishing write tid 5 to smithi16030472-351 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: oid 351 updating version 0 to 852 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: oid 351 version 852 is already newer than 850 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: oid 351 version 852 is already newer than 849 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: oid 351 version 852 is already newer than 851 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 351 v 852 (ObjNum 1166 snap 0 seq_num 1166) dirty exists 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2535: left oid 351 (ObjNum 1166 snap 0 seq_num 1166) 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 1 to smithi16030472-464 2024-09-21T18:09:19.535 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 2 to smithi16030472-464 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2539: finishing write tid 2 to smithi16030472-297 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2536: got (0) Success 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 0 (ObjNum 819 snap 0 seq_num 819) dirty exists 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 3 to smithi16030472-464 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 4 to smithi16030472-464 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 5 to smithi16030472-464 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: finishing write tid 6 to smithi16030472-464 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: oid 464 updating version 0 to 794 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: oid 464 updating version 794 to 795 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: oid 464 updating version 795 to 797 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: oid 464 updating version 797 to 798 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: oid 464 version 798 is already newer than 796 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 798 (ObjNum 1168 snap 0 seq_num 1168) dirty exists 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2544: left oid 464 (ObjNum 1168 snap 0 seq_num 1168) 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2539: finishing write tid 3 to smithi16030472-297 2024-09-21T18:09:19.536 INFO:tasks.rados.rados.0.smithi160.stdout:2529: done (15 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2530: done (14 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2531: done (13 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2532: done (12 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2533: done (11 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2535: done (10 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2536: done (9 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2537: done (8 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2538: done (7 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2540: done (6 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2541: done (5 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2542: done (4 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2543: done (3 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2544: done (2 left) 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2546: cache_try_flush oid 56 snap -1 2024-09-21T18:09:19.537 INFO:tasks.rados.rados.0.smithi160.stdout:2547: cache_try_flush oid 15 snap -1 2024-09-21T18:09:19.538 INFO:tasks.rados.rados.0.smithi160.stdout:2548: cache_flush oid 27 snap -1 2024-09-21T18:09:19.538 INFO:tasks.rados.rados.0.smithi160.stdout:2549: cache_flush oid 477 snap -1 2024-09-21T18:09:19.538 INFO:tasks.rados.rados.0.smithi160.stdout:2550: delete oid 57 current snap is 0 2024-09-21T18:09:19.547 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:19.544+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:19.579 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.579 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:19.600 INFO:tasks.rados.rados.0.smithi160.stdout:2545: finishing write tid 1 to smithi16030472-462 2024-09-21T18:09:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:2545: finishing write tid 2 to smithi16030472-462 2024-09-21T18:09:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:2545: finishing write tid 3 to smithi16030472-462 2024-09-21T18:09:19.606 INFO:tasks.rados.rados.0.smithi160.stdout:2545: finishing write tid 4 to smithi16030472-462 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: finishing write tid 5 to smithi16030472-462 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: oid 462 updating version 0 to 816 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: oid 462 updating version 816 to 818 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: oid 462 version 818 is already newer than 817 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: oid 462 version 818 is already newer than 815 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 462 v 818 (ObjNum 1169 snap 0 seq_num 1169) dirty exists 2024-09-21T18:09:19.607 INFO:tasks.rados.rados.0.smithi160.stdout:2545: left oid 462 (ObjNum 1169 snap 0 seq_num 1169) 2024-09-21T18:09:19.634 INFO:tasks.rados.rados.0.smithi160.stdout:2546: got (2) No such file or directory 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: finishing write tid 4 to smithi16030472-297 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: finishing write tid 5 to smithi16030472-297 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: oid 297 updating version 0 to 737 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: oid 297 updating version 737 to 813 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: oid 297 version 813 is already newer than 738 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: oid 297 version 813 is already newer than 812 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 297 v 813 (ObjNum 1167 snap 0 seq_num 1167) dirty exists 2024-09-21T18:09:19.635 INFO:tasks.rados.rados.0.smithi160.stdout:2539: left oid 297 (ObjNum 1167 snap 0 seq_num 1167) 2024-09-21T18:09:19.683 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:19.680+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2539: done (6 left) 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2545: done (5 left) 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2546: done (4 left) 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2550: done (3 left) 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2551: read oid 9 snap -1 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2551: expect deleted 2024-09-21T18:09:19.714 INFO:tasks.rados.rados.0.smithi160.stdout:2552: read oid 270 snap -1 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2552: expect (ObjNum 682 snap 0 seq_num 682) 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2553: read oid 127 snap -1 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2553: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2554: cache_flush oid 296 snap -1 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2555: read oid 274 snap -1 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2555: expect (ObjNum 273 snap 0 seq_num 273) 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2556: cache_evict oid 101 snap -1 2024-09-21T18:09:19.715 INFO:tasks.rados.rados.0.smithi160.stdout:2557: delete oid 261 current snap is 0 2024-09-21T18:09:19.729 INFO:tasks.rados.rados.0.smithi160.stdout:2553: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:19.749 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:19.743+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:19.765 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.765 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.779 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:19.785 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 509863 2024-09-21T18:09:19.823 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:19.863 INFO:tasks.rados.rados.0.smithi160.stdout:2549: got (0) Success 2024-09-21T18:09:19.863 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 477 v 0 (ObjNum 3 snap 0 seq_num 304) clean dne 2024-09-21T18:09:19.863 INFO:tasks.rados.rados.0.smithi160.stdout:2552: expect (ObjNum 682 snap 0 seq_num 682) 2024-09-21T18:09:19.863 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:19.867 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:19.864+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:19.945 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:19.945 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:19.951 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:19.948+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:19.959 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:19.982 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1190049 2024-09-21T18:09:20.021 INFO:tasks.rados.rados.0.smithi160.stdout:2556: got (16) Device or resource busy 2024-09-21T18:09:20.021 INFO:tasks.rados.rados.0.smithi160.stdout:2548: got (0) Success 2024-09-21T18:09:20.021 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 0 (ObjNum 908 snap 0 seq_num 908) dirty exists 2024-09-21T18:09:20.021 INFO:tasks.rados.rados.0.smithi160.stdout:2555: expect (ObjNum 273 snap 0 seq_num 273) 2024-09-21T18:09:20.021 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:20.052 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:20.050+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:20.098 INFO:tasks.rados.rados.0.smithi160.stdout:2554: got (0) Success 2024-09-21T18:09:20.098 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 296 v 0 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:09:20.098 INFO:tasks.rados.rados.0.smithi160.stdout:2548: done (9 left) 2024-09-21T18:09:20.098 INFO:tasks.rados.rados.0.smithi160.stdout:2549: done (8 left) 2024-09-21T18:09:20.098 INFO:tasks.rados.rados.0.smithi160.stdout:2551: done (7 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2552: done (6 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2553: done (5 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2554: done (4 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2555: done (3 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2556: done (2 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2557: done (1 left) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2558: read oid 327 snap -1 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2558: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2559: cache_try_flush oid 378 snap -1 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2560: read oid 46 snap -1 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2560: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2561: read oid 156 snap -1 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2561: expect (ObjNum 1019 snap 0 seq_num 1019) 2024-09-21T18:09:20.099 INFO:tasks.rados.rados.0.smithi160.stdout:2562: delete oid 69 current snap is 0 2024-09-21T18:09:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:2561: expect (ObjNum 1019 snap 0 seq_num 1019) 2024-09-21T18:09:20.107 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:20.125 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:20.125 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:20.140 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:20.187 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:20.184+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:20.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:20.255 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:20.249+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:20.268 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:20.268 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:20.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:20.296 INFO:tasks.rados.rados.0.smithi160.stdout:2547: got (0) Success 2024-09-21T18:09:20.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 15 v 0 (ObjNum 1079 snap 0 seq_num 1079) dirty exists 2024-09-21T18:09:20.296 INFO:tasks.rados.rados.0.smithi160.stdout:2558: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:20.344 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2149907 2024-09-21T18:09:20.355 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:20.349+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:20.382 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:20.450 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:20.451 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:20.451 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:20.451 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:20.452 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:20.452 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:20.453 INFO:tasks.rados.rados.0.smithi160.stdout:2560: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-21T18:09:20.453 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2780067 2024-09-21T18:09:20.453 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 492676 2024-09-21T18:09:20.456 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:20.456 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:20.456 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:20.451+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2559: got (0) Success 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 378 v 0 (ObjNum 733 snap 0 seq_num 733) dirty exists 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2547: done (5 left) 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2558: done (4 left) 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2559: done (3 left) 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2560: done (2 left) 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2561: done (1 left) 2024-09-21T18:09:20.459 INFO:tasks.rados.rados.0.smithi160.stdout:2562: done (0 left) 2024-09-21T18:09:20.460 INFO:tasks.rados.rados.0.smithi160.stdout:2563: read oid 45 snap -1 2024-09-21T18:09:20.460 INFO:tasks.rados.rados.0.smithi160.stdout:2563: expect deleted 2024-09-21T18:09:20.460 INFO:tasks.rados.rados.0.smithi160.stdout:2564: write (excl) oid 368 current snap is 0 2024-09-21T18:09:20.460 INFO:tasks.rados.rados.0.smithi160.stdout:2564: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-21T18:09:20.465 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:20.465 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:20.470 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:20.477 INFO:tasks.rados.rados.0.smithi160.stdout:2564: writing smithi16030472-368 from 514611 to 1100050 tid 1 2024-09-21T18:09:20.486 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:20.493 INFO:tasks.rados.rados.0.smithi160.stdout:2564: writing smithi16030472-368 from 1524437 to 2116832 tid 2 2024-09-21T18:09:20.508 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:20.508 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:20.511 INFO:tasks.rados.rados.0.smithi160.stdout:2564: writing smithi16030472-368 from 2895994 to 3304418 tid 3 2024-09-21T18:09:20.511 INFO:tasks.rados.rados.0.smithi160.stdout:2565: delete oid 245 current snap is 0 2024-09-21T18:09:20.528 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:20.548 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:20.548 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:20.557 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:20.551+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:20.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:20.572 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:20.600 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:20.600 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:20.602 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:20.602 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:20.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:20.624 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:20.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:20.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:20.658 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:20.653+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:20.667 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:20.668 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:20.684 INFO:tasks.rados.rados.0.smithi160.stdout:2564: finishing write tid 1 to smithi16030472-368 2024-09-21T18:09:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:2564: finishing write tid 2 to smithi16030472-368 2024-09-21T18:09:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:2564: finishing write tid 3 to smithi16030472-368 2024-09-21T18:09:20.688 INFO:tasks.rados.rados.0.smithi160.stdout:2564: finishing write tid 4 to smithi16030472-368 2024-09-21T18:09:20.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:20.696 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:20.747 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:20.748 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:20.748 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:20.748 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:20.759 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:20.755+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:20.762 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: finishing write tid 5 to smithi16030472-368 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: oid 368 updating version 0 to 801 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: oid 368 version 801 is already newer than 800 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: oid 368 version 801 is already newer than 799 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: oid 368 updating version 801 to 802 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 368 v 802 (ObjNum 1173 snap 0 seq_num 1173) dirty exists 2024-09-21T18:09:20.784 INFO:tasks.rados.rados.0.smithi160.stdout:2564: left oid 368 (ObjNum 1173 snap 0 seq_num 1173) 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2563: done (2 left) 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2564: done (1 left) 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2565: done (0 left) 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2566: cache_flush oid 283 snap -1 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2567: cache_try_flush oid 388 snap -1 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2568: copy_from oid 123 from oid 259 current snap is 0 2024-09-21T18:09:20.789 INFO:tasks.rados.rados.0.smithi160.stdout:2569: write (excl) oid 400 current snap is 0 2024-09-21T18:09:20.790 INFO:tasks.rados.rados.0.smithi160.stdout:2569: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2024-09-21T18:09:20.806 INFO:tasks.rados.rados.0.smithi160.stdout:2569: writing smithi16030472-400 from 548225 to 1284162 tid 1 2024-09-21T18:09:20.822 INFO:tasks.rados.rados.0.smithi160.stdout:2569: writing smithi16030472-400 from 1925048 to 2521112 tid 2 2024-09-21T18:09:20.835 INFO:tasks.rados.rados.0.smithi160.stdout:2569: writing smithi16030472-400 from 2950840 to 3498161 tid 3 2024-09-21T18:09:20.835 INFO:tasks.rados.rados.0.smithi160.stdout:2570: read oid 381 snap -1 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2570: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2571: cache_evict oid 77 snap -1 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2568: finishing copy_from to smithi16030472-123 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2568: got expected ENOENT (src dne) 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2568: finishing copy_from racing read to smithi16030472-123 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2568: done (5 left) 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2572: write (excl) oid 432 current snap is 0 2024-09-21T18:09:20.836 INFO:tasks.rados.rados.0.smithi160.stdout:2572: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-21T18:09:20.847 INFO:tasks.rados.rados.0.smithi160.stdout:2572: writing smithi16030472-432 from 565032 to 976218 tid 1 2024-09-21T18:09:20.859 INFO:tasks.rados.rados.0.smithi160.stdout:2572: writing smithi16030472-432 from 1583530 to 2039605 tid 2 2024-09-21T18:09:20.860 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:20.855+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:20.874 INFO:tasks.rados.rados.0.smithi160.stdout:2572: writing smithi16030472-432 from 2578263 to 3336856 tid 3 2024-09-21T18:09:20.874 INFO:tasks.rados.rados.0.smithi160.stdout:2573: delete oid 125 current snap is 0 2024-09-21T18:09:20.874 INFO:tasks.rados.rados.0.smithi160.stdout:2572: finishing write tid 1 to smithi16030472-432 2024-09-21T18:09:20.874 INFO:tasks.rados.rados.0.smithi160.stdout:2572: finishing write tid 2 to smithi16030472-432 2024-09-21T18:09:20.881 INFO:tasks.rados.rados.0.smithi160.stdout:2572: finishing write tid 3 to smithi16030472-432 2024-09-21T18:09:20.881 INFO:tasks.rados.rados.0.smithi160.stdout:2572: finishing write tid 4 to smithi16030472-432 2024-09-21T18:09:20.881 INFO:tasks.rados.rados.0.smithi160.stdout:2572: finishing write tid 5 to smithi16030472-432 2024-09-21T18:09:20.881 INFO:tasks.rados.rados.0.smithi160.stdout:2572: oid 432 updating version 0 to 815 2024-09-21T18:09:20.882 INFO:tasks.rados.rados.0.smithi160.stdout:2572: oid 432 updating version 815 to 818 2024-09-21T18:09:20.882 INFO:tasks.rados.rados.0.smithi160.stdout:2572: oid 432 version 818 is already newer than 817 2024-09-21T18:09:20.882 INFO:tasks.rados.rados.0.smithi160.stdout:2572: oid 432 version 818 is already newer than 816 2024-09-21T18:09:20.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 432 v 818 (ObjNum 1176 snap 0 seq_num 1176) dirty exists 2024-09-21T18:09:20.882 INFO:tasks.rados.rados.0.smithi160.stdout:2572: left oid 432 (ObjNum 1176 snap 0 seq_num 1176) 2024-09-21T18:09:20.929 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:20.929 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:20.929 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:20.929 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:20.929 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:20.936 INFO:tasks.rados.rados.0.smithi160.stdout:2569: finishing write tid 1 to smithi16030472-400 2024-09-21T18:09:20.936 INFO:tasks.rados.rados.0.smithi160.stdout:2566: got (0) Success 2024-09-21T18:09:20.937 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 283 v 0 (ObjNum 129 snap 0 seq_num 3149870424) clean dne 2024-09-21T18:09:20.944 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:20.961 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:20.956+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:20.982 INFO:tasks.rados.rados.0.smithi160.stdout:2566: done (6 left) 2024-09-21T18:09:20.982 INFO:tasks.rados.rados.0.smithi160.stdout:2572: done (5 left) 2024-09-21T18:09:20.982 INFO:tasks.rados.rados.0.smithi160.stdout:2573: done (4 left) 2024-09-21T18:09:20.983 INFO:tasks.rados.rados.0.smithi160.stdout:2574: copy_from oid 70 from oid 22 current snap is 0 2024-09-21T18:09:20.983 INFO:tasks.rados.rados.0.smithi160.stdout:2575: write oid 1 current snap is 0 2024-09-21T18:09:20.983 INFO:tasks.rados.rados.0.smithi160.stdout:2575: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-21T18:09:20.993 INFO:tasks.rados.rados.0.smithi160.stdout:2575: writing smithi16030472-1 from 598646 to 1160330 tid 1 2024-09-21T18:09:21.002 INFO:tasks.rados.rados.0.smithi160.stdout:2575: writing smithi16030472-1 from 1700494 to 2160238 tid 2 2024-09-21T18:09:21.014 INFO:tasks.rados.rados.0.smithi160.stdout:2575: writing smithi16030472-1 from 2633109 to 3414246 tid 3 2024-09-21T18:09:21.014 INFO:tasks.rados.rados.0.smithi160.stdout:2576: cache_flush oid 470 snap -1 2024-09-21T18:09:21.014 INFO:tasks.rados.rados.0.smithi160.stdout:2577: write (excl) oid 308 current snap is 0 2024-09-21T18:09:21.014 INFO:tasks.rados.rados.0.smithi160.stdout:2577: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-09-21T18:09:21.027 INFO:tasks.rados.rados.0.smithi160.stdout:2577: writing smithi16030472-308 from 615453 to 1368739 tid 1 2024-09-21T18:09:21.039 INFO:tasks.rados.rados.0.smithi160.stdout:2577: writing smithi16030472-308 from 2158976 to 2762378 tid 2 2024-09-21T18:09:21.047 INFO:tasks.rados.rados.0.smithi160.stdout:2577: writing smithi16030472-308 from 3460532 to 3815453 tid 3 2024-09-21T18:09:21.047 INFO:tasks.rados.rados.0.smithi160.stdout:2578: cache_flush oid 241 snap -1 2024-09-21T18:09:21.047 INFO:tasks.rados.rados.0.smithi160.stdout:2579: cache_flush oid 78 snap -1 2024-09-21T18:09:21.048 INFO:tasks.rados.rados.0.smithi160.stdout:2580: read oid 249 snap -1 2024-09-21T18:09:21.048 INFO:tasks.rados.rados.0.smithi160.stdout:2580: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:09:21.048 INFO:tasks.rados.rados.0.smithi160.stdout:2581: delete oid 416 current snap is 0 2024-09-21T18:09:21.062 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:21.059+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:21.099 INFO:tasks.rados.rados.0.smithi160.stdout:2574: finishing copy_from racing read to smithi16030472-70 2024-09-21T18:09:21.099 INFO:tasks.rados.rados.0.smithi160.stdout:2574: finishing copy_from to smithi16030472-70 2024-09-21T18:09:21.099 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 27 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:09:21.116 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.116 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 122.78300789799999, 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:21.117 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.118 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:09:21.119 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:09:21.120 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.943155902, 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:09:21.121 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.122 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:09:21.123 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:09:21.124 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.125 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.839435454, 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.126 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.127 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.129 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.838660711, 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.130 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.131 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.132 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.133 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.83801269200001, 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.134 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.135 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.136 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.137 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:09:21.138 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.139 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.824140997, 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.140 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.141 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:09:21.142 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:09:21.143 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.82119998100001, 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.144 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.145 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.146 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.147 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.148 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.81818313300001, 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.149 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.150 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:09:21.151 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.152 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.153 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.815971551, 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.154 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.155 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:09:21.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.710634459, 2024-09-21T18:09:21.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:09:21.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.162 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.01921208500001, 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.163 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:09:21.167 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.016182085, 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:21.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:09:21.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:09:21.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 111.049618123, 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.172 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.173 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:09:21.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:09:21.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 109.38278268800001, 2024-09-21T18:09:21.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:09:21.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:09:21.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.079291607, 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:21.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:09:21.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:09:21.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.076994574, 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:09:21.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:21.195 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.07494412600001, 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:21.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:09:21.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:09:21.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:09:21.199 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.072788765, 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:09:21.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:09:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.07202600700001, 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 101.970120962, 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:21.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:21.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:21.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:21.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:21.216 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:21.218 INFO:tasks.rados.rados.0.smithi160.stdout:2569: finishing write tid 2 to smithi16030472-400 2024-09-21T18:09:21.218 INFO:tasks.rados.rados.0.smithi160.stdout:2569: finishing write tid 3 to smithi16030472-400 2024-09-21T18:09:21.218 INFO:tasks.rados.rados.0.smithi160.stdout:2569: finishing write tid 4 to smithi16030472-400 2024-09-21T18:09:21.218 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:21.219 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:21.212+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:2567: got (0) Success 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 388 v 0 (ObjNum 930 snap 0 seq_num 930) dirty exists 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:2569: finishing write tid 5 to smithi16030472-400 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:2569: oid 400 updating version 0 to 821 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:2569: oid 400 updating version 821 to 823 2024-09-21T18:09:21.241 INFO:tasks.rados.rados.0.smithi160.stdout:2569: oid 400 version 823 is already newer than 820 2024-09-21T18:09:21.242 INFO:tasks.rados.rados.0.smithi160.stdout:2569: oid 400 version 823 is already newer than 822 2024-09-21T18:09:21.242 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 400 v 823 (ObjNum 1175 snap 0 seq_num 1175) dirty exists 2024-09-21T18:09:21.242 INFO:tasks.rados.rados.0.smithi160.stdout:2569: left oid 400 (ObjNum 1175 snap 0 seq_num 1175) 2024-09-21T18:09:21.242 INFO:tasks.rados.rados.0.smithi160.stdout:2577: finishing write tid 1 to smithi16030472-308 2024-09-21T18:09:21.244 INFO:tasks.rados.rados.0.smithi160.stdout:2577: finishing write tid 2 to smithi16030472-308 2024-09-21T18:09:21.244 INFO:tasks.rados.rados.0.smithi160.stdout:2577: finishing write tid 3 to smithi16030472-308 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2567: done (11 left) 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2569: done (10 left) 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2574: done (9 left) 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2581: done (8 left) 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2582: write (excl) oid 433 current snap is 0 2024-09-21T18:09:21.252 INFO:tasks.rados.rados.0.smithi160.stdout:2582: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-21T18:09:21.261 INFO:tasks.rados.rados.0.smithi160.stdout:2582: writing smithi16030472-433 from 649067 to 1152851 tid 1 2024-09-21T18:09:21.270 INFO:tasks.rados.rados.0.smithi160.stdout:2582: writing smithi16030472-433 from 1875940 to 2483011 tid 2 2024-09-21T18:09:21.281 INFO:tasks.rados.rados.0.smithi160.stdout:2582: writing smithi16030472-433 from 3115378 to 3846684 tid 3 2024-09-21T18:09:21.281 INFO:tasks.rados.rados.0.smithi160.stdout:2583: read oid 334 snap -1 2024-09-21T18:09:21.281 INFO:tasks.rados.rados.0.smithi160.stdout:2583: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:21.281 INFO:tasks.rados.rados.0.smithi160.stdout:2584: copy_from oid 46 from oid 13 current snap is 0 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:2585: cache_try_flush oid 229 snap -1 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:2576: got (0) Success 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 470 v 0 (ObjNum 643 snap 0 seq_num 643) dirty exists 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:2576: done (11 left) 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:2586: cache_flush oid 453 snap -1 2024-09-21T18:09:21.282 INFO:tasks.rados.rados.0.smithi160.stdout:2587: cache_flush oid 28 snap -1 2024-09-21T18:09:21.283 INFO:tasks.rados.rados.0.smithi160.stdout:2588: cache_try_flush oid 440 snap -1 2024-09-21T18:09:21.283 INFO:tasks.rados.rados.0.smithi160.stdout:2589: write oid 289 current snap is 0 2024-09-21T18:09:21.283 INFO:tasks.rados.rados.0.smithi160.stdout:2589: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-21T18:09:21.290 INFO:tasks.rados.rados.0.smithi160.stdout:2589: writing smithi16030472-289 from 665874 to 1244907 tid 1 2024-09-21T18:09:21.299 INFO:tasks.rados.rados.0.smithi160.stdout:2589: writing smithi16030472-289 from 1934422 to 2401504 tid 2 2024-09-21T18:09:21.309 INFO:tasks.rados.rados.0.smithi160.stdout:2589: writing smithi16030472-289 from 3142801 to 3685379 tid 3 2024-09-21T18:09:21.309 INFO:tasks.rados.rados.0.smithi160.stdout:2590: read oid 405 snap -1 2024-09-21T18:09:21.309 INFO:tasks.rados.rados.0.smithi160.stdout:2590: expect (ObjNum 1146 snap 0 seq_num 1146) 2024-09-21T18:09:21.309 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:21.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:09:21.318 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:21.314+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:21.343 INFO:tasks.rados.rados.0.smithi160.stdout:2571: got (2) No such file or directory 2024-09-21T18:09:21.344 INFO:tasks.rados.rados.0.smithi160.stdout:2571: done (15 left) 2024-09-21T18:09:21.344 INFO:tasks.rados.rados.0.smithi160.stdout:2591: cache_flush oid 41 snap -1 2024-09-21T18:09:21.344 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:2588: got (0) Success 2024-09-21T18:09:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 440 v 0 (ObjNum 177 snap 0 seq_num 3138262304) clean dne 2024-09-21T18:09:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:2588: done (15 left) 2024-09-21T18:09:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:2592: write oid 361 current snap is 0 2024-09-21T18:09:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:2592: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2024-09-21T18:09:21.365 INFO:tasks.rados.rados.0.smithi160.stdout:2592: writing smithi16030472-361 from 682681 to 1336963 tid 1 2024-09-21T18:09:21.375 INFO:tasks.rados.rados.0.smithi160.stdout:2592: writing smithi16030472-361 from 1876551 to 2487291 tid 2 2024-09-21T18:09:21.385 INFO:tasks.rados.rados.0.smithi160.stdout:2592: writing smithi16030472-361 from 3053871 to 3524074 tid 3 2024-09-21T18:09:21.385 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:21.385 INFO:tasks.rados.rados.0.smithi160.stdout:2587: got (0) Success 2024-09-21T18:09:21.385 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:2591: got (0) Success 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3131692952) clean dne 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:2587: done (15 left) 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:2591: done (14 left) 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:2593: copy_from oid 103 from oid 301 current snap is 0 2024-09-21T18:09:21.386 INFO:tasks.rados.rados.0.smithi160.stdout:2594: delete oid 39 current snap is 0 2024-09-21T18:09:21.393 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.394 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:21.394 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:21.394 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:21.408 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:21.419 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:21.413+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:21.454 INFO:tasks.rados.rados.0.smithi160.stdout:2575: finishing write tid 1 to smithi16030472-1 2024-09-21T18:09:21.454 INFO:tasks.rados.rados.0.smithi160.stdout:2575: finishing write tid 2 to smithi16030472-1 2024-09-21T18:09:21.457 INFO:tasks.rados.rados.0.smithi160.stdout:2580: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:09:21.457 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 307583 2024-09-21T18:09:21.520 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:21.516+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:21.533 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.533 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:21.533 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:21.533 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:21.533 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:21.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:21.620 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:21.618+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:21.671 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.671 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:21.671 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:21.671 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:09:21.671 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:21.686 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:21.688 INFO:tasks.rados.rados.0.smithi160.stdout:2577: finishing write tid 4 to smithi16030472-308 2024-09-21T18:09:21.688 INFO:tasks.rados.rados.0.smithi160.stdout:2578: got (0) Success 2024-09-21T18:09:21.688 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 241 v 0 (ObjNum 943 snap 0 seq_num 943) dirty exists 2024-09-21T18:09:21.688 INFO:tasks.rados.rados.0.smithi160.stdout:2593: finishing copy_from racing read to smithi16030472-103 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2593: finishing copy_from to smithi16030472-103 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 253 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: finishing write tid 3 to smithi16030472-1 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: finishing write tid 4 to smithi16030472-1 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: finishing write tid 5 to smithi16030472-1 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: oid 1 updating version 0 to 827 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: oid 1 version 827 is already newer than 825 2024-09-21T18:09:21.689 INFO:tasks.rados.rados.0.smithi160.stdout:2575: oid 1 version 827 is already newer than 824 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2575: oid 1 version 827 is already newer than 826 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 1 v 827 (ObjNum 1178 snap 0 seq_num 1178) dirty exists 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2575: left oid 1 (ObjNum 1178 snap 0 seq_num 1178) 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2577: finishing write tid 5 to smithi16030472-308 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2577: oid 308 updating version 0 to 856 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2577: oid 308 version 856 is already newer than 855 2024-09-21T18:09:21.690 INFO:tasks.rados.rados.0.smithi160.stdout:2577: oid 308 updating version 856 to 857 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:2577: oid 308 version 857 is already newer than 854 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 857 (ObjNum 1179 snap 0 seq_num 1179) dirty exists 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:2577: left oid 308 (ObjNum 1179 snap 0 seq_num 1179) 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:2582: finishing write tid 1 to smithi16030472-433 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:2586: got (0) Success 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 453 v 0 (ObjNum 1072 snap 0 seq_num 1072) dirty exists 2024-09-21T18:09:21.691 INFO:tasks.rados.rados.0.smithi160.stdout:2582: finishing write tid 2 to smithi16030472-433 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: finishing write tid 3 to smithi16030472-433 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: finishing write tid 4 to smithi16030472-433 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: finishing write tid 5 to smithi16030472-433 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: oid 433 updating version 0 to 821 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: oid 433 version 821 is already newer than 820 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: oid 433 updating version 821 to 822 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:2582: oid 433 version 822 is already newer than 819 2024-09-21T18:09:21.692 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 822 (ObjNum 1181 snap 0 seq_num 1181) dirty exists 2024-09-21T18:09:21.693 INFO:tasks.rados.rados.0.smithi160.stdout:2582: left oid 433 (ObjNum 1181 snap 0 seq_num 1181) 2024-09-21T18:09:21.693 INFO:tasks.rados.rados.0.smithi160.stdout:2592: finishing write tid 1 to smithi16030472-361 2024-09-21T18:09:21.693 INFO:tasks.rados.rados.0.smithi160.stdout:2583: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:09:21.714 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:21.738 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:21.735+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:21.761 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2462439 2024-09-21T18:09:21.774 INFO:tasks.rados.rados.0.smithi160.stdout:2589: finishing write tid 1 to smithi16030472-289 2024-09-21T18:09:21.775 INFO:tasks.rados.rados.0.smithi160.stdout:2590: expect (ObjNum 1146 snap 0 seq_num 1146) 2024-09-21T18:09:21.813 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.813 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:21.813 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:21.813 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:21.822 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:21.817+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:21.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: finishing write tid 2 to smithi16030472-361 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: finishing write tid 3 to smithi16030472-361 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: finishing write tid 4 to smithi16030472-361 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: finishing write tid 5 to smithi16030472-361 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: oid 361 updating version 0 to 824 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: oid 361 updating version 824 to 825 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: oid 361 updating version 825 to 826 2024-09-21T18:09:21.874 INFO:tasks.rados.rados.0.smithi160.stdout:2592: oid 361 version 826 is already newer than 823 2024-09-21T18:09:21.875 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 361 v 826 (ObjNum 1183 snap 0 seq_num 1183) dirty exists 2024-09-21T18:09:21.875 INFO:tasks.rados.rados.0.smithi160.stdout:2592: left oid 361 (ObjNum 1183 snap 0 seq_num 1183) 2024-09-21T18:09:21.875 INFO:tasks.rados.rados.0.smithi160.stdout:2584: finishing copy_from racing read to smithi16030472-46 2024-09-21T18:09:21.875 INFO:tasks.rados.rados.0.smithi160.stdout:2589: finishing write tid 2 to smithi16030472-289 2024-09-21T18:09:21.875 INFO:tasks.rados.rados.0.smithi160.stdout:2589: finishing write tid 3 to smithi16030472-289 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2585: got (0) Success 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 229 v 0 (ObjNum 1050 snap 0 seq_num 1050) dirty exists 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: finishing write tid 4 to smithi16030472-289 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: finishing write tid 5 to smithi16030472-289 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: oid 289 updating version 0 to 858 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: oid 289 updating version 858 to 859 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: oid 289 updating version 859 to 861 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: oid 289 version 861 is already newer than 860 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 861 (ObjNum 1182 snap 0 seq_num 1182) dirty exists 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2589: left oid 289 (ObjNum 1182 snap 0 seq_num 1182) 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2575: done (15 left) 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2577: done (14 left) 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2578: done (13 left) 2024-09-21T18:09:21.876 INFO:tasks.rados.rados.0.smithi160.stdout:2580: done (12 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2582: done (11 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2583: done (10 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2585: done (9 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2586: done (8 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2589: done (7 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2590: done (6 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2592: done (5 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2593: done (4 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2594: done (3 left) 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2595: copy_from oid 202 from oid 340 current snap is 0 2024-09-21T18:09:21.877 INFO:tasks.rados.rados.0.smithi160.stdout:2596: delete oid 149 current snap is 0 2024-09-21T18:09:21.923 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:21.919+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:21.973 INFO:tasks.rados.rados.0.smithi160.stdout:2579: got (0) Success 2024-09-21T18:09:21.973 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 0 (ObjNum 1102 snap 0 seq_num 1102) dirty exists 2024-09-21T18:09:21.973 INFO:tasks.rados.rados.0.smithi160.stdout:2584: finishing copy_from to smithi16030472-46 2024-09-21T18:09:21.974 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T18:09:21.974 INFO:tasks.rados.rados.0.smithi160.stdout:2570: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:21.975 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:21.990 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:21.990 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:21.990 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:21.990 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:21.991 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:22.006 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:22.024 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:22.020+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2595: finishing copy_from racing read to smithi16030472-202 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2570: done (4 left) 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2579: done (3 left) 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2584: done (2 left) 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2596: done (1 left) 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2597: write oid 27 current snap is 0 2024-09-21T18:09:22.031 INFO:tasks.rados.rados.0.smithi160.stdout:2597: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-21T18:09:22.038 INFO:tasks.rados.rados.0.smithi160.stdout:2597: writing smithi16030472-27 from 733102 to 1329484 tid 1 2024-09-21T18:09:22.045 INFO:tasks.rados.rados.0.smithi160.stdout:2597: writing smithi16030472-27 from 1768350 to 2526417 tid 2 2024-09-21T18:09:22.052 INFO:tasks.rados.rados.0.smithi160.stdout:2597: writing smithi16030472-27 from 3136140 to 3840159 tid 3 2024-09-21T18:09:22.052 INFO:tasks.rados.rados.0.smithi160.stdout:2598: write oid 237 current snap is 0 2024-09-21T18:09:22.052 INFO:tasks.rados.rados.0.smithi160.stdout:2598: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-21T18:09:22.060 INFO:tasks.rados.rados.0.smithi160.stdout:2598: writing smithi16030472-237 from 749909 to 1421540 tid 1 2024-09-21T18:09:22.068 INFO:tasks.rados.rados.0.smithi160.stdout:2598: writing smithi16030472-237 from 2110479 to 2728557 tid 2 2024-09-21T18:09:22.076 INFO:tasks.rados.rados.0.smithi160.stdout:2598: writing smithi16030472-237 from 3163563 to 3678854 tid 3 2024-09-21T18:09:22.076 INFO:tasks.rados.rados.0.smithi160.stdout:2599: cache_try_flush oid 88 snap -1 2024-09-21T18:09:22.076 INFO:tasks.rados.rados.0.smithi160.stdout:2600: cache_flush oid 142 snap -1 2024-09-21T18:09:22.076 INFO:tasks.rados.rados.0.smithi160.stdout:2601: delete oid 224 current snap is 0 2024-09-21T18:09:22.077 INFO:tasks.rados.rados.0.smithi160.stdout:2600: got (0) Success 2024-09-21T18:09:22.077 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 0 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:09:22.125 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:22.120+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:22.176 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 110.577863748, 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.177 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:09:22.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:09:22.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.147893800999995, 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.182 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:09:22.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:09:22.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:09:22.186 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.050373425999993, 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:22.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:09:22.190 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:09:22.191 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.049689176000001, 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.192 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:09:22.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:09:22.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.048855821000004, 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.201 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:09:22.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:09:22.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.208 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.046137191, 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.209 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:09:22.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:09:22.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:09:22.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.950328784999996, 2024-09-21T18:09:22.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:09:22.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:22.218 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 87.948024243999996, 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:09:22.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:09:22.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.970395624000005, 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:09:22.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:09:22.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.869239941999993, 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:09:22.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.868444280000006, 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:09:22.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.867308460000004, 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.235 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.846590867000003, 2024-09-21T18:09:22.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:09:22.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:09:22.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.837238182999997, 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:09:22.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:09:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.831863534999997, 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:09:22.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 68.829315260000001, 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:09:22.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.601614722999997, 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.256 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:09:22.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.600948430000003, 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:09:22.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 62.600856317999998, 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:09:22.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:6894 4.2 e96/33 4:60b99da3:::smithi16030472-227:head v 96'728, mlcod=96'726)", 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 6.3302158750000004, 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.303226251, 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843452+0000", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0429999999999999e-06 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844181+0000", 2024-09-21T18:09:22.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072923900000000002 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844209+0000", 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7313000000000001e-05 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944352+0000", 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100143339 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944508+0000", 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015568600000000001 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944543+0000", 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5649999999999999e-05 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146570+0000", 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20202621900000001 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146675+0000", 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010576199999999999 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.273 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:22.275 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2595: finishing copy_from to smithi16030472-202 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 202 v 234 (ObjNum 339 snap 0 seq_num 339) dirty exists 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2595: done (5 left) 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2600: done (4 left) 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2601: done (3 left) 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2602: write (excl) oid 375 current snap is 0 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2602: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2602: writing smithi16030472-375 from 783523 to 1205652 tid 1 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2602: writing smithi16030472-375 from 1827443 to 2449190 tid 2 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2602: writing smithi16030472-375 from 3218409 to 3872597 tid 3 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2603: write (excl) oid 383 current snap is 0 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2603: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2603: writing smithi16030472-383 from 400330 to 897708 tid 1 2024-09-21T18:09:22.277 INFO:tasks.rados.rados.0.smithi160.stdout:2603: writing smithi16030472-383 from 1485925 to 2000330 tid 2 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2604: write oid 350 current snap is 0 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2604: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2604: writing smithi16030472-350 from 417137 to 989764 tid 1 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2604: writing smithi16030472-350 from 1428054 to 2017137 tid 2 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2605: read oid 41 snap -1 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2605: expect deleted 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2606: cache_evict oid 147 snap -1 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2607: write oid 221 current snap is 0 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2607: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2607: writing smithi16030472-221 from 433944 to 1081820 tid 1 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2607: writing smithi16030472-221 from 1486536 to 2033944 tid 2 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2608: write (excl) oid 214 current snap is 0 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2608: seq_num 1193 ranges {450751=723125,1945018=105733} 2024-09-21T18:09:22.278 INFO:tasks.rados.rados.0.smithi160.stdout:2608: writing smithi16030472-214 from 450751 to 1173876 tid 1 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2608: writing smithi16030472-214 from 1945018 to 2050751 tid 2 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2609: read oid 105 snap -1 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2609: expect deleted 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2610: cache_try_flush oid 269 snap -1 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2606: got (16) Device or resource busy 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2606: done (11 left) 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2611: write (excl) oid 24 current snap is 0 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2611: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2611: writing smithi16030472-24 from 467558 to 982285 tid 1 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2611: writing smithi16030472-24 from 1719853 to 2067558 tid 2 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2612: cache_try_flush oid 215 snap -1 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2613: copy_from oid 169 from oid 35 current snap is 0 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2614: write oid 103 current snap is 0 2024-09-21T18:09:22.279 INFO:tasks.rados.rados.0.smithi160.stdout:2614: seq_num 1195 ranges {484365=589976,1661982=422383} 2024-09-21T18:09:22.280 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:22.270+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:22.280 INFO:tasks.rados.rados.0.smithi160.stdout:2614: writing smithi16030472-103 from 484365 to 1074341 tid 1 2024-09-21T18:09:22.284 INFO:tasks.rados.rados.0.smithi160.stdout:2614: writing smithi16030472-103 from 1661982 to 2084365 tid 2 2024-09-21T18:09:22.284 INFO:tasks.rados.rados.0.smithi160.stdout:2615: delete oid 343 current snap is 0 2024-09-21T18:09:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:09:22.366 INFO:tasks.rados.rados.0.smithi160.stdout:2597: finishing write tid 1 to smithi16030472-27 2024-09-21T18:09:22.366 INFO:tasks.rados.rados.0.smithi160.stdout:2597: finishing write tid 2 to smithi16030472-27 2024-09-21T18:09:22.366 INFO:tasks.rados.rados.0.smithi160.stdout:2597: finishing write tid 3 to smithi16030472-27 2024-09-21T18:09:22.366 INFO:tasks.rados.rados.0.smithi160.stdout:2597: finishing write tid 4 to smithi16030472-27 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: finishing write tid 5 to smithi16030472-27 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: oid 27 updating version 0 to 830 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: oid 27 version 830 is already newer than 829 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: oid 27 updating version 830 to 831 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: oid 27 updating version 831 to 832 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 832 (ObjNum 1186 snap 0 seq_num 1186) dirty exists 2024-09-21T18:09:22.367 INFO:tasks.rados.rados.0.smithi160.stdout:2597: left oid 27 (ObjNum 1186 snap 0 seq_num 1186) 2024-09-21T18:09:22.375 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:22.369+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:22.413 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:22.413 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:22.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.413 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:7147 3.1 e100/31)", 2024-09-21T18:09:22.413 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:22.313114+0000", 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.099868291999999997, 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099903730999999996, 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313114+0000", 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313114+0000", 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313116+0000", 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.508e-06 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313122+0000", 2024-09-21T18:09:22.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3829999999999997e-06 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313133+0000", 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1015999999999999e-05 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:22.313143+0000", 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0011e-05 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.416 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:09:22.417 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:09:22.417 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:22.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:22.476 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:22.472+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:22.508 INFO:tasks.rados.rados.0.smithi160.stdout:2598: finishing write tid 1 to smithi16030472-237 2024-09-21T18:09:22.509 INFO:tasks.rados.rados.0.smithi160.stdout:2598: finishing write tid 2 to smithi16030472-237 2024-09-21T18:09:22.552 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:22.552 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:22.553 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:22.553 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:22.553 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:22.567 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:22.571 INFO:tasks.rados.rados.0.smithi160.stdout:2598: finishing write tid 3 to smithi16030472-237 2024-09-21T18:09:22.571 INFO:tasks.rados.rados.0.smithi160.stdout:2598: finishing write tid 4 to smithi16030472-237 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: finishing write tid 5 to smithi16030472-237 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: oid 237 updating version 0 to 835 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: oid 237 updating version 835 to 836 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: oid 237 version 836 is already newer than 833 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: oid 237 version 836 is already newer than 834 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 237 v 836 (ObjNum 1187 snap 0 seq_num 1187) dirty exists 2024-09-21T18:09:22.572 INFO:tasks.rados.rados.0.smithi160.stdout:2598: left oid 237 (ObjNum 1187 snap 0 seq_num 1187) 2024-09-21T18:09:22.600 INFO:tasks.rados.rados.0.smithi160.stdout:2599: got (0) Success 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 0 (ObjNum 1103 snap 0 seq_num 1103) dirty exists 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2602: finishing write tid 1 to smithi16030472-375 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2602: finishing write tid 2 to smithi16030472-375 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2602: finishing write tid 3 to smithi16030472-375 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2603: finishing write tid 1 to smithi16030472-383 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2603: finishing write tid 2 to smithi16030472-383 2024-09-21T18:09:22.601 INFO:tasks.rados.rados.0.smithi160.stdout:2603: finishing write tid 3 to smithi16030472-383 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:2603: finishing write tid 4 to smithi16030472-383 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:2603: oid 383 updating version 0 to 830 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:2603: oid 383 version 830 is already newer than 829 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:2603: oid 383 version 830 is already newer than 828 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 830 (ObjNum 1190 snap 0 seq_num 1190) dirty exists 2024-09-21T18:09:22.602 INFO:tasks.rados.rados.0.smithi160.stdout:2603: left oid 383 (ObjNum 1190 snap 0 seq_num 1190) 2024-09-21T18:09:22.603 INFO:tasks.rados.rados.0.smithi160.stdout:2602: finishing write tid 4 to smithi16030472-375 2024-09-21T18:09:22.609 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:22.603+0000 7f9bb588f640 -1 osd.10 100 handle_osd_map injecting map crc failure 2024-09-21T18:09:22.609 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:22.607+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2597: done (15 left) 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2598: done (14 left) 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2599: done (13 left) 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2603: done (12 left) 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2615: done (11 left) 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2616: copy_from oid 449 from oid 13 current snap is 0 2024-09-21T18:09:22.610 INFO:tasks.rados.rados.0.smithi160.stdout:2617: cache_evict oid 483 snap -1 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2618: read oid 139 snap -1 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2618: expect (ObjNum 978 snap 0 seq_num 978) 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2619: copy_from oid 290 from oid 355 current snap is 0 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2620: cache_evict oid 469 snap -1 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2605: done (15 left) 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2621: write oid 215 current snap is 0 2024-09-21T18:09:22.611 INFO:tasks.rados.rados.0.smithi160.stdout:2621: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-21T18:09:22.621 INFO:tasks.rados.rados.0.smithi160.stdout:2621: writing smithi16030472-215 from 517979 to 1258453 tid 1 2024-09-21T18:09:22.628 INFO:tasks.rados.rados.0.smithi160.stdout:2621: writing smithi16030472-215 from 1778946 to 2117979 tid 2 2024-09-21T18:09:22.628 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:22.628 INFO:tasks.rados.rados.0.smithi160.stdout:2618: expect (ObjNum 978 snap 0 seq_num 978) 2024-09-21T18:09:22.678 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:22.672+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 123.853763091, 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.695 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:09:22.696 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:09:22.697 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:09:22.698 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.442120797, 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.699 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.700 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.701 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:09:22.702 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:09:22.703 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.439289925, 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.704 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.705 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.706 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.707 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.435317017, 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.708 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.709 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.710 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.711 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.43220006599999, 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.712 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.713 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.714 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.715 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.43026906199999, 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.716 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:09:22.717 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.718 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.719 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.427162303, 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.720 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:09:22.721 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.722 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.723 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.424004619, 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.724 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:09:22.725 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:09:22.726 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:09:22.727 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:09:22.728 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:09:22.729 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.421771568, 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.730 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:09:22.731 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.732 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.41951989499999, 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:09:22.733 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.734 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.735 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.736 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.737 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.738 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.326942307, 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.739 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.740 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.741 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:09:22.742 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.743 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.744 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:09:22.745 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.55027293099999, 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.746 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:09:22.747 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:09:22.748 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:09:22.749 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 110.540765347, 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.750 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.751 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.752 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.753 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 110.43393536799999, 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.754 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:09:22.755 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:09:22.756 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.757 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.100563816, 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.758 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:09:22.759 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:09:22.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 98.328572347000005, 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.762 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.766 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 88.446423891999999, 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:09:22.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:09:22.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:09:22.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 63.906299443999998, 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:09:22.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:09:22.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 46.596340984000001, 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.775 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:22.779 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 43.351938672999999, 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:22.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:09:22.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:09:22.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:22.784 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:22.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:22.785 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:22.786 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:22.787 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:22.783+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2618: done (15 left) 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2622: cache_try_flush oid 137 snap -1 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2620: got (2) No such file or directory 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2604: finishing write tid 1 to smithi16030472-350 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2611: finishing write tid 1 to smithi16030472-24 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2610: got (0) Success 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 269 v 0 (ObjNum 33 snap 0 seq_num 3111029040) clean dne 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2609: done (15 left) 2024-09-21T18:09:22.827 INFO:tasks.rados.rados.0.smithi160.stdout:2610: done (14 left) 2024-09-21T18:09:22.828 INFO:tasks.rados.rados.0.smithi160.stdout:2620: done (13 left) 2024-09-21T18:09:22.828 INFO:tasks.rados.rados.0.smithi160.stdout:2623: read oid 264 snap -1 2024-09-21T18:09:22.828 INFO:tasks.rados.rados.0.smithi160.stdout:2623: expect (ObjNum 1144 snap 0 seq_num 1144) 2024-09-21T18:09:22.828 INFO:tasks.rados.rados.0.smithi160.stdout:2624: write (excl) oid 5 current snap is 0 2024-09-21T18:09:22.828 INFO:tasks.rados.rados.0.smithi160.stdout:2624: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-21T18:09:22.833 INFO:tasks.rados.rados.0.smithi160.stdout:2624: writing smithi16030472-5 from 534786 to 950509 tid 1 2024-09-21T18:09:22.843 INFO:tasks.rados.rados.0.smithi160.stdout:2624: writing smithi16030472-5 from 1437428 to 2134786 tid 2 2024-09-21T18:09:22.843 INFO:tasks.rados.rados.0.smithi160.stdout:2625: cache_evict oid 213 snap -1 2024-09-21T18:09:22.843 INFO:tasks.rados.rados.0.smithi160.stdout:2607: finishing write tid 1 to smithi16030472-221 2024-09-21T18:09:22.844 INFO:tasks.rados.rados.0.smithi160.stdout:2607: finishing write tid 2 to smithi16030472-221 2024-09-21T18:09:22.844 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:22.844 INFO:tasks.rados.rados.0.smithi160.stdout:2619: finishing copy_from to smithi16030472-290 2024-09-21T18:09:22.844 INFO:tasks.rados.rados.0.smithi160.stdout:2619: got expected ENOENT (src dne) 2024-09-21T18:09:22.844 INFO:tasks.rados.rados.0.smithi160.stdout:2623: expect (ObjNum 1144 snap 0 seq_num 1144) 2024-09-21T18:09:22.851 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 558086 2024-09-21T18:09:22.887 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:22.884+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:22.912 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:22.912 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:22.912 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:22.912 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:22.929 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:22.987 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:22.983+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:22.988 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:22.995 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:23.016 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:23.016 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:23.037 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:23.037 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2617: got (2) No such file or directory 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: finishing write tid 5 to smithi16030472-375 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: oid 375 updating version 0 to 741 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: oid 375 updating version 741 to 743 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: oid 375 version 743 is already newer than 742 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: oid 375 updating version 743 to 744 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 375 v 744 (ObjNum 1189 snap 0 seq_num 1189) dirty exists 2024-09-21T18:09:23.051 INFO:tasks.rados.rados.0.smithi160.stdout:2602: left oid 375 (ObjNum 1189 snap 0 seq_num 1189) 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2608: finishing write tid 1 to smithi16030472-214 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2602: done (15 left) 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2617: done (14 left) 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2623: done (13 left) 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2626: cache_evict oid 407 snap -1 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2619: finishing copy_from racing read to smithi16030472-290 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2619: done (13 left) 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2627: cache_flush oid 385 snap -1 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2628: write (excl) oid 425 current snap is 0 2024-09-21T18:09:23.052 INFO:tasks.rados.rados.0.smithi160.stdout:2628: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-21T18:09:23.055 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:23.064 INFO:tasks.rados.rados.0.smithi160.stdout:2628: writing smithi16030472-425 from 551593 to 1042565 tid 1 2024-09-21T18:09:23.075 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:23.075 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:23.079 INFO:tasks.rados.rados.0.smithi160.stdout:2628: writing smithi16030472-425 from 1779557 to 2151593 tid 2 2024-09-21T18:09:23.079 INFO:tasks.rados.rados.0.smithi160.stdout:2629: write oid 373 current snap is 0 2024-09-21T18:09:23.079 INFO:tasks.rados.rados.0.smithi160.stdout:2629: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-21T18:09:23.087 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:23.086+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:23.093 INFO:tasks.rados.rados.0.smithi160.stdout:2629: writing smithi16030472-373 from 568400 to 1134621 tid 1 2024-09-21T18:09:23.095 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.095 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.097 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:23.097 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:23.106 INFO:tasks.rados.rados.0.smithi160.stdout:2629: writing smithi16030472-373 from 1838039 to 2168400 tid 2 2024-09-21T18:09:23.106 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.106 INFO:tasks.rados.rados.0.smithi160.stdout:2629: finishing write tid 1 to smithi16030472-373 2024-09-21T18:09:23.109 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:23.116 INFO:tasks.rados.rados.0.smithi160.stdout:2616: finishing copy_from racing read to smithi16030472-449 2024-09-21T18:09:23.116 INFO:tasks.rados.rados.0.smithi160.stdout:2608: finishing write tid 2 to smithi16030472-214 2024-09-21T18:09:23.116 INFO:tasks.rados.rados.0.smithi160.stdout:2608: finishing write tid 3 to smithi16030472-214 2024-09-21T18:09:23.116 INFO:tasks.rados.rados.0.smithi160.stdout:2608: finishing write tid 4 to smithi16030472-214 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2608: oid 214 updating version 0 to 833 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2608: oid 214 version 833 is already newer than 831 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2608: oid 214 version 833 is already newer than 832 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 214 v 833 (ObjNum 1193 snap 0 seq_num 1193) dirty exists 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2608: left oid 214 (ObjNum 1193 snap 0 seq_num 1193) 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2608: done (15 left) 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2630: cache_flush oid 173 snap -1 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.117 INFO:tasks.rados.rados.0.smithi160.stdout:2611: finishing write tid 2 to smithi16030472-24 2024-09-21T18:09:23.119 INFO:tasks.rados.rados.0.smithi160.stdout:2616: finishing copy_from to smithi16030472-449 2024-09-21T18:09:23.119 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 12 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T18:09:23.119 INFO:tasks.rados.rados.0.smithi160.stdout:2616: done (15 left) 2024-09-21T18:09:23.119 INFO:tasks.rados.rados.0.smithi160.stdout:2631: copy_from oid 109 from oid 298 current snap is 0 2024-09-21T18:09:23.119 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.121 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:23.121 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:23.122 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:23.141 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:23.141 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: finishing write tid 2 to smithi16030472-350 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: finishing write tid 3 to smithi16030472-350 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: finishing write tid 4 to smithi16030472-350 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: oid 350 updating version 0 to 834 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: oid 350 updating version 834 to 835 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: oid 350 version 835 is already newer than 745 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 835 (ObjNum 1191 snap 0 seq_num 1191) dirty exists 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: left oid 350 (ObjNum 1191 snap 0 seq_num 1191) 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2604: done (15 left) 2024-09-21T18:09:23.142 INFO:tasks.rados.rados.0.smithi160.stdout:2632: cache_flush oid 38 snap -1 2024-09-21T18:09:23.143 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.143 INFO:tasks.rados.rados.0.smithi160.stdout:2607: finishing write tid 3 to smithi16030472-221 2024-09-21T18:09:23.151 INFO:tasks.rados.rados.0.smithi160.stdout:2626: got (16) Device or resource busy 2024-09-21T18:09:23.152 INFO:tasks.rados.rados.0.smithi160.stdout:2626: done (15 left) 2024-09-21T18:09:23.152 INFO:tasks.rados.rados.0.smithi160.stdout:2633: cache_try_flush oid 104 snap -1 2024-09-21T18:09:23.152 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.162 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:23.189 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:23.184+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:23.233 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.234 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.234 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:23.234 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:23.249 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:23.252 INFO:tasks.rados.rados.0.smithi160.stdout:2611: finishing write tid 3 to smithi16030472-24 2024-09-21T18:09:23.252 INFO:tasks.rados.rados.0.smithi160.stdout:2611: finishing write tid 4 to smithi16030472-24 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2611: oid 24 updating version 0 to 805 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2611: oid 24 updating version 805 to 807 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2611: oid 24 version 807 is already newer than 806 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 24 v 807 (ObjNum 1194 snap 0 seq_num 1194) dirty exists 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2611: left oid 24 (ObjNum 1194 snap 0 seq_num 1194) 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2614: finishing write tid 1 to smithi16030472-103 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2614: finishing write tid 2 to smithi16030472-103 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2614: finishing write tid 3 to smithi16030472-103 2024-09-21T18:09:23.253 INFO:tasks.rados.rados.0.smithi160.stdout:2614: finishing write tid 4 to smithi16030472-103 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2614: oid 103 updating version 0 to 808 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2614: oid 103 updating version 808 to 809 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2614: oid 103 updating version 809 to 810 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 810 (ObjNum 1195 snap 0 seq_num 1195) dirty exists 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2614: left oid 103 (ObjNum 1195 snap 0 seq_num 1195) 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2607: finishing write tid 4 to smithi16030472-221 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2607: oid 221 updating version 0 to 746 2024-09-21T18:09:23.254 INFO:tasks.rados.rados.0.smithi160.stdout:2607: oid 221 updating version 746 to 747 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2607: oid 221 updating version 747 to 836 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 221 v 836 (ObjNum 1192 snap 0 seq_num 1192) dirty exists 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2607: left oid 221 (ObjNum 1192 snap 0 seq_num 1192) 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2607: done (15 left) 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2611: done (14 left) 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2614: done (13 left) 2024-09-21T18:09:23.255 INFO:tasks.rados.rados.0.smithi160.stdout:2634: cache_try_flush oid 335 snap -1 2024-09-21T18:09:23.256 INFO:tasks.rados.rados.0.smithi160.stdout:2635: write oid 383 current snap is 0 2024-09-21T18:09:23.256 INFO:tasks.rados.rados.0.smithi160.stdout:2635: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-21T18:09:23.259 INFO:tasks.rados.rados.0.smithi160.stdout:2635: writing smithi16030472-383 from 585207 to 1226677 tid 1 2024-09-21T18:09:23.264 INFO:tasks.rados.rados.0.smithi160.stdout:2635: writing smithi16030472-383 from 1896521 to 2185207 tid 2 2024-09-21T18:09:23.264 INFO:tasks.rados.rados.0.smithi160.stdout:2636: cache_evict oid 443 snap -1 2024-09-21T18:09:23.264 INFO:tasks.rados.rados.0.smithi160.stdout:2634: got (0) Success 2024-09-21T18:09:23.264 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 335 v 0 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:09:23.264 INFO:tasks.rados.rados.0.smithi160.stdout:2634: done (15 left) 2024-09-21T18:09:23.265 INFO:tasks.rados.rados.0.smithi160.stdout:2637: read oid 458 snap -1 2024-09-21T18:09:23.265 INFO:tasks.rados.rados.0.smithi160.stdout:2637: expect deleted 2024-09-21T18:09:23.265 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.278 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:23.289 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:23.286+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:23.307 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:23.307 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:23.309 INFO:tasks.rados.rados.0.smithi160.stdout:2629: finishing write tid 2 to smithi16030472-373 2024-09-21T18:09:23.309 INFO:tasks.rados.rados.0.smithi160.stdout:2629: finishing write tid 3 to smithi16030472-373 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: finishing write tid 4 to smithi16030472-373 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: oid 373 updating version 0 to 839 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: oid 373 version 839 is already newer than 837 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: oid 373 version 839 is already newer than 838 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 373 v 839 (ObjNum 1200 snap 0 seq_num 1200) dirty exists 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: left oid 373 (ObjNum 1200 snap 0 seq_num 1200) 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2629: done (15 left) 2024-09-21T18:09:23.310 INFO:tasks.rados.rados.0.smithi160.stdout:2638: write (excl) oid 395 current snap is 0 2024-09-21T18:09:23.311 INFO:tasks.rados.rados.0.smithi160.stdout:2638: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-21T18:09:23.318 INFO:tasks.rados.rados.0.smithi160.stdout:2638: writing smithi16030472-395 from 602014 to 1035086 tid 1 2024-09-21T18:09:23.329 INFO:tasks.rados.rados.0.smithi160.stdout:2638: writing smithi16030472-395 from 1671356 to 2202014 tid 2 2024-09-21T18:09:23.329 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.329 INFO:tasks.rados.rados.0.smithi160.stdout:2621: finishing write tid 1 to smithi16030472-215 2024-09-21T18:09:23.329 INFO:tasks.rados.rados.0.smithi160.stdout:2621: finishing write tid 2 to smithi16030472-215 2024-09-21T18:09:23.329 INFO:tasks.rados.rados.0.smithi160.stdout:2621: finishing write tid 3 to smithi16030472-215 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: finishing write tid 4 to smithi16030472-215 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: oid 215 updating version 0 to 813 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: oid 215 version 813 is already newer than 812 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: oid 215 version 813 is already newer than 811 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 215 v 813 (ObjNum 1197 snap 0 seq_num 1197) dirty exists 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: left oid 215 (ObjNum 1197 snap 0 seq_num 1197) 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2612: got (16) Device or resource busy 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2612: done (15 left) 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2621: done (14 left) 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2639: read oid 315 snap -1 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2639: expect (ObjNum 737 snap 0 seq_num 737) 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2640: read oid 490 snap -1 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout:2640: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-21T18:09:23.330 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.340 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:23.340 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:23.367 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:23.378 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.379 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.390 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:23.387+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:23.393 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:23.394 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:23.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:23.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:23.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:23.415 INFO:tasks.rados.rados.0.smithi160.stdout:2640: expect (ObjNum 74 snap 0 seq_num 74) 2024-09-21T18:09:23.432 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3042586 2024-09-21T18:09:23.484 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2623383 2024-09-21T18:09:23.491 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:23.487+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:23.496 INFO:tasks.rados.rados.0.smithi160.stdout:2613: finishing copy_from racing read to smithi16030472-169 2024-09-21T18:09:23.496 INFO:tasks.rados.rados.0.smithi160.stdout:2625: got (2) No such file or directory 2024-09-21T18:09:23.496 INFO:tasks.rados.rados.0.smithi160.stdout:2630: got (2) No such file or directory 2024-09-21T18:09:23.496 INFO:tasks.rados.rados.0.smithi160.stdout:2624: finishing write tid 1 to smithi16030472-5 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2632: got (0) Success 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 0 (ObjNum 623 snap 0 seq_num 623) dirty exists 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2628: finishing write tid 1 to smithi16030472-425 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2628: finishing write tid 2 to smithi16030472-425 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2628: finishing write tid 3 to smithi16030472-425 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2628: finishing write tid 4 to smithi16030472-425 2024-09-21T18:09:23.497 INFO:tasks.rados.rados.0.smithi160.stdout:2628: oid 425 updating version 0 to 837 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2628: oid 425 updating version 837 to 839 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2628: oid 425 version 839 is already newer than 838 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 425 v 839 (ObjNum 1199 snap 0 seq_num 1199) dirty exists 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2628: left oid 425 (ObjNum 1199 snap 0 seq_num 1199) 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2636: got (0) Success 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2625: done (15 left) 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2628: done (14 left) 2024-09-21T18:09:23.498 INFO:tasks.rados.rados.0.smithi160.stdout:2630: done (13 left) 2024-09-21T18:09:23.499 INFO:tasks.rados.rados.0.smithi160.stdout:2632: done (12 left) 2024-09-21T18:09:23.499 INFO:tasks.rados.rados.0.smithi160.stdout:2636: done (11 left) 2024-09-21T18:09:23.499 INFO:tasks.rados.rados.0.smithi160.stdout:2640: done (10 left) 2024-09-21T18:09:23.499 INFO:tasks.rados.rados.0.smithi160.stdout:2641: write (excl) oid 465 current snap is 0 2024-09-21T18:09:23.499 INFO:tasks.rados.rados.0.smithi160.stdout:2641: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-21T18:09:23.511 INFO:tasks.rados.rados.0.smithi160.stdout:2641: writing smithi16030472-465 from 618821 to 1127142 tid 1 2024-09-21T18:09:23.524 INFO:tasks.rados.rados.0.smithi160.stdout:2641: writing smithi16030472-465 from 1613485 to 2144562 tid 2 2024-09-21T18:09:23.524 INFO:tasks.rados.rados.0.smithi160.stdout:2642: delete oid 46 current snap is 0 2024-09-21T18:09:23.543 INFO:tasks.rados.rados.0.smithi160.stdout:2633: got (0) Success 2024-09-21T18:09:23.543 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 104 v 0 (ObjNum 103 snap 0 seq_num 103) dirty exists 2024-09-21T18:09:23.545 INFO:tasks.rados.rados.0.smithi160.stdout:2613: finishing copy_from to smithi16030472-169 2024-09-21T18:09:23.546 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 169 v 729 (ObjNum 1025 snap 0 seq_num 1025) dirty exists 2024-09-21T18:09:23.546 INFO:tasks.rados.rados.0.smithi160.stdout:2624: finishing write tid 2 to smithi16030472-5 2024-09-21T18:09:23.548 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.548 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.561 INFO:tasks.rados.rados.0.smithi160.stdout:2635: finishing write tid 1 to smithi16030472-383 2024-09-21T18:09:23.561 INFO:tasks.rados.rados.0.smithi160.stdout:2635: finishing write tid 2 to smithi16030472-383 2024-09-21T18:09:23.561 INFO:tasks.rados.rados.0.smithi160.stdout:2635: finishing write tid 3 to smithi16030472-383 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:2635: finishing write tid 4 to smithi16030472-383 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:2635: oid 383 updating version 0 to 842 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:2635: oid 383 version 842 is already newer than 841 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:2635: oid 383 version 842 is already newer than 840 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 842 (ObjNum 1201 snap 0 seq_num 1201) dirty exists 2024-09-21T18:09:23.562 INFO:tasks.rados.rados.0.smithi160.stdout:2635: left oid 383 (ObjNum 1201 snap 0 seq_num 1201) 2024-09-21T18:09:23.563 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:23.567 INFO:tasks.rados.rados.0.smithi160.stdout:2639: expect (ObjNum 737 snap 0 seq_num 737) 2024-09-21T18:09:23.583 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:23.580+0000 7f4976023640 -1 osd.5 101 handle_osd_map injecting map crc failure 2024-09-21T18:09:23.610 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:23.605+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:23.615 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:2627: got (0) Success 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 0 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:2622: got (0) Success 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 137 v 0 (ObjNum 505 snap 0 seq_num 505) dirty exists 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:2624: finishing write tid 3 to smithi16030472-5 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:2624: finishing write tid 4 to smithi16030472-5 2024-09-21T18:09:23.686 INFO:tasks.rados.rados.0.smithi160.stdout:2624: oid 5 updating version 0 to 816 2024-09-21T18:09:23.687 INFO:tasks.rados.rados.0.smithi160.stdout:2624: oid 5 version 816 is already newer than 815 2024-09-21T18:09:23.687 INFO:tasks.rados.rados.0.smithi160.stdout:2624: oid 5 version 816 is already newer than 814 2024-09-21T18:09:23.687 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 5 v 816 (ObjNum 1198 snap 0 seq_num 1198) dirty exists 2024-09-21T18:09:23.687 INFO:tasks.rados.rados.0.smithi160.stdout:2624: left oid 5 (ObjNum 1198 snap 0 seq_num 1198) 2024-09-21T18:09:23.687 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.687 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.692 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:23.691+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:23.700 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:23.727 INFO:tasks.rados.rados.0.smithi160.stdout:2641: finishing write tid 1 to smithi16030472-465 2024-09-21T18:09:23.727 INFO:tasks.rados.rados.0.smithi160.stdout:2641: finishing write tid 2 to smithi16030472-465 2024-09-21T18:09:23.778 INFO:tasks.rados.rados.0.smithi160.stdout:2631: finishing copy_from racing read to smithi16030472-109 2024-09-21T18:09:23.778 INFO:tasks.rados.rados.0.smithi160.stdout:2631: finishing copy_from to smithi16030472-109 2024-09-21T18:09:23.778 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 476 (ObjNum 732 snap 0 seq_num 732) dirty exists 2024-09-21T18:09:23.793 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:23.785+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: finishing write tid 3 to smithi16030472-465 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: finishing write tid 4 to smithi16030472-465 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: oid 465 updating version 0 to 841 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: oid 465 version 841 is already newer than 840 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: oid 465 updating version 841 to 842 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 465 v 842 (ObjNum 1203 snap 0 seq_num 1203) dirty exists 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2641: left oid 465 (ObjNum 1203 snap 0 seq_num 1203) 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2613: done (11 left) 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2622: done (10 left) 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2624: done (9 left) 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2627: done (8 left) 2024-09-21T18:09:23.832 INFO:tasks.rados.rados.0.smithi160.stdout:2631: done (7 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2633: done (6 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2635: done (5 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2637: done (4 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2639: done (3 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2641: done (2 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2642: done (1 left) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2643: copy_from oid 56 from oid 427 current snap is 0 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2644: read oid 27 snap -1 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2644: expect (ObjNum 1186 snap 0 seq_num 1186) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2645: read oid 17 snap -1 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2645: expect (ObjNum 532 snap 0 seq_num 532) 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2646: write oid 481 current snap is 0 2024-09-21T18:09:23.833 INFO:tasks.rados.rados.0.smithi160.stdout:2646: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-21T18:09:23.840 INFO:tasks.rados.rados.0.smithi160.stdout:2646: writing smithi16030472-481 from 652435 to 1311254 tid 1 2024-09-21T18:09:23.847 INFO:tasks.rados.rados.0.smithi160.stdout:2646: writing smithi16030472-481 from 1730449 to 2252435 tid 2 2024-09-21T18:09:23.847 INFO:tasks.rados.rados.0.smithi160.stdout:2647: write oid 117 current snap is 0 2024-09-21T18:09:23.847 INFO:tasks.rados.rados.0.smithi160.stdout:2638: finishing write tid 1 to smithi16030472-395 2024-09-21T18:09:23.847 INFO:tasks.rados.rados.0.smithi160.stdout:2647: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-21T18:09:23.856 INFO:tasks.rados.rados.0.smithi160.stdout:2647: writing smithi16030472-117 from 669242 to 1403310 tid 1 2024-09-21T18:09:23.861 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:23.861 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2647: writing smithi16030472-117 from 2188931 to 2269242 tid 2 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2648: read oid 246 snap -1 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2648: expect deleted 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2649: cache_evict oid 105 snap -1 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2650: cache_try_flush oid 407 snap -1 2024-09-21T18:09:23.863 INFO:tasks.rados.rados.0.smithi160.stdout:2651: write (excl) oid 350 current snap is 0 2024-09-21T18:09:23.864 INFO:tasks.rados.rados.0.smithi160.stdout:2651: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-21T18:09:23.869 INFO:tasks.rados.rados.0.smithi160.stdout:2651: writing smithi16030472-350 from 686049 to 1095366 tid 1 2024-09-21T18:09:23.876 INFO:tasks.rados.rados.0.smithi160.stdout:2651: writing smithi16030472-350 from 1847413 to 2286049 tid 2 2024-09-21T18:09:23.876 INFO:tasks.rados.rados.0.smithi160.stdout:2652: write oid 24 current snap is 0 2024-09-21T18:09:23.876 INFO:tasks.rados.rados.0.smithi160.stdout:2652: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-21T18:09:23.879 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:23.883 INFO:tasks.rados.rados.0.smithi160.stdout:2652: writing smithi16030472-24 from 702856 to 1187422 tid 1 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2652: writing smithi16030472-24 from 1789542 to 2302856 tid 2 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2653: cache_evict oid 187 snap -1 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2654: copy_from oid 272 from oid 23 current snap is 0 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2655: cache_flush oid 159 snap -1 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2656: copy_from oid 397 from oid 6 current snap is 0 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout:2657: cache_evict oid 361 snap -1 2024-09-21T18:09:23.890 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:23.893 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:23.889+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:23.958 INFO:tasks.rados.rados.0.smithi160.stdout:2644: expect (ObjNum 1186 snap 0 seq_num 1186) 2024-09-21T18:09:23.995 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:23.989+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:24.020 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.020 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.034 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:24.067 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:24.064+0000 7fd79a4fd640 -1 osd.1 101 handle_osd_map injecting map crc failure 2024-09-21T18:09:24.071 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2842677 2024-09-21T18:09:24.096 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:24.091+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:24.164 INFO:tasks.rados.rados.0.smithi160.stdout:2653: got (16) Device or resource busy 2024-09-21T18:09:24.165 INFO:tasks.rados.rados.0.smithi160.stdout:2656: finishing copy_from racing read to smithi16030472-397 2024-09-21T18:09:24.165 INFO:tasks.rados.rados.0.smithi160.stdout:2645: expect (ObjNum 532 snap 0 seq_num 532) 2024-09-21T18:09:24.188 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.188 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.196 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:24.191+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:24.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: finishing write tid 2 to smithi16030472-395 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: finishing write tid 3 to smithi16030472-395 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: finishing write tid 4 to smithi16030472-395 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: oid 395 updating version 0 to 812 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: oid 395 updating version 812 to 844 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:2638: oid 395 version 844 is already newer than 843 2024-09-21T18:09:24.292 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 395 v 844 (ObjNum 1202 snap 0 seq_num 1202) dirty exists 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2638: left oid 395 (ObjNum 1202 snap 0 seq_num 1202) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2651: finishing write tid 1 to smithi16030472-350 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2656: finishing copy_from to smithi16030472-397 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 397 v 4 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2651: finishing write tid 2 to smithi16030472-350 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2655: got (0) Success 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 159 v 0 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2657: got (16) Device or resource busy 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2638: done (15 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2644: done (14 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2645: done (13 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2648: done (12 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2653: done (11 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2655: done (10 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2656: done (9 left) 2024-09-21T18:09:24.293 INFO:tasks.rados.rados.0.smithi160.stdout:2657: done (8 left) 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2658: write oid 413 current snap is 0 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: finishing write tid 3 to smithi16030472-350 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: finishing write tid 1 to smithi16030472-481 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: finishing write tid 4 to smithi16030472-350 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: oid 350 updating version 0 to 845 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: oid 350 updating version 845 to 846 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: oid 350 updating version 846 to 847 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 847 (ObjNum 1207 snap 0 seq_num 1207) dirty exists 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2651: left oid 350 (ObjNum 1207 snap 0 seq_num 1207) 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: finishing write tid 2 to smithi16030472-481 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: finishing write tid 3 to smithi16030472-481 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: finishing write tid 4 to smithi16030472-481 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: oid 481 updating version 0 to 814 2024-09-21T18:09:24.294 INFO:tasks.rados.rados.0.smithi160.stdout:2646: oid 481 updating version 814 to 815 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:2646: oid 481 version 815 is already newer than 813 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 815 (ObjNum 1205 snap 0 seq_num 1205) dirty exists 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:2646: left oid 481 (ObjNum 1205 snap 0 seq_num 1205) 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:2654: finishing copy_from racing read to smithi16030472-272 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:2650: got (0) Success 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 0 (ObjNum 814 snap 0 seq_num 814) dirty exists 2024-09-21T18:09:24.295 INFO:tasks.rados.rados.0.smithi160.stdout:2658: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-21T18:09:24.297 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:24.293+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:24.302 INFO:tasks.rados.rados.0.smithi160.stdout:2658: writing smithi16030472-413 from 719663 to 1395831 tid 1 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2658: writing smithi16030472-413 from 1964377 to 2319663 tid 2 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2646: done (8 left) 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2650: done (7 left) 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2651: done (6 left) 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2659: cache_try_flush oid 448 snap -1 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2654: finishing copy_from to smithi16030472-272 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 272 v 337 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2654: done (6 left) 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2660: write (excl) oid 111 current snap is 0 2024-09-21T18:09:24.310 INFO:tasks.rados.rados.0.smithi160.stdout:2660: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-21T18:09:24.326 INFO:tasks.rados.rados.0.smithi160.stdout:2660: writing smithi16030472-111 from 736470 to 1487887 tid 1 2024-09-21T18:09:24.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2660: writing smithi16030472-111 from 2022859 to 2336470 tid 2 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2661: copy_from oid 88 from oid 124 current snap is 0 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2662: cache_evict oid 481 snap -1 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2663: copy_from oid 173 from oid 463 current snap is 0 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2664: copy_from oid 396 from oid 74 current snap is 0 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2665: read oid 444 snap -1 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2665: expect deleted 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2666: cache_flush oid 5 snap -1 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2667: read oid 79 snap -1 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2667: expect (ObjNum 785 snap 0 seq_num 785) 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2668: cache_try_flush oid 426 snap -1 2024-09-21T18:09:24.337 INFO:tasks.rados.rados.0.smithi160.stdout:2669: cache_flush oid 72 snap -1 2024-09-21T18:09:24.338 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:24.339 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.340 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:24.398 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:24.396+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:24.495 INFO:tasks.rados.rados.0.smithi160.stdout:2660: finishing write tid 1 to smithi16030472-111 2024-09-21T18:09:24.495 INFO:tasks.rados.rados.0.smithi160.stdout:2660: finishing write tid 2 to smithi16030472-111 2024-09-21T18:09:24.498 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:24.498+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: finishing write tid 1 to smithi16030472-117 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: finishing write tid 2 to smithi16030472-117 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: finishing write tid 3 to smithi16030472-117 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: finishing write tid 4 to smithi16030472-117 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: oid 117 updating version 0 to 817 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: oid 117 updating version 817 to 819 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:2647: oid 117 version 819 is already newer than 818 2024-09-21T18:09:24.513 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 117 v 819 (ObjNum 1206 snap 0 seq_num 1206) dirty exists 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2647: left oid 117 (ObjNum 1206 snap 0 seq_num 1206) 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2649: got (0) Success 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2652: finishing write tid 1 to smithi16030472-24 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2652: finishing write tid 2 to smithi16030472-24 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2652: finishing write tid 3 to smithi16030472-24 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2658: finishing write tid 1 to smithi16030472-413 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2647: done (15 left) 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2649: done (14 left) 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2670: copy_from oid 354 from oid 49 current snap is 0 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2659: got (0) Success 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 0 (ObjNum 777 snap 0 seq_num 777) dirty exists 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2662: got (0) Success 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2652: finishing write tid 4 to smithi16030472-24 2024-09-21T18:09:24.514 INFO:tasks.rados.rados.0.smithi160.stdout:2652: oid 24 updating version 0 to 820 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2652: oid 24 updating version 820 to 821 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2652: oid 24 updating version 821 to 822 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 24 v 822 (ObjNum 1208 snap 0 seq_num 1208) dirty exists 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2652: left oid 24 (ObjNum 1208 snap 0 seq_num 1208) 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2643: finishing copy_from to smithi16030472-56 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 669 (ObjNum 939 snap 0 seq_num 939) dirty exists 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2668: got (0) Success 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 426 v 0 (ObjNum 425 snap 0 seq_num 425) dirty exists 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2652: done (14 left) 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2659: done (13 left) 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2662: done (12 left) 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2668: done (11 left) 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2671: write (excl) oid 343 current snap is 0 2024-09-21T18:09:24.515 INFO:tasks.rados.rados.0.smithi160.stdout:2671: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-09-21T18:09:24.523 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.523 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.525 INFO:tasks.rados.rados.0.smithi160.stdout:2671: writing smithi16030472-343 from 753277 to 1179943 tid 1 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2671: writing smithi16030472-343 from 1681341 to 2227094 tid 2 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2672: read oid 175 snap -1 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2672: expect deleted 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2673: cache_flush oid 110 snap -1 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2674: write oid 46 current snap is 0 2024-09-21T18:09:24.536 INFO:tasks.rados.rados.0.smithi160.stdout:2674: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-21T18:09:24.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:24.549 INFO:tasks.rados.rados.0.smithi160.stdout:2674: writing smithi16030472-46 from 770084 to 1271999 tid 1 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2674: writing smithi16030472-46 from 2023470 to 2370084 tid 2 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2675: cache_flush oid 4 snap -1 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2673: got (0) Success 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 110 v 0 (ObjNum 109 snap 0 seq_num 109) dirty exists 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2673: done (15 left) 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2676: read oid 174 snap -1 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout:2676: expect (ObjNum 1121 snap 0 seq_num 1121) 2024-09-21T18:09:24.561 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:24.596 INFO:tasks.rados.rados.0.smithi160.stdout:2660: finishing write tid 3 to smithi16030472-111 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: finishing write tid 4 to smithi16030472-111 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: oid 111 updating version 0 to 864 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: oid 111 updating version 864 to 865 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: oid 111 version 865 is already newer than 863 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 111 v 865 (ObjNum 1210 snap 0 seq_num 1210) dirty exists 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: left oid 111 (ObjNum 1210 snap 0 seq_num 1210) 2024-09-21T18:09:24.597 INFO:tasks.rados.rados.0.smithi160.stdout:2660: done (15 left) 2024-09-21T18:09:24.598 INFO:tasks.rados.rados.0.smithi160.stdout:2677: cache_try_flush oid 91 snap -1 2024-09-21T18:09:24.598 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:24.600 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:24.595+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:24.651 INFO:tasks.rados.rados.0.smithi160.stdout:2672: done (15 left) 2024-09-21T18:09:24.652 INFO:tasks.rados.rados.0.smithi160.stdout:2678: cache_evict oid 486 snap -1 2024-09-21T18:09:24.652 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:24.661 INFO:tasks.rados.rados.0.smithi160.stdout:2658: finishing write tid 2 to smithi16030472-413 2024-09-21T18:09:24.662 INFO:tasks.rados.rados.0.smithi160.stdout:2665: done (15 left) 2024-09-21T18:09:24.662 INFO:tasks.rados.rados.0.smithi160.stdout:2679: cache_evict oid 296 snap -1 2024-09-21T18:09:24.662 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:24.665 INFO:tasks.rados.rados.0.smithi160.stdout:2667: expect (ObjNum 785 snap 0 seq_num 785) 2024-09-21T18:09:24.682 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 825093 2024-09-21T18:09:24.700 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.700 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.701 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:24.695+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:24.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:24.801 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:24.798+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:24.804 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 833574 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:2661: finishing copy_from racing read to smithi16030472-88 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:2643: finishing copy_from racing read to smithi16030472-56 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:2669: got (2) No such file or directory 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:2661: finishing copy_from to smithi16030472-88 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 665 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:09:24.839 INFO:tasks.rados.rados.0.smithi160.stdout:2676: expect (ObjNum 1121 snap 0 seq_num 1121) 2024-09-21T18:09:24.842 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:24.843 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:24.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:24.901 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2786681 2024-09-21T18:09:24.902 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:24.899+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:24.913 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 887102 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2678: got (2) No such file or directory 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: finishing write tid 3 to smithi16030472-413 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: finishing write tid 4 to smithi16030472-413 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: oid 413 updating version 0 to 850 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: oid 413 version 850 is already newer than 849 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: oid 413 version 850 is already newer than 848 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 850 (ObjNum 1209 snap 0 seq_num 1209) dirty exists 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2658: left oid 413 (ObjNum 1209 snap 0 seq_num 1209) 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2664: finishing copy_from racing read to smithi16030472-396 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2671: finishing write tid 1 to smithi16030472-343 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2671: finishing write tid 2 to smithi16030472-343 2024-09-21T18:09:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:2671: finishing write tid 3 to smithi16030472-343 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: finishing write tid 4 to smithi16030472-343 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: oid 343 updating version 0 to 845 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: oid 343 version 845 is already newer than 844 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: oid 343 updating version 845 to 846 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 846 (ObjNum 1211 snap 0 seq_num 1211) dirty exists 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: left oid 343 (ObjNum 1211 snap 0 seq_num 1211) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2670: finishing copy_from racing read to smithi16030472-354 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2643: done (15 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2658: done (14 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2661: done (13 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2667: done (12 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2669: done (11 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2671: done (10 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2676: done (9 left) 2024-09-21T18:09:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:2678: done (8 left) 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2680: copy_from oid 469 from oid 486 current snap is 0 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: finishing write tid 1 to smithi16030472-46 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: finishing write tid 2 to smithi16030472-46 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: finishing write tid 3 to smithi16030472-46 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: finishing write tid 4 to smithi16030472-46 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: oid 46 updating version 0 to 849 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: oid 46 version 849 is already newer than 848 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: oid 46 version 849 is already newer than 847 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 849 (ObjNum 1212 snap 0 seq_num 1212) dirty exists 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2674: left oid 46 (ObjNum 1212 snap 0 seq_num 1212) 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2679: got (0) Success 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2666: got (0) Success 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 5 v 0 (ObjNum 1198 snap 0 seq_num 1198) dirty exists 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2663: finishing copy_from racing read to smithi16030472-173 2024-09-21T18:09:24.950 INFO:tasks.rados.rados.0.smithi160.stdout:2675: got (0) Success 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 4 v 0 (ObjNum 805 snap 0 seq_num 805) dirty exists 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2666: done (8 left) 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2674: done (7 left) 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2675: done (6 left) 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2679: done (5 left) 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2681: write (excl) oid 385 current snap is 0 2024-09-21T18:09:24.951 INFO:tasks.rados.rados.0.smithi160.stdout:2681: seq_num 1213 ranges {786891=577164,2081952=304939} 2024-09-21T18:09:24.969 INFO:tasks.rados.rados.0.smithi160.stdout:2681: writing smithi16030472-385 from 786891 to 1364055 tid 1 2024-09-21T18:09:24.986 INFO:tasks.rados.rados.0.smithi160.stdout:2681: writing smithi16030472-385 from 2081952 to 2386891 tid 2 2024-09-21T18:09:24.986 INFO:tasks.rados.rados.0.smithi160.stdout:2682: read oid 131 snap -1 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2682: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2683: copy_from oid 184 from oid 265 current snap is 0 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2664: finishing copy_from to smithi16030472-396 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 396 v 367 (ObjNum 575 snap 0 seq_num 575) dirty exists 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2670: finishing copy_from to smithi16030472-354 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 354 v 793 (ObjNum 1100 snap 0 seq_num 1100) dirty exists 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2677: got (0) Success 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 0 (ObjNum 956 snap 0 seq_num 956) dirty exists 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2664: done (7 left) 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2670: done (6 left) 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2677: done (5 left) 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2684: write oid 309 current snap is 0 2024-09-21T18:09:24.987 INFO:tasks.rados.rados.0.smithi160.stdout:2684: seq_num 1214 ranges {403698=652413,1740434=663264} 2024-09-21T18:09:25.003 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:24.999+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:25.003 INFO:tasks.rados.rados.0.smithi160.stdout:2684: writing smithi16030472-309 from 403698 to 1056111 tid 1 2024-09-21T18:09:25.019 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.019 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.023 INFO:tasks.rados.rados.0.smithi160.stdout:2684: writing smithi16030472-309 from 1740434 to 2403698 tid 2 2024-09-21T18:09:25.023 INFO:tasks.rados.rados.0.smithi160.stdout:2685: read oid 491 snap -1 2024-09-21T18:09:25.023 INFO:tasks.rados.rados.0.smithi160.stdout:2685: expect (ObjNum 935 snap 0 seq_num 935) 2024-09-21T18:09:25.023 INFO:tasks.rados.rados.0.smithi160.stdout:2686: delete oid 261 current snap is 0 2024-09-21T18:09:25.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:25.105 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:25.099+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:25.136 INFO:tasks.rados.rados.0.smithi160.stdout:2684: finishing write tid 1 to smithi16030472-309 2024-09-21T18:09:25.136 INFO:tasks.rados.rados.0.smithi160.stdout:2684: finishing write tid 2 to smithi16030472-309 2024-09-21T18:09:25.165 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.165 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:25.190 INFO:tasks.rados.rados.0.smithi160.stdout:2685: expect (ObjNum 935 snap 0 seq_num 935) 2024-09-21T18:09:25.205 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:25.197+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:25.237 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1984116 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2681: finishing write tid 1 to smithi16030472-385 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2663: finishing copy_from to smithi16030472-173 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 173 v 704 (ObjNum 1093 snap 0 seq_num 1093) dirty exists 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2681: finishing write tid 2 to smithi16030472-385 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2681: finishing write tid 3 to smithi16030472-385 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2683: finishing copy_from racing read to smithi16030472-184 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2680: finishing copy_from to smithi16030472-469 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 469 v 370 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:09:25.300 INFO:tasks.rados.rados.0.smithi160.stdout:2684: finishing write tid 3 to smithi16030472-309 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: finishing write tid 4 to smithi16030472-309 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: oid 309 updating version 0 to 852 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: oid 309 version 852 is already newer than 850 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: oid 309 version 852 is already newer than 851 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 309 v 852 (ObjNum 1214 snap 0 seq_num 1214) dirty exists 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: left oid 309 (ObjNum 1214 snap 0 seq_num 1214) 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: finishing write tid 4 to smithi16030472-385 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: oid 385 updating version 0 to 851 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: oid 385 updating version 851 to 853 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: oid 385 version 853 is already newer than 852 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 853 (ObjNum 1213 snap 0 seq_num 1213) dirty exists 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: left oid 385 (ObjNum 1213 snap 0 seq_num 1213) 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2663: done (7 left) 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2681: done (6 left) 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2684: done (5 left) 2024-09-21T18:09:25.301 INFO:tasks.rados.rados.0.smithi160.stdout:2685: done (4 left) 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2686: done (3 left) 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2687: cache_flush oid 318 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2688: cache_flush oid 7 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2689: cache_evict oid 279 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2690: cache_evict oid 150 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2691: cache_flush oid 321 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2692: cache_try_flush oid 483 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2693: cache_flush oid 139 snap -1 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2694: write oid 180 current snap is 0 2024-09-21T18:09:25.302 INFO:tasks.rados.rados.0.smithi160.stdout:2694: seq_num 1216 ranges {437312=402911,1341045=696749} 2024-09-21T18:09:25.305 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:25.303+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:25.313 INFO:tasks.rados.rados.0.smithi160.stdout:2694: writing smithi16030472-180 from 437312 to 840223 tid 1 2024-09-21T18:09:25.320 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.320 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:09:25.331 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:25.330+0000 7ff40b999640 -1 osd.4 102 handle_osd_map injecting map crc failure 2024-09-21T18:09:25.333 INFO:tasks.rados.rados.0.smithi160.stdout:2694: writing smithi16030472-180 from 1341045 to 2037794 tid 2 2024-09-21T18:09:25.333 INFO:tasks.rados.rados.0.smithi160.stdout:2695: write oid 84 current snap is 0 2024-09-21T18:09:25.333 INFO:tasks.rados.rados.0.smithi160.stdout:2683: finishing copy_from to smithi16030472-184 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 184 v 496 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:2692: got (2) No such file or directory 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:2693: got (0) Success 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 0 (ObjNum 978 snap 0 seq_num 978) dirty exists 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:2691: got (0) Success 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 321 v 0 (ObjNum 129 snap 0 seq_num 3128801368) clean dne 2024-09-21T18:09:25.334 INFO:tasks.rados.rados.0.smithi160.stdout:2695: seq_num 1217 ranges {454119=594513,1515880=556760} 2024-09-21T18:09:25.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:25.351 INFO:tasks.rados.rados.0.smithi160.stdout:2695: writing smithi16030472-84 from 454119 to 1048632 tid 1 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2695: writing smithi16030472-84 from 1515880 to 2072640 tid 2 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2683: done (11 left) 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2691: done (10 left) 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2692: done (9 left) 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2693: done (8 left) 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2696: write oid 187 current snap is 0 2024-09-21T18:09:25.367 INFO:tasks.rados.rados.0.smithi160.stdout:2696: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-21T18:09:25.381 INFO:tasks.rados.rados.0.smithi160.stdout:2696: writing smithi16030472-187 from 470926 to 1140688 tid 1 2024-09-21T18:09:25.395 INFO:tasks.rados.rados.0.smithi160.stdout:2696: writing smithi16030472-187 from 1574362 to 2274780 tid 2 2024-09-21T18:09:25.395 INFO:tasks.rados.rados.0.smithi160.stdout:2697: cache_flush oid 430 snap -1 2024-09-21T18:09:25.395 INFO:tasks.rados.rados.0.smithi160.stdout:2682: expect (ObjNum 130 snap 0 seq_num 130) 2024-09-21T18:09:25.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:25.406 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:25.401+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:25.454 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:25.462 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.462 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2688: got (2) No such file or directory 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2690: got (16) Device or resource busy 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2680: finishing copy_from racing read to smithi16030472-469 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2680: done (9 left) 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2682: done (8 left) 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2688: done (7 left) 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2690: done (6 left) 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2698: write (excl) oid 369 current snap is 0 2024-09-21T18:09:25.481 INFO:tasks.rados.rados.0.smithi160.stdout:2698: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-21T18:09:25.493 INFO:tasks.rados.rados.0.smithi160.stdout:2698: writing smithi16030472-369 from 487733 to 1232744 tid 1 2024-09-21T18:09:25.503 INFO:tasks.rados.rados.0.smithi160.stdout:2698: writing smithi16030472-369 from 1632844 to 2193273 tid 2 2024-09-21T18:09:25.503 INFO:tasks.rados.rados.0.smithi160.stdout:2699: write (excl) oid 493 current snap is 0 2024-09-21T18:09:25.503 INFO:tasks.rados.rados.0.smithi160.stdout:2699: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-21T18:09:25.507 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:25.503+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:25.513 INFO:tasks.rados.rados.0.smithi160.stdout:2699: writing smithi16030472-493 from 504540 to 924800 tid 1 2024-09-21T18:09:25.527 INFO:tasks.rados.rados.0.smithi160.stdout:2699: writing smithi16030472-493 from 1574973 to 2279060 tid 2 2024-09-21T18:09:25.527 INFO:tasks.rados.rados.0.smithi160.stdout:2700: delete oid 337 current snap is 0 2024-09-21T18:09:25.527 INFO:tasks.rados.rados.0.smithi160.stdout:2687: got (0) Success 2024-09-21T18:09:25.527 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 318 v 0 (ObjNum 431 snap 0 seq_num 431) dirty exists 2024-09-21T18:09:25.552 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:25.552 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:25.561 INFO:tasks.rados.rados.0.smithi160.stdout:2695: finishing write tid 1 to smithi16030472-84 2024-09-21T18:09:25.573 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:25.594 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:25.600 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.600 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.608 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:25.604+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:25.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:25.615 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:25.661 INFO:tasks.rados.rados.0.smithi160.stdout:2689: got (0) Success 2024-09-21T18:09:25.662 INFO:tasks.rados.rados.0.smithi160.stdout:2694: finishing write tid 1 to smithi16030472-180 2024-09-21T18:09:25.662 INFO:tasks.rados.rados.0.smithi160.stdout:2694: finishing write tid 2 to smithi16030472-180 2024-09-21T18:09:25.662 INFO:tasks.rados.rados.0.smithi160.stdout:2694: finishing write tid 3 to smithi16030472-180 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2694: finishing write tid 4 to smithi16030472-180 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2694: oid 180 updating version 0 to 866 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2694: oid 180 updating version 866 to 867 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2694: oid 180 updating version 867 to 868 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 868 (ObjNum 1216 snap 0 seq_num 1216) dirty exists 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2694: left oid 180 (ObjNum 1216 snap 0 seq_num 1216) 2024-09-21T18:09:25.663 INFO:tasks.rados.rados.0.smithi160.stdout:2696: finishing write tid 1 to smithi16030472-187 2024-09-21T18:09:25.671 INFO:tasks.rados.rados.0.smithi160.stdout:2696: finishing write tid 2 to smithi16030472-187 2024-09-21T18:09:25.674 INFO:tasks.rados.rados.0.smithi160.stdout:2696: finishing write tid 3 to smithi16030472-187 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:2696: finishing write tid 4 to smithi16030472-187 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:2696: oid 187 updating version 0 to 871 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:2696: oid 187 version 871 is already newer than 870 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:2696: oid 187 version 871 is already newer than 869 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 871 (ObjNum 1218 snap 0 seq_num 1218) dirty exists 2024-09-21T18:09:25.675 INFO:tasks.rados.rados.0.smithi160.stdout:2696: left oid 187 (ObjNum 1218 snap 0 seq_num 1218) 2024-09-21T18:09:25.693 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:25.709 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:25.707+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:25.715 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:25.715 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:25.736 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:25.737 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:25.750 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:25.750 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:25.756 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:25.756 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:25.761 INFO:tasks.rados.rados.0.smithi160.stdout:2695: finishing write tid 2 to smithi16030472-84 2024-09-21T18:09:25.761 INFO:tasks.rados.rados.0.smithi160.stdout:2695: finishing write tid 3 to smithi16030472-84 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:2695: finishing write tid 4 to smithi16030472-84 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:2695: oid 84 updating version 0 to 854 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:2695: oid 84 updating version 854 to 855 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:2695: oid 84 version 855 is already newer than 853 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 84 v 855 (ObjNum 1217 snap 0 seq_num 1217) dirty exists 2024-09-21T18:09:25.762 INFO:tasks.rados.rados.0.smithi160.stdout:2695: left oid 84 (ObjNum 1217 snap 0 seq_num 1217) 2024-09-21T18:09:25.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:25.777 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:25.780 INFO:tasks.rados.rados.0.smithi160.stdout:2698: finishing write tid 1 to smithi16030472-369 2024-09-21T18:09:25.780 INFO:tasks.rados.rados.0.smithi160.stdout:2698: finishing write tid 2 to smithi16030472-369 2024-09-21T18:09:25.799 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:25.810 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:25.807+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:25.820 INFO:tasks.rados.rados.0.smithi160.stdout:2697: got (0) Success 2024-09-21T18:09:25.820 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 430 v 0 (ObjNum 989 snap 0 seq_num 989) dirty exists 2024-09-21T18:09:25.821 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:25.821 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:25.838 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:25.838 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:25.859 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:25.862 INFO:tasks.rados.rados.0.smithi160.stdout:2687: done (8 left) 2024-09-21T18:09:25.862 INFO:tasks.rados.rados.0.smithi160.stdout:2689: done (7 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2694: done (6 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2695: done (5 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2696: done (4 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2697: done (3 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2700: done (2 left) 2024-09-21T18:09:25.863 INFO:tasks.rados.rados.0.smithi160.stdout:2701: delete oid 318 current snap is 0 2024-09-21T18:09:25.880 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:25.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:25.880 INFO:tasks.rados.rados.0.smithi160.stdout:2698: finishing write tid 3 to smithi16030472-369 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:2698: finishing write tid 4 to smithi16030472-369 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:2698: oid 369 updating version 0 to 872 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:2698: oid 369 updating version 872 to 873 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:2698: oid 369 updating version 873 to 874 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 369 v 874 (ObjNum 1219 snap 0 seq_num 1219) dirty exists 2024-09-21T18:09:25.881 INFO:tasks.rados.rados.0.smithi160.stdout:2698: left oid 369 (ObjNum 1219 snap 0 seq_num 1219) 2024-09-21T18:09:25.882 INFO:tasks.rados.rados.0.smithi160.stdout:2699: finishing write tid 1 to smithi16030472-493 2024-09-21T18:09:25.882 INFO:tasks.rados.rados.0.smithi160.stdout:2699: finishing write tid 2 to smithi16030472-493 2024-09-21T18:09:25.882 INFO:tasks.rados.rados.0.smithi160.stdout:2699: finishing write tid 3 to smithi16030472-493 2024-09-21T18:09:25.882 INFO:tasks.rados.rados.0.smithi160.stdout:2699: finishing write tid 4 to smithi16030472-493 2024-09-21T18:09:25.883 INFO:tasks.rados.rados.0.smithi160.stdout:2699: oid 493 updating version 0 to 876 2024-09-21T18:09:25.883 INFO:tasks.rados.rados.0.smithi160.stdout:2699: oid 493 version 876 is already newer than 875 2024-09-21T18:09:25.883 INFO:tasks.rados.rados.0.smithi160.stdout:2699: oid 493 updating version 876 to 877 2024-09-21T18:09:25.883 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 493 v 877 (ObjNum 1220 snap 0 seq_num 1220) dirty exists 2024-09-21T18:09:25.883 INFO:tasks.rados.rados.0.smithi160.stdout:2699: left oid 493 (ObjNum 1220 snap 0 seq_num 1220) 2024-09-21T18:09:25.883 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:25.883 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:25.884 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:25.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:25.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:25.906 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:25.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:25.912 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:25.905+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:26.013 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:26.005+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 160.127561899, 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:26.025 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:09:26.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:09:26.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:09:26.029 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 159.04216951000001, 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:26.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:26.033 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:26.034 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:26.034 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:26.034 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:26.040 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:26.067 INFO:tasks.rados.rados.0.smithi160.stdout:2698: done (2 left) 2024-09-21T18:09:26.067 INFO:tasks.rados.rados.0.smithi160.stdout:2699: done (1 left) 2024-09-21T18:09:26.067 INFO:tasks.rados.rados.0.smithi160.stdout:2701: done (0 left) 2024-09-21T18:09:26.067 INFO:tasks.rados.rados.0.smithi160.stdout:2702: delete oid 140 current snap is 0 2024-09-21T18:09:26.113 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:26.108+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:26.170 INFO:tasks.rados.rados.0.smithi160.stdout:2702: done (0 left) 2024-09-21T18:09:26.170 INFO:tasks.rados.rados.0.smithi160.stdout:2703: write oid 63 current snap is 0 2024-09-21T18:09:26.171 INFO:tasks.rados.rados.0.smithi160.stdout:2703: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-21T18:09:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:26.176 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:26.176 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:26.187 INFO:tasks.rados.rados.0.smithi160.stdout:2703: writing smithi16030472-63 from 571768 to 1009377 tid 1 2024-09-21T18:09:26.190 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:26.211 INFO:tasks.rados.rados.0.smithi160.stdout:2703: writing smithi16030472-63 from 1808901 to 2520326 tid 2 2024-09-21T18:09:26.212 INFO:tasks.rados.rados.0.smithi160.stdout:2704: write (excl) oid 132 current snap is 0 2024-09-21T18:09:26.212 INFO:tasks.rados.rados.0.smithi160.stdout:2704: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-21T18:09:26.214 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:26.209+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:26.227 INFO:tasks.rados.rados.0.smithi160.stdout:2704: writing smithi16030472-132 from 588575 to 1101433 tid 1 2024-09-21T18:09:26.241 INFO:tasks.rados.rados.0.smithi160.stdout:2704: writing smithi16030472-132 from 1867383 to 2322466 tid 2 2024-09-21T18:09:26.241 INFO:tasks.rados.rados.0.smithi160.stdout:2705: write (excl) oid 364 current snap is 0 2024-09-21T18:09:26.241 INFO:tasks.rados.rados.0.smithi160.stdout:2705: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-21T18:09:26.251 INFO:tasks.rados.rados.0.smithi160.stdout:2705: writing smithi16030472-364 from 605382 to 1193489 tid 1 2024-09-21T18:09:26.263 INFO:tasks.rados.rados.0.smithi160.stdout:2705: writing smithi16030472-364 from 1925865 to 2605382 tid 2 2024-09-21T18:09:26.263 INFO:tasks.rados.rados.0.smithi160.stdout:2706: cache_evict oid 475 snap -1 2024-09-21T18:09:26.263 INFO:tasks.rados.rados.0.smithi160.stdout:2705: finishing write tid 1 to smithi16030472-364 2024-09-21T18:09:26.263 INFO:tasks.rados.rados.0.smithi160.stdout:2707: write oid 243 current snap is 0 2024-09-21T18:09:26.263 INFO:tasks.rados.rados.0.smithi160.stdout:2707: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-21T18:09:26.272 INFO:tasks.rados.rados.0.smithi160.stdout:2707: writing smithi16030472-243 from 622189 to 1285545 tid 1 2024-09-21T18:09:26.280 INFO:tasks.rados.rados.0.smithi160.stdout:2707: writing smithi16030472-243 from 1984347 to 2443099 tid 2 2024-09-21T18:09:26.280 INFO:tasks.rados.rados.0.smithi160.stdout:2708: read oid 464 snap -1 2024-09-21T18:09:26.281 INFO:tasks.rados.rados.0.smithi160.stdout:2708: expect (ObjNum 1168 snap 0 seq_num 1168) 2024-09-21T18:09:26.281 INFO:tasks.rados.rados.0.smithi160.stdout:2709: cache_evict oid 49 snap -1 2024-09-21T18:09:26.281 INFO:tasks.rados.rados.0.smithi160.stdout:2710: write (excl) oid 102 current snap is 0 2024-09-21T18:09:26.281 INFO:tasks.rados.rados.0.smithi160.stdout:2710: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-21T18:09:26.290 INFO:tasks.rados.rados.0.smithi160.stdout:2710: writing smithi16030472-102 from 638996 to 1377601 tid 1 2024-09-21T18:09:26.299 INFO:tasks.rados.rados.0.smithi160.stdout:2710: writing smithi16030472-102 from 1926476 to 2638996 tid 2 2024-09-21T18:09:26.299 INFO:tasks.rados.rados.0.smithi160.stdout:2711: copy_from oid 159 from oid 266 current snap is 0 2024-09-21T18:09:26.299 INFO:tasks.rados.rados.0.smithi160.stdout:2712: write oid 183 current snap is 0 2024-09-21T18:09:26.299 INFO:tasks.rados.rados.0.smithi160.stdout:2712: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-21T18:09:26.306 INFO:tasks.rados.rados.0.smithi160.stdout:2712: writing smithi16030472-183 from 655803 to 1069657 tid 1 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2712: writing smithi16030472-183 from 1584958 to 2047379 tid 2 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2713: cache_evict oid 238 snap -1 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2714: read oid 64 snap -1 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2714: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2715: copy_from oid 226 from oid 206 current snap is 0 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2705: finishing write tid 2 to smithi16030472-364 2024-09-21T18:09:26.313 INFO:tasks.rados.rados.0.smithi160.stdout:2716: write oid 297 current snap is 0 2024-09-21T18:09:26.314 INFO:tasks.rados.rados.0.smithi160.stdout:2716: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-21T18:09:26.315 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:26.314+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:26.320 INFO:tasks.rados.rados.0.smithi160.stdout:2716: writing smithi16030472-297 from 672610 to 1161713 tid 1 2024-09-21T18:09:26.328 INFO:tasks.rados.rados.0.smithi160.stdout:2716: writing smithi16030472-297 from 1643440 to 2365872 tid 2 2024-09-21T18:09:26.328 INFO:tasks.rados.rados.0.smithi160.stdout:2717: delete oid 246 current snap is 0 2024-09-21T18:09:26.328 INFO:tasks.rados.rados.0.smithi160.stdout:2705: finishing write tid 3 to smithi16030472-364 2024-09-21T18:09:26.366 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:26.366 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:26.367 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:26.380 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:26.381 INFO:tasks.rados.rados.0.smithi160.stdout:2705: finishing write tid 4 to smithi16030472-364 2024-09-21T18:09:26.381 INFO:tasks.rados.rados.0.smithi160.stdout:2705: oid 364 updating version 0 to 878 2024-09-21T18:09:26.382 INFO:tasks.rados.rados.0.smithi160.stdout:2705: oid 364 updating version 878 to 879 2024-09-21T18:09:26.382 INFO:tasks.rados.rados.0.smithi160.stdout:2705: oid 364 updating version 879 to 880 2024-09-21T18:09:26.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 880 (ObjNum 1226 snap 0 seq_num 1226) dirty exists 2024-09-21T18:09:26.382 INFO:tasks.rados.rados.0.smithi160.stdout:2705: left oid 364 (ObjNum 1226 snap 0 seq_num 1226) 2024-09-21T18:09:26.382 INFO:tasks.rados.rados.0.smithi160.stdout:2707: finishing write tid 1 to smithi16030472-243 2024-09-21T18:09:26.383 INFO:tasks.rados.rados.0.smithi160.stdout:2709: got (16) Device or resource busy 2024-09-21T18:09:26.383 INFO:tasks.rados.rados.0.smithi160.stdout:2707: finishing write tid 2 to smithi16030472-243 2024-09-21T18:09:26.383 INFO:tasks.rados.rados.0.smithi160.stdout:2707: finishing write tid 3 to smithi16030472-243 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:2707: finishing write tid 4 to smithi16030472-243 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:2707: oid 243 updating version 0 to 883 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:2707: oid 243 version 883 is already newer than 881 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:2707: oid 243 version 883 is already newer than 882 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 883 (ObjNum 1227 snap 0 seq_num 1227) dirty exists 2024-09-21T18:09:26.384 INFO:tasks.rados.rados.0.smithi160.stdout:2707: left oid 243 (ObjNum 1227 snap 0 seq_num 1227) 2024-09-21T18:09:26.388 INFO:tasks.rados.rados.0.smithi160.stdout:2713: got (0) Success 2024-09-21T18:09:26.415 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:26.409+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:26.446 INFO:tasks.rados.rados.0.smithi160.stdout:2706: got (2) No such file or directory 2024-09-21T18:09:26.462 INFO:tasks.rados.rados.0.smithi160.stdout:2708: expect (ObjNum 1168 snap 0 seq_num 1168) 2024-09-21T18:09:26.462 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:26.510 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:26.510 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.46868453800001, 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:26.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:09:26.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:09:26.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:09:26.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.516 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.468444914, 2024-09-21T18:09:26.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:09:26.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:09:26.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:09:26.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.523 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.468409507, 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:26.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:09:26.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:09:26.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.528 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.468376334, 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:09:26.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:09:26.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.468346774, 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:09:26.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:09:26.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:09:26.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.468317118, 2024-09-21T18:09:26.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:09:26.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:09:26.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:09:26.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:09:26.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.467830862, 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.544 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:09:26.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:09:26.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:09:26.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.548 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.467797269, 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.367579577, 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:09:26.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:09:26.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:09:26.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:09:26.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.366980673, 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.366266056, 2024-09-21T18:09:26.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:09:26.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:09:26.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:09:26.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.35701332399999, 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:09:26.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.356748623, 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.578 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:09:26.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:09:26.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:09:26.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.35665393399999, 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:09:26.583 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:26.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:09:26.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.35661264399999, 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.595 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.345342741, 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:26.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:09:26.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:09:26.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.336226508, 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:09:26.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:09:26.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:09:26.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:09:26.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.335163141, 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:09:26.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:09:26.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.615 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.33512283100001, 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:09:26.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 114.572484989, 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:09:26.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:26.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:26.626 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:26.627 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:26.628 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:26.551+0000 7fca4e93e640 -1 osd.2 103 handle_osd_map injecting map crc failure 2024-09-21T18:09:26.628 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:26.629 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:26.624+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:26.644 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 7856 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2703: finishing write tid 1 to smithi16030472-63 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2703: finishing write tid 2 to smithi16030472-63 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2703: finishing write tid 3 to smithi16030472-63 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2716: finishing write tid 1 to smithi16030472-297 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2716: finishing write tid 2 to smithi16030472-297 2024-09-21T18:09:26.720 INFO:tasks.rados.rados.0.smithi160.stdout:2716: finishing write tid 3 to smithi16030472-297 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2716: finishing write tid 4 to smithi16030472-297 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2716: oid 297 updating version 0 to 856 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2716: oid 297 version 856 is already newer than 855 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2716: oid 297 version 856 is already newer than 854 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 297 v 856 (ObjNum 1230 snap 0 seq_num 1230) dirty exists 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2716: left oid 297 (ObjNum 1230 snap 0 seq_num 1230) 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: finishing write tid 1 to smithi16030472-183 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: finishing write tid 2 to smithi16030472-183 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: finishing write tid 3 to smithi16030472-183 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: finishing write tid 4 to smithi16030472-183 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: oid 183 updating version 0 to 858 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: oid 183 updating version 858 to 859 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:2712: oid 183 updating version 859 to 860 2024-09-21T18:09:26.721 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 860 (ObjNum 1229 snap 0 seq_num 1229) dirty exists 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2712: left oid 183 (ObjNum 1229 snap 0 seq_num 1229) 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2710: finishing write tid 1 to smithi16030472-102 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: finishing write tid 4 to smithi16030472-63 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: oid 63 updating version 0 to 819 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: oid 63 version 819 is already newer than 818 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: oid 63 version 819 is already newer than 817 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 63 v 819 (ObjNum 1224 snap 0 seq_num 1224) dirty exists 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: left oid 63 (ObjNum 1224 snap 0 seq_num 1224) 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2704: finishing write tid 1 to smithi16030472-132 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2711: finishing copy_from racing read to smithi16030472-159 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2704: finishing write tid 2 to smithi16030472-132 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2703: done (14 left) 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2705: done (13 left) 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2706: done (12 left) 2024-09-21T18:09:26.722 INFO:tasks.rados.rados.0.smithi160.stdout:2707: done (11 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2708: done (10 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2709: done (9 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2712: done (8 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2713: done (7 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2716: done (6 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2717: done (5 left) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2718: read oid 362 snap -1 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2718: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2719: write (excl) oid 217 current snap is 0 2024-09-21T18:09:26.723 INFO:tasks.rados.rados.0.smithi160.stdout:2719: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-21T18:09:26.729 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:26.726+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:26.735 INFO:tasks.rados.rados.0.smithi160.stdout:2719: writing smithi16030472-217 from 706224 to 1462178 tid 1 2024-09-21T18:09:26.749 INFO:tasks.rados.rados.0.smithi160.stdout:2719: writing smithi16030472-217 from 2160404 to 2706224 tid 2 2024-09-21T18:09:26.749 INFO:tasks.rados.rados.0.smithi160.stdout:2720: cache_evict oid 314 snap -1 2024-09-21T18:09:26.749 INFO:tasks.rados.rados.0.smithi160.stdout:2721: cache_evict oid 43 snap -1 2024-09-21T18:09:26.750 INFO:tasks.rados.rados.0.smithi160.stdout:2722: write (excl) oid 455 current snap is 0 2024-09-21T18:09:26.750 INFO:tasks.rados.rados.0.smithi160.stdout:2722: seq_num 1233 ranges {723031=431203,1818886=469759} 2024-09-21T18:09:26.762 INFO:tasks.rados.rados.0.smithi160.stdout:2722: writing smithi16030472-455 from 723031 to 1154234 tid 1 2024-09-21T18:09:26.774 INFO:tasks.rados.rados.0.smithi160.stdout:2722: writing smithi16030472-455 from 1818886 to 2288645 tid 2 2024-09-21T18:09:26.774 INFO:tasks.rados.rados.0.smithi160.stdout:2723: read oid 97 snap -1 2024-09-21T18:09:26.774 INFO:tasks.rados.rados.0.smithi160.stdout:2723: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:26.774 INFO:tasks.rados.rados.0.smithi160.stdout:2724: delete oid 400 current snap is 0 2024-09-21T18:09:26.774 INFO:tasks.rados.rados.0.smithi160.stdout:2714: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:09:26.804 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:26.804 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:26.804 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:26.819 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:26.830 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:26.826+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:26.902 INFO:tasks.rados.rados.0.smithi160.stdout:2710: finishing write tid 2 to smithi16030472-102 2024-09-21T18:09:26.902 INFO:tasks.rados.rados.0.smithi160.stdout:2710: finishing write tid 3 to smithi16030472-102 2024-09-21T18:09:26.902 INFO:tasks.rados.rados.0.smithi160.stdout:2722: finishing write tid 1 to smithi16030472-455 2024-09-21T18:09:26.903 INFO:tasks.rados.rados.0.smithi160.stdout:2715: finishing copy_from to smithi16030472-226 2024-09-21T18:09:26.903 INFO:tasks.rados.rados.0.smithi160.stdout:2715: got expected ENOENT (src dne) 2024-09-21T18:09:26.903 INFO:tasks.rados.rados.0.smithi160.stdout:2715: finishing copy_from racing read to smithi16030472-226 2024-09-21T18:09:26.903 INFO:tasks.rados.rados.0.smithi160.stdout:2718: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:09:26.903 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:26.932 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:26.930+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:26.940 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:26.940 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:26.940 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:26.940 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:26.940 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:26.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:26.968 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1234705 2024-09-21T18:09:27.033 INFO:tasks.rados.rados.0.smithi160.stdout:2711: finishing copy_from to smithi16030472-159 2024-09-21T18:09:27.033 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 159 v 744 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:09:27.033 INFO:tasks.rados.rados.0.smithi160.stdout:2722: finishing write tid 2 to smithi16030472-455 2024-09-21T18:09:27.033 INFO:tasks.rados.rados.0.smithi160.stdout:2722: finishing write tid 3 to smithi16030472-455 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2722: finishing write tid 4 to smithi16030472-455 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2722: oid 455 updating version 0 to 885 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2722: oid 455 version 885 is already newer than 884 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2722: oid 455 updating version 885 to 886 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 886 (ObjNum 1233 snap 0 seq_num 1233) dirty exists 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2722: left oid 455 (ObjNum 1233 snap 0 seq_num 1233) 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2704: finishing write tid 3 to smithi16030472-132 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2710: finishing write tid 4 to smithi16030472-102 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2710: oid 102 updating version 0 to 823 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2710: oid 102 updating version 823 to 824 2024-09-21T18:09:27.034 INFO:tasks.rados.rados.0.smithi160.stdout:2710: oid 102 updating version 824 to 825 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 102 v 825 (ObjNum 1228 snap 0 seq_num 1228) dirty exists 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2710: left oid 102 (ObjNum 1228 snap 0 seq_num 1228) 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2721: got (0) Success 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2704: finishing write tid 4 to smithi16030472-132 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2704: oid 132 updating version 0 to 857 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2704: oid 132 version 857 is already newer than 820 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2704: oid 132 version 857 is already newer than 821 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 857 (ObjNum 1225 snap 0 seq_num 1225) dirty exists 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2704: left oid 132 (ObjNum 1225 snap 0 seq_num 1225) 2024-09-21T18:09:27.035 INFO:tasks.rados.rados.0.smithi160.stdout:2720: got (0) Success 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2719: finishing write tid 1 to smithi16030472-217 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2719: finishing write tid 2 to smithi16030472-217 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2719: finishing write tid 3 to smithi16030472-217 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2704: done (11 left) 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2710: done (10 left) 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2711: done (9 left) 2024-09-21T18:09:27.036 INFO:tasks.rados.rados.0.smithi160.stdout:2714: done (8 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2715: done (7 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2718: done (6 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2720: done (5 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2721: done (4 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2722: done (3 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2724: done (2 left) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2725: copy_from oid 258 from oid 177 current snap is 0 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: finishing write tid 4 to smithi16030472-217 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: oid 217 updating version 0 to 823 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: oid 217 version 823 is already newer than 822 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: oid 217 updating version 823 to 824 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 217 v 824 (ObjNum 1232 snap 0 seq_num 1232) dirty exists 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: left oid 217 (ObjNum 1232 snap 0 seq_num 1232) 2024-09-21T18:09:27.037 INFO:tasks.rados.rados.0.smithi160.stdout:2719: done (2 left) 2024-09-21T18:09:27.038 INFO:tasks.rados.rados.0.smithi160.stdout:2726: cache_try_flush oid 469 snap -1 2024-09-21T18:09:27.038 INFO:tasks.rados.rados.0.smithi160.stdout:2727: write (excl) oid 462 current snap is 0 2024-09-21T18:09:27.038 INFO:tasks.rados.rados.0.smithi160.stdout:2727: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-09-21T18:09:27.038 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:27.025+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:27.054 INFO:tasks.rados.rados.0.smithi160.stdout:2727: writing smithi16030472-462 from 756645 to 1338346 tid 1 2024-09-21T18:09:27.072 INFO:tasks.rados.rados.0.smithi160.stdout:2727: writing smithi16030472-462 from 1935850 to 2409278 tid 2 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2728: cache_evict oid 213 snap -1 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2726: got (0) Success 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 469 v 0 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2725: finishing copy_from to smithi16030472-258 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2725: got expected ENOENT (src dne) 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2727: finishing write tid 1 to smithi16030472-462 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2726: done (4 left) 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2729: cache_try_flush oid 383 snap -1 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2730: read oid 244 snap -1 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2730: expect (ObjNum 243 snap 0 seq_num 243) 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2731: write (excl) oid 54 current snap is 0 2024-09-21T18:09:27.073 INFO:tasks.rados.rados.0.smithi160.stdout:2731: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 162.77450977000001, 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:09:27.077 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 162.773295711, 2024-09-21T18:09:27.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:09:27.086 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 162.77194479100001, 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 156.48436415399999, 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:09:27.090 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.093 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 134.38577152400001, 2024-09-21T18:09:27.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:09:27.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.098 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 134.38529431399999, 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:09:27.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 116.085124914, 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.103 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:27.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:09:27.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:09:27.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.98455552199999, 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:09:27.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:09:27.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:09:27.112 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.984197455, 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:09:27.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.116 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.984160622, 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:09:27.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:09:27.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:09:27.121 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.98370088999999, 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:09:27.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:09:27.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.98326936300001, 2024-09-21T18:09:27.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.127 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.128 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.129 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:09:27.130 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.982332748, 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:27.131 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:09:27.132 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.133 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.134 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.982294065, 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.135 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:09:27.136 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.137 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.138 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.981007413, 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.139 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.140 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.141 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.142 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.87886294899999, 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.143 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.144 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.145 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.146 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:27.147 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.878139156, 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.148 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:09:27.149 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:09:27.150 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.151 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:27.152 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 107.752428917, 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.153 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:09:27.154 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.155 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.156 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:09:27.157 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:09:27.158 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 83.11263443, 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.159 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:09:27.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:09:27.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:09:27.162 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:7174 4.3 e101/33 4:fbf38165:::smithi16030472-24:head v 101'482, mlcod=100'479)", 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 4.2713045980000004, 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21150047199999999, 2024-09-21T18:09:27.163 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.164 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802260+0000", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8399999999999999e-06 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803007+0000", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00074713700000000004 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803040+0000", 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3009999999999997e-05 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.165 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803061+0000", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0676e-05 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.904316+0000", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101255181 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.904349+0000", 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3226999999999999e-05 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.166 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:23.013660+0000", 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109310465 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:23.013758+0000", 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7936000000000006e-05 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.167 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:27.170 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2731: writing smithi16030472-54 from 773452 to 1430402 tid 1 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2731: writing smithi16030472-54 from 1877979 to 2495065 tid 2 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2732: cache_evict oid 146 snap -1 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2733: copy_from oid 380 from oid 319 current snap is 0 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2734: copy_from oid 455 from oid 248 current snap is 0 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2735: cache_evict oid 397 snap -1 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2736: write (excl) oid 326 current snap is 0 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2736: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-09-21T18:09:27.171 INFO:tasks.rados.rados.0.smithi160.stdout:2736: writing smithi16030472-326 from 790259 to 1522458 tid 1 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2736: writing smithi16030472-326 from 1936461 to 2413558 tid 2 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2737: write oid 94 current snap is 0 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2731: finishing write tid 1 to smithi16030472-54 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2737: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2737: writing smithi16030472-94 from 407066 to 814514 tid 1 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2737: writing smithi16030472-94 from 1594943 to 2215698 tid 2 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2738: cache_try_flush oid 407 snap -1 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2739: cache_try_flush oid 283 snap -1 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2740: write oid 459 current snap is 0 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2740: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2740: writing smithi16030472-459 from 423873 to 906570 tid 1 2024-09-21T18:09:27.172 INFO:tasks.rados.rados.0.smithi160.stdout:2740: writing smithi16030472-459 from 1653425 to 2134191 tid 2 2024-09-21T18:09:27.173 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:27.163+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:27.174 INFO:tasks.rados.rados.0.smithi160.stdout:2740: writing smithi16030472-459 from 2691441 to 2823873 tid 3 2024-09-21T18:09:27.174 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:27.177 INFO:tasks.rados.rados.0.smithi160.stdout:2723: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:27.195 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1484082 2024-09-21T18:09:27.270 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:27.265+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:27.302 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.302 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:27.302 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:27.302 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:27.317 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:27.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:09:27.371 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:27.368+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2725: finishing copy_from racing read to smithi16030472-258 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2728: got (2) No such file or directory 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2727: finishing write tid 2 to smithi16030472-462 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2727: finishing write tid 3 to smithi16030472-462 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2727: finishing write tid 4 to smithi16030472-462 2024-09-21T18:09:27.374 INFO:tasks.rados.rados.0.smithi160.stdout:2727: oid 462 updating version 0 to 862 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2727: oid 462 updating version 862 to 864 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2727: oid 462 version 864 is already newer than 863 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 462 v 864 (ObjNum 1235 snap 0 seq_num 1235) dirty exists 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2727: left oid 462 (ObjNum 1235 snap 0 seq_num 1235) 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2732: got (16) Device or resource busy 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2739: got (0) Success 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 283 v 0 (ObjNum 129 snap 0 seq_num 3149870424) clean dne 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: finishing write tid 2 to smithi16030472-54 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: finishing write tid 3 to smithi16030472-54 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: finishing write tid 4 to smithi16030472-54 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: oid 54 updating version 0 to 889 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: oid 54 version 889 is already newer than 888 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: oid 54 version 889 is already newer than 887 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 889 (ObjNum 1236 snap 0 seq_num 1236) dirty exists 2024-09-21T18:09:27.375 INFO:tasks.rados.rados.0.smithi160.stdout:2731: left oid 54 (ObjNum 1236 snap 0 seq_num 1236) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2740: finishing write tid 1 to smithi16030472-459 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2740: finishing write tid 2 to smithi16030472-459 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2735: got (16) Device or resource busy 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2723: done (15 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2725: done (14 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2727: done (13 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2728: done (12 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2731: done (11 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2732: done (10 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2735: done (9 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2739: done (8 left) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2741: cache_try_flush oid 119 snap -1 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2742: read oid 388 snap -1 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2742: expect (ObjNum 930 snap 0 seq_num 930) 2024-09-21T18:09:27.376 INFO:tasks.rados.rados.0.smithi160.stdout:2743: write (excl) oid 77 current snap is 0 2024-09-21T18:09:27.377 INFO:tasks.rados.rados.0.smithi160.stdout:2743: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-21T18:09:27.383 INFO:tasks.rados.rados.0.smithi160.stdout:2743: writing smithi16030472-77 from 440680 to 1114979 tid 1 2024-09-21T18:09:27.394 INFO:tasks.rados.rados.0.smithi160.stdout:2743: writing smithi16030472-77 from 1711907 to 2336331 tid 2 2024-09-21T18:09:27.394 INFO:tasks.rados.rados.0.smithi160.stdout:2744: cache_try_flush oid 228 snap -1 2024-09-21T18:09:27.394 INFO:tasks.rados.rados.0.smithi160.stdout:2745: write oid 274 current snap is 0 2024-09-21T18:09:27.394 INFO:tasks.rados.rados.0.smithi160.stdout:2745: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-21T18:09:27.405 INFO:tasks.rados.rados.0.smithi160.stdout:2745: writing smithi16030472-274 from 457487 to 1207035 tid 1 2024-09-21T18:09:27.415 INFO:tasks.rados.rados.0.smithi160.stdout:2745: writing smithi16030472-274 from 1770389 to 2254824 tid 2 2024-09-21T18:09:27.421 INFO:tasks.rados.rados.0.smithi160.stdout:2745: writing smithi16030472-274 from 2746287 to 2857487 tid 3 2024-09-21T18:09:27.421 INFO:tasks.rados.rados.0.smithi160.stdout:2746: copy_from oid 38 from oid 1 current snap is 0 2024-09-21T18:09:27.422 INFO:tasks.rados.rados.0.smithi160.stdout:2737: finishing write tid 1 to smithi16030472-94 2024-09-21T18:09:27.422 INFO:tasks.rados.rados.0.smithi160.stdout:2747: write (excl) oid 457 current snap is 0 2024-09-21T18:09:27.422 INFO:tasks.rados.rados.0.smithi160.stdout:2747: seq_num 1242 ranges {474294=424797,1428871=628093,2657357=216937} 2024-09-21T18:09:27.429 INFO:tasks.rados.rados.0.smithi160.stdout:2747: writing smithi16030472-457 from 474294 to 899091 tid 1 2024-09-21T18:09:27.439 INFO:tasks.rados.rados.0.smithi160.stdout:2747: writing smithi16030472-457 from 1428871 to 2056964 tid 2 2024-09-21T18:09:27.441 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.442 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:27.442 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:27.442 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:27.442 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:27.446 INFO:tasks.rados.rados.0.smithi160.stdout:2747: writing smithi16030472-457 from 2657357 to 2874294 tid 3 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2748: cache_try_flush oid 122 snap -1 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: finishing write tid 3 to smithi16030472-459 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: finishing write tid 4 to smithi16030472-459 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: finishing write tid 5 to smithi16030472-459 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: oid 459 updating version 0 to 867 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: oid 459 version 867 is already newer than 866 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: oid 459 version 867 is already newer than 865 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: oid 459 updating version 867 to 868 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 868 (ObjNum 1239 snap 0 seq_num 1239) dirty exists 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2740: left oid 459 (ObjNum 1239 snap 0 seq_num 1239) 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:2730: expect (ObjNum 243 snap 0 seq_num 243) 2024-09-21T18:09:27.447 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 137460 2024-09-21T18:09:27.455 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:27.472 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:27.467+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:27.536 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:27.529+0000 7f94d9542640 -1 osd.15 104 handle_osd_map injecting map crc failure 2024-09-21T18:09:27.548 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1808058 2024-09-21T18:09:27.552 INFO:tasks.rados.rados.0.smithi160.stdout:2741: got (2) No such file or directory 2024-09-21T18:09:27.552 INFO:tasks.rados.rados.0.smithi160.stdout:2738: got (0) Success 2024-09-21T18:09:27.552 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 0 (ObjNum 814 snap 0 seq_num 814) dirty exists 2024-09-21T18:09:27.552 INFO:tasks.rados.rados.0.smithi160.stdout:2736: finishing write tid 1 to smithi16030472-326 2024-09-21T18:09:27.552 INFO:tasks.rados.rados.0.smithi160.stdout:2736: finishing write tid 2 to smithi16030472-326 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: finishing write tid 3 to smithi16030472-326 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: finishing write tid 4 to smithi16030472-326 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: oid 326 updating version 0 to 890 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: oid 326 updating version 890 to 892 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: oid 326 version 892 is already newer than 891 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 326 v 892 (ObjNum 1237 snap 0 seq_num 1237) dirty exists 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: left oid 326 (ObjNum 1237 snap 0 seq_num 1237) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2730: done (15 left) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2736: done (14 left) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2738: done (13 left) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2740: done (12 left) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2741: done (11 left) 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2749: cache_flush oid 247 snap -1 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2750: cache_flush oid 372 snap -1 2024-09-21T18:09:27.553 INFO:tasks.rados.rados.0.smithi160.stdout:2751: cache_evict oid 269 snap -1 2024-09-21T18:09:27.554 INFO:tasks.rados.rados.0.smithi160.stdout:2752: cache_flush oid 441 snap -1 2024-09-21T18:09:27.554 INFO:tasks.rados.rados.0.smithi160.stdout:2753: write oid 132 current snap is 0 2024-09-21T18:09:27.554 INFO:tasks.rados.rados.0.smithi160.stdout:2753: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-21T18:09:27.563 INFO:tasks.rados.rados.0.smithi160.stdout:2753: writing smithi16030472-132 from 491101 to 991147 tid 1 2024-09-21T18:09:27.573 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:27.570+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:27.574 INFO:tasks.rados.rados.0.smithi160.stdout:2753: writing smithi16030472-132 from 1487353 to 1975457 tid 2 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2753: writing smithi16030472-132 from 2401133 to 2891101 tid 3 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2745: finishing write tid 1 to smithi16030472-274 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2734: finishing copy_from racing read to smithi16030472-455 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2745: finishing write tid 2 to smithi16030472-274 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2745: finishing write tid 3 to smithi16030472-274 2024-09-21T18:09:27.585 INFO:tasks.rados.rados.0.smithi160.stdout:2745: finishing write tid 4 to smithi16030472-274 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: finishing write tid 5 to smithi16030472-274 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: oid 274 updating version 0 to 894 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: oid 274 updating version 894 to 895 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: oid 274 version 895 is already newer than 893 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: oid 274 updating version 895 to 896 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 274 v 896 (ObjNum 1241 snap 0 seq_num 1241) dirty exists 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2745: left oid 274 (ObjNum 1241 snap 0 seq_num 1241) 2024-09-21T18:09:27.586 INFO:tasks.rados.rados.0.smithi160.stdout:2742: expect (ObjNum 930 snap 0 seq_num 930) 2024-09-21T18:09:27.629 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.629 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:27.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 116.595330495, 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:27.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:09:27.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:09:27.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 116.591719944, 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:09:27.634 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:09:27.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 116.588913405, 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:09:27.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.643 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 116.585165205, 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:09:27.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.964314401, 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:27.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:27.651 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.86346032900001, 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:27.655 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.862704049, 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.862312492, 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.659 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:09:27.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.862011368, 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:09:27.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:09:27.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.861287518, 2024-09-21T18:09:27.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:09:27.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:09:27.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.760390679, 2024-09-21T18:09:27.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.672 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.673 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.760296396, 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.675 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:09:27.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.75938320100001, 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:09:27.679 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:09:27.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:27.683 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.75881351300001, 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:27.684 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:09:27.687 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.758086029, 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:27.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.691 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 115.69190412, 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.694 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:09:27.695 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 114.435858315, 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 114.435575517, 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.700 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:09:27.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:27.704 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 114.435496532, 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:09:27.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:09:27.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.708 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 114.43287820099999, 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:27.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:09:27.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:09:27.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:27.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:27.713 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:27.714 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:27.715 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:27.711+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:27.726 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 160584 2024-09-21T18:09:27.794 INFO:tasks.rados.rados.0.smithi160.stdout:2753: finishing write tid 1 to smithi16030472-132 2024-09-21T18:09:27.794 INFO:tasks.rados.rados.0.smithi160.stdout:2733: finishing copy_from racing read to smithi16030472-380 2024-09-21T18:09:27.794 INFO:tasks.rados.rados.0.smithi160.stdout:2742: done (15 left) 2024-09-21T18:09:27.794 INFO:tasks.rados.rados.0.smithi160.stdout:2745: done (14 left) 2024-09-21T18:09:27.794 INFO:tasks.rados.rados.0.smithi160.stdout:2754: write (excl) oid 2 current snap is 0 2024-09-21T18:09:27.795 INFO:tasks.rados.rados.0.smithi160.stdout:2754: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-21T18:09:27.800 INFO:tasks.rados.rados.0.smithi160.stdout:2754: writing smithi16030472-2 from 507908 to 1083203 tid 1 2024-09-21T18:09:27.808 INFO:tasks.rados.rados.0.smithi160.stdout:2754: writing smithi16030472-2 from 1829482 to 2461244 tid 2 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2755: cache_evict oid 395 snap -1 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: finishing write tid 2 to smithi16030472-94 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: finishing write tid 3 to smithi16030472-94 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: finishing write tid 4 to smithi16030472-94 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: oid 94 updating version 0 to 827 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: oid 94 version 827 is already newer than 826 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: oid 94 updating version 827 to 828 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 828 (ObjNum 1238 snap 0 seq_num 1238) dirty exists 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2737: left oid 94 (ObjNum 1238 snap 0 seq_num 1238) 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2733: finishing copy_from to smithi16030472-380 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 380 v 202 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2729: got (0) Success 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 0 (ObjNum 1201 snap 0 seq_num 1201) dirty exists 2024-09-21T18:09:27.809 INFO:tasks.rados.rados.0.smithi160.stdout:2744: got (0) Success 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 228 v 0 (ObjNum 300 snap 0 seq_num 300) dirty exists 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2748: got (0) Success 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 122 v 0 (ObjNum 742 snap 0 seq_num 742) dirty exists 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2751: got (0) Success 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2753: finishing write tid 2 to smithi16030472-132 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2746: finishing copy_from racing read to smithi16030472-38 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2734: finishing copy_from to smithi16030472-455 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 785 (ObjNum 1075 snap 0 seq_num 1075) dirty exists 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2747: finishing write tid 1 to smithi16030472-457 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2729: done (15 left) 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2733: done (14 left) 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2734: done (13 left) 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2737: done (12 left) 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2744: done (11 left) 2024-09-21T18:09:27.810 INFO:tasks.rados.rados.0.smithi160.stdout:2748: done (10 left) 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2751: done (9 left) 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2756: delete oid 427 current snap is 0 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: finishing write tid 2 to smithi16030472-457 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: finishing write tid 3 to smithi16030472-457 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2746: finishing copy_from to smithi16030472-38 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 827 (ObjNum 1178 snap 0 seq_num 1178) dirty exists 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: finishing write tid 4 to smithi16030472-457 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: finishing write tid 5 to smithi16030472-457 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: oid 457 updating version 0 to 899 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: oid 457 version 899 is already newer than 898 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: oid 457 updating version 899 to 900 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: oid 457 version 900 is already newer than 897 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 457 v 900 (ObjNum 1242 snap 0 seq_num 1242) dirty exists 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2747: left oid 457 (ObjNum 1242 snap 0 seq_num 1242) 2024-09-21T18:09:27.811 INFO:tasks.rados.rados.0.smithi160.stdout:2754: finishing write tid 1 to smithi16030472-2 2024-09-21T18:09:27.815 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:27.809+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:27.847 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.847 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:27.847 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:27.847 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:27.863 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:27.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:27.916 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:27.911+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:27.989 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:27.989 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:27.989 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:27.990 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:27.990 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:28.004 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:28.012 INFO:tasks.rados.rados.0.smithi160.stdout:2755: got (16) Device or resource busy 2024-09-21T18:09:28.012 INFO:tasks.rados.rados.0.smithi160.stdout:2753: finishing write tid 3 to smithi16030472-132 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: finishing write tid 4 to smithi16030472-132 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: finishing write tid 5 to smithi16030472-132 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: oid 132 updating version 0 to 859 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: oid 132 updating version 859 to 861 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: oid 132 version 861 is already newer than 858 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: oid 132 version 861 is already newer than 860 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 861 (ObjNum 1243 snap 0 seq_num 1243) dirty exists 2024-09-21T18:09:28.013 INFO:tasks.rados.rados.0.smithi160.stdout:2753: left oid 132 (ObjNum 1243 snap 0 seq_num 1243) 2024-09-21T18:09:28.016 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:28.009+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:28.021 INFO:tasks.rados.rados.0.smithi160.stdout:2752: got (0) Success 2024-09-21T18:09:28.021 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 0 (ObjNum 928 snap 0 seq_num 928) dirty exists 2024-09-21T18:09:28.066 INFO:tasks.rados.rados.0.smithi160.stdout:2743: finishing write tid 1 to smithi16030472-77 2024-09-21T18:09:28.069 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:28.076 INFO:tasks.rados.rados.0.smithi160.stdout:2743: finishing write tid 2 to smithi16030472-77 2024-09-21T18:09:28.095 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:28.095 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2746: done (9 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2747: done (8 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2752: done (7 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2753: done (6 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2755: done (5 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2756: done (4 left) 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2757: write oid 347 current snap is 0 2024-09-21T18:09:28.097 INFO:tasks.rados.rados.0.smithi160.stdout:2757: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-21T18:09:28.116 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:28.116 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:28.117 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:28.113+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:28.118 INFO:tasks.rados.rados.0.smithi160.stdout:2757: writing smithi16030472-347 from 541522 to 1267315 tid 1 2024-09-21T18:09:28.137 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:28.137 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:28.139 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:28.140 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:28.140 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:28.140 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:09:28.140 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2757: writing smithi16030472-347 from 1946446 to 2581877 tid 2 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2758: cache_evict oid 363 snap -1 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2754: finishing write tid 2 to smithi16030472-2 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2757: finishing write tid 1 to smithi16030472-347 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2759: read oid 339 snap -1 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2759: expect (ObjNum 516 snap 0 seq_num 516) 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2760: cache_flush oid 28 snap -1 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2761: read oid 336 snap -1 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2761: expect (ObjNum 335 snap 0 seq_num 335) 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2762: read oid 230 snap -1 2024-09-21T18:09:28.142 INFO:tasks.rados.rados.0.smithi160.stdout:2762: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:09:28.143 INFO:tasks.rados.rados.0.smithi160.stdout:2763: delete oid 37 current snap is 0 2024-09-21T18:09:28.152 INFO:tasks.rados.rados.0.smithi160.stdout:2759: expect (ObjNum 516 snap 0 seq_num 516) 2024-09-21T18:09:28.155 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:28.155 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:28.155 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:28.173 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:28.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:28.194 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:28.195 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:28.195 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:28.196 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:28.212 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:28.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:28.217 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:28.214+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:28.233 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:28.234 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:28.237 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:28.251 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:28.252 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:28.274 INFO:tasks.rados.rados.0.smithi160.stdout:2760: got (0) Success 2024-09-21T18:09:28.274 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 28 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:28.274 INFO:tasks.rados.rados.0.smithi160.stdout:2761: expect (ObjNum 335 snap 0 seq_num 335) 2024-09-21T18:09:28.289 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:28.290 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:28.304 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:28.308 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 203608 2024-09-21T18:09:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:09:28.353 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:28.351+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:28.354 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:28.354 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:28.381 INFO:tasks.rados.rados.0.smithi160.stdout:2743: finishing write tid 3 to smithi16030472-77 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2749: got (0) Success 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 0 (ObjNum 737 snap 0 seq_num 737) dirty exists 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2743: finishing write tid 4 to smithi16030472-77 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2743: oid 77 updating version 0 to 801 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2743: oid 77 updating version 801 to 869 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2743: oid 77 updating version 869 to 870 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 77 v 870 (ObjNum 1240 snap 0 seq_num 1240) dirty exists 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2743: left oid 77 (ObjNum 1240 snap 0 seq_num 1240) 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2758: got (16) Device or resource busy 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2750: got (0) Success 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 372 v 0 (ObjNum 390 snap 0 seq_num 390) dirty exists 2024-09-21T18:09:28.382 INFO:tasks.rados.rados.0.smithi160.stdout:2762: expect (ObjNum 796 snap 0 seq_num 796) 2024-09-21T18:09:28.384 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:28.409 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:28.419 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:28.413+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:28.429 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:28.429 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:28.429 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:28.430 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:28.443 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:28.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:28.450 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:28.453 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:28.453 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:28.517 INFO:tasks.rados.rados.0.smithi160.stdout:2754: finishing write tid 3 to smithi16030472-2 2024-09-21T18:09:28.517 INFO:tasks.rados.rados.0.smithi160.stdout:2754: finishing write tid 4 to smithi16030472-2 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2754: oid 2 updating version 0 to 817 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2754: oid 2 updating version 817 to 818 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2754: oid 2 updating version 818 to 829 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 2 v 829 (ObjNum 1244 snap 0 seq_num 1244) dirty exists 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2754: left oid 2 (ObjNum 1244 snap 0 seq_num 1244) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2757: finishing write tid 2 to smithi16030472-347 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2757: finishing write tid 3 to smithi16030472-347 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2743: done (10 left) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2749: done (9 left) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2750: done (8 left) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2754: done (7 left) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2758: done (6 left) 2024-09-21T18:09:28.518 INFO:tasks.rados.rados.0.smithi160.stdout:2759: done (5 left) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2760: done (4 left) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2761: done (3 left) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2762: done (2 left) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2763: done (1 left) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2764: read oid 49 snap -1 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2764: expect (ObjNum 1100 snap 0 seq_num 1100) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2765: cache_try_flush oid 344 snap -1 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: finishing write tid 4 to smithi16030472-347 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: oid 347 updating version 0 to 903 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: oid 347 version 903 is already newer than 902 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: oid 347 version 903 is already newer than 901 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 903 (ObjNum 1246 snap 0 seq_num 1246) dirty exists 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: left oid 347 (ObjNum 1246 snap 0 seq_num 1246) 2024-09-21T18:09:28.519 INFO:tasks.rados.rados.0.smithi160.stdout:2757: done (2 left) 2024-09-21T18:09:28.520 INFO:tasks.rados.rados.0.smithi160.stdout:2766: read oid 89 snap -1 2024-09-21T18:09:28.520 INFO:tasks.rados.rados.0.smithi160.stdout:2766: expect deleted 2024-09-21T18:09:28.520 INFO:tasks.rados.rados.0.smithi160.stdout:2767: delete oid 185 current snap is 0 2024-09-21T18:09:28.524 INFO:tasks.rados.rados.0.smithi160.stdout:2764: expect (ObjNum 1100 snap 0 seq_num 1100) 2024-09-21T18:09:28.524 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:28.529 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:28.527+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:28.610 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:28.610 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:28.621 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:28.617+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:28.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2765: got (0) Success 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 344 v 0 (ObjNum 815 snap 0 seq_num 815) dirty exists 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2764: done (3 left) 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2765: done (2 left) 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2766: done (1 left) 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2767: done (0 left) 2024-09-21T18:09:28.723 INFO:tasks.rados.rados.0.smithi160.stdout:2768: read oid 74 snap -1 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2768: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2769: cache_try_flush oid 120 snap -1 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2770: cache_evict oid 52 snap -1 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2771: cache_try_flush oid 190 snap -1 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2772: cache_try_flush oid 43 snap -1 2024-09-21T18:09:28.724 INFO:tasks.rados.rados.0.smithi160.stdout:2773: delete oid 204 current snap is 0 2024-09-21T18:09:28.724 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:28.717+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:28.791 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:28.791 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:28.806 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:28.823 INFO:tasks.rados.rados.0.smithi160.stdout:2769: got (0) Success 2024-09-21T18:09:28.823 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 120 v 0 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-21T18:09:28.823 INFO:tasks.rados.rados.0.smithi160.stdout:2772: got (0) Success 2024-09-21T18:09:28.824 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 43 v 0 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T18:09:28.824 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:28.821+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:28.830 INFO:tasks.rados.rados.0.smithi160.stdout:2768: expect (ObjNum 575 snap 0 seq_num 575) 2024-09-21T18:09:28.830 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:28.925 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:28.919+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:28.928 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:28.928 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:28.942 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2770: got (16) Device or resource busy 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2771: got (0) Success 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 190 v 0 (ObjNum 1150 snap 0 seq_num 1150) dirty exists 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2768: done (5 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2769: done (4 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2770: done (3 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2771: done (2 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2772: done (1 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2773: done (0 left) 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2774: cache_evict oid 30 snap -1 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2775: copy_from oid 326 from oid 215 current snap is 0 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2776: cache_flush oid 40 snap -1 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2777: copy_from oid 332 from oid 119 current snap is 0 2024-09-21T18:09:28.999 INFO:tasks.rados.rados.0.smithi160.stdout:2778: delete oid 130 current snap is 0 2024-09-21T18:09:29.026 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:29.021+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:29.100 INFO:tasks.rados.rados.0.smithi160.stdout:2776: got (0) Success 2024-09-21T18:09:29.100 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 0 (ObjNum 841 snap 0 seq_num 841) dirty exists 2024-09-21T18:09:29.105 INFO:tasks.rados.rados.0.smithi160.stdout:2774: got (0) Success 2024-09-21T18:09:29.106 INFO:tasks.rados.rados.0.smithi160.stdout:2774: done (4 left) 2024-09-21T18:09:29.106 INFO:tasks.rados.rados.0.smithi160.stdout:2776: done (3 left) 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2778: done (2 left) 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2779: cache_try_flush oid 293 snap -1 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2780: cache_evict oid 212 snap -1 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2781: cache_try_flush oid 282 snap -1 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2782: copy_from oid 149 from oid 42 current snap is 0 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2783: copy_from oid 100 from oid 373 current snap is 0 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2784: write (excl) oid 304 current snap is 0 2024-09-21T18:09:29.107 INFO:tasks.rados.rados.0.smithi160.stdout:2784: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-21T18:09:29.110 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.110 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.118 INFO:tasks.rados.rados.0.smithi160.stdout:2784: writing smithi16030472-304 from 625557 to 1043948 tid 1 2024-09-21T18:09:29.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:29.127 INFO:tasks.rados.rados.0.smithi160.stdout:2784: writing smithi16030472-304 from 1838856 to 2625283 tid 2 2024-09-21T18:09:29.127 INFO:tasks.rados.rados.0.smithi160.stdout:2785: write oid 148 current snap is 0 2024-09-21T18:09:29.127 INFO:tasks.rados.rados.0.smithi160.stdout:2775: finishing copy_from racing read to smithi16030472-326 2024-09-21T18:09:29.127 INFO:tasks.rados.rados.0.smithi160.stdout:2775: finishing copy_from to smithi16030472-326 2024-09-21T18:09:29.128 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 326 v 813 (ObjNum 1197 snap 0 seq_num 1197) dirty exists 2024-09-21T18:09:29.128 INFO:tasks.rados.rados.0.smithi160.stdout:2785: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-21T18:09:29.128 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:29.124+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:29.135 INFO:tasks.rados.rados.0.smithi160.stdout:2785: writing smithi16030472-148 from 642364 to 1136004 tid 1 2024-09-21T18:09:29.147 INFO:tasks.rados.rados.0.smithi160.stdout:2785: writing smithi16030472-148 from 1897338 to 2543776 tid 2 2024-09-21T18:09:29.147 INFO:tasks.rados.rados.0.smithi160.stdout:2775: done (8 left) 2024-09-21T18:09:29.147 INFO:tasks.rados.rados.0.smithi160.stdout:2786: cache_try_flush oid 37 snap -1 2024-09-21T18:09:29.147 INFO:tasks.rados.rados.0.smithi160.stdout:2787: delete oid 500 current snap is 0 2024-09-21T18:09:29.223 INFO:tasks.rados.rados.0.smithi160.stdout:2780: got (0) Success 2024-09-21T18:09:29.227 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:29.224+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:29.249 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.250 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:29.309 INFO:tasks.rados.rados.0.smithi160.stdout:2781: got (0) Success 2024-09-21T18:09:29.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 282 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:29.328 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:29.321+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:29.338 INFO:tasks.rados.rados.0.smithi160.stdout:2786: got (0) Success 2024-09-21T18:09:29.338 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 37 v 0 (ObjNum 129 snap 0 seq_num 3127277400) clean dne 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2780: done (9 left) 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2781: done (8 left) 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2786: done (7 left) 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2787: done (6 left) 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2788: read oid 40 snap -1 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2788: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2789: cache_flush oid 336 snap -1 2024-09-21T18:09:29.344 INFO:tasks.rados.rados.0.smithi160.stdout:2790: cache_flush oid 271 snap -1 2024-09-21T18:09:29.345 INFO:tasks.rados.rados.0.smithi160.stdout:2791: delete oid 339 current snap is 0 2024-09-21T18:09:29.376 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:29.372+0000 7f440980f640 -1 osd.13 106 handle_osd_map injecting map crc failure 2024-09-21T18:09:29.391 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.391 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.405 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:29.425 INFO:tasks.rados.rados.0.smithi160.stdout:2777: finishing copy_from racing read to smithi16030472-332 2024-09-21T18:09:29.430 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:29.424+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:29.436 INFO:tasks.rados.rados.0.smithi160.stdout:2788: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:09:29.452 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1101713 2024-09-21T18:09:29.495 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:29.531 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:29.527+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:29.537 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.537 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:29.578 INFO:tasks.rados.rados.0.smithi160.stdout:2789: got (0) Success 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 0 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2785: finishing write tid 1 to smithi16030472-148 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2785: finishing write tid 2 to smithi16030472-148 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2785: finishing write tid 3 to smithi16030472-148 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2779: got (0) Success 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 0 (ObjNum 523 snap 0 seq_num 523) dirty exists 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2783: finishing copy_from racing read to smithi16030472-100 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2782: finishing copy_from racing read to smithi16030472-149 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2777: finishing copy_from to smithi16030472-332 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 332 v 430 (ObjNum 534 snap 0 seq_num 534) dirty exists 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2782: finishing copy_from to smithi16030472-149 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 719 (ObjNum 1002 snap 0 seq_num 1002) dirty exists 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2790: got (2) No such file or directory 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2784: finishing write tid 1 to smithi16030472-304 2024-09-21T18:09:29.579 INFO:tasks.rados.rados.0.smithi160.stdout:2785: finishing write tid 4 to smithi16030472-148 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2785: oid 148 updating version 0 to 845 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2785: oid 148 version 845 is already newer than 844 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2785: oid 148 version 845 is already newer than 843 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 148 v 845 (ObjNum 1252 snap 0 seq_num 1252) dirty exists 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2785: left oid 148 (ObjNum 1252 snap 0 seq_num 1252) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2777: done (9 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2779: done (8 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2782: done (7 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2785: done (6 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2788: done (5 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2789: done (4 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2790: done (3 left) 2024-09-21T18:09:29.580 INFO:tasks.rados.rados.0.smithi160.stdout:2791: done (2 left) 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2792: copy_from oid 380 from oid 245 current snap is 0 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2793: read oid 278 snap -1 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2793: expect (ObjNum 1043 snap 0 seq_num 1043) 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2794: read oid 172 snap -1 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2794: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2795: write oid 16 current snap is 0 2024-09-21T18:09:29.581 INFO:tasks.rados.rados.0.smithi160.stdout:2795: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-21T18:09:29.597 INFO:tasks.rados.rados.0.smithi160.stdout:2795: writing smithi16030472-16 from 692785 to 1128525 tid 1 2024-09-21T18:09:29.619 INFO:tasks.rados.rados.0.smithi160.stdout:2795: writing smithi16030472-16 from 1672784 to 2466549 tid 2 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2795: writing smithi16030472-16 from 2897503 to 3092785 tid 3 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2796: cache_try_flush oid 295 snap -1 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2795: finishing write tid 1 to smithi16030472-16 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2797: cache_try_flush oid 256 snap -1 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2795: finishing write tid 2 to smithi16030472-16 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2798: read oid 424 snap -1 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2798: expect deleted 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2799: write oid 420 current snap is 0 2024-09-21T18:09:29.629 INFO:tasks.rados.rados.0.smithi160.stdout:2799: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-21T18:09:29.632 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:29.629+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:29.643 INFO:tasks.rados.rados.0.smithi160.stdout:2799: writing smithi16030472-420 from 709592 to 1220581 tid 1 2024-09-21T18:09:29.655 INFO:tasks.rados.rados.0.smithi160.stdout:2799: writing smithi16030472-420 from 1731266 to 2385042 tid 2 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2799: writing smithi16030472-420 from 3041279 to 3109592 tid 3 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2800: cache_try_flush oid 447 snap -1 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2801: cache_try_flush oid 176 snap -1 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2802: copy_from oid 102 from oid 191 current snap is 0 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2803: copy_from oid 127 from oid 398 current snap is 0 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2804: cache_evict oid 44 snap -1 2024-09-21T18:09:29.661 INFO:tasks.rados.rados.0.smithi160.stdout:2805: read oid 471 snap -1 2024-09-21T18:09:29.662 INFO:tasks.rados.rados.0.smithi160.stdout:2805: expect (ObjNum 746 snap 0 seq_num 746) 2024-09-21T18:09:29.662 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.715 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.715 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:29.732 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:29.729+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:29.777 INFO:tasks.rados.rados.0.smithi160.stdout:2798: done (15 left) 2024-09-21T18:09:29.777 INFO:tasks.rados.rados.0.smithi160.stdout:2806: cache_flush oid 288 snap -1 2024-09-21T18:09:29.777 INFO:tasks.rados.rados.0.smithi160.stdout:2795: finishing write tid 3 to smithi16030472-16 2024-09-21T18:09:29.777 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.830 INFO:tasks.rados.rados.0.smithi160.stdout:2800: got (2) No such file or directory 2024-09-21T18:09:29.830 INFO:tasks.rados.rados.0.smithi160.stdout:2800: done (15 left) 2024-09-21T18:09:29.830 INFO:tasks.rados.rados.0.smithi160.stdout:2807: read oid 145 snap -1 2024-09-21T18:09:29.830 INFO:tasks.rados.rados.0.smithi160.stdout:2807: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:29.830 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.833 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:29.827+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:29.834 INFO:tasks.rados.rados.0.smithi160.stdout:2795: finishing write tid 4 to smithi16030472-16 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: finishing write tid 5 to smithi16030472-16 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: oid 16 updating version 0 to 905 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: oid 16 updating version 905 to 908 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: oid 16 version 908 is already newer than 906 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: oid 16 version 908 is already newer than 907 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 16 v 908 (ObjNum 1255 snap 0 seq_num 1255) dirty exists 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: left oid 16 (ObjNum 1255 snap 0 seq_num 1255) 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2795: done (15 left) 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout:2808: cache_evict oid 249 snap -1 2024-09-21T18:09:29.835 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.837 INFO:tasks.rados.rados.0.smithi160.stdout:2808: got (0) Success 2024-09-21T18:09:29.838 INFO:tasks.rados.rados.0.smithi160.stdout:2808: done (15 left) 2024-09-21T18:09:29.838 INFO:tasks.rados.rados.0.smithi160.stdout:2809: read oid 300 snap -1 2024-09-21T18:09:29.838 INFO:tasks.rados.rados.0.smithi160.stdout:2809: expect deleted 2024-09-21T18:09:29.838 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.839 INFO:tasks.rados.rados.0.smithi160.stdout:2809: done (15 left) 2024-09-21T18:09:29.839 INFO:tasks.rados.rados.0.smithi160.stdout:2810: write oid 70 current snap is 0 2024-09-21T18:09:29.839 INFO:tasks.rados.rados.0.smithi160.stdout:2810: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-09-21T18:09:29.851 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:29.851 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:29.861 INFO:tasks.rados.rados.0.smithi160.stdout:2810: writing smithi16030472-70 from 726399 to 1312637 tid 1 2024-09-21T18:09:29.865 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:2810: writing smithi16030472-70 from 2073395 to 2870829 tid 2 2024-09-21T18:09:29.886 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:29.886 INFO:tasks.rados.rados.0.smithi160.stdout:2783: finishing copy_from to smithi16030472-100 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 100 v 839 (ObjNum 1200 snap 0 seq_num 1200) dirty exists 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: finishing write tid 2 to smithi16030472-304 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: finishing write tid 3 to smithi16030472-304 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: finishing write tid 4 to smithi16030472-304 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: oid 304 updating version 0 to 872 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: oid 304 version 872 is already newer than 871 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: oid 304 version 872 is already newer than 802 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 304 v 872 (ObjNum 1251 snap 0 seq_num 1251) dirty exists 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2784: left oid 304 (ObjNum 1251 snap 0 seq_num 1251) 2024-09-21T18:09:29.887 INFO:tasks.rados.rados.0.smithi160.stdout:2793: expect (ObjNum 1043 snap 0 seq_num 1043) 2024-09-21T18:09:29.932 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1433649 2024-09-21T18:09:29.933 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:29.928+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:29.945 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:29.940+0000 7f4976023640 -1 osd.5 106 handle_osd_map injecting map crc failure 2024-09-21T18:09:30.034 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:30.029+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:30.034 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:30.034 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:30.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:30.132 INFO:tasks.rados.rados.0.smithi160.stdout:2799: finishing write tid 1 to smithi16030472-420 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: finishing write tid 2 to smithi16030472-420 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: finishing write tid 3 to smithi16030472-420 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: finishing write tid 4 to smithi16030472-420 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: finishing write tid 5 to smithi16030472-420 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: oid 420 updating version 0 to 875 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: oid 420 version 875 is already newer than 874 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: oid 420 version 875 is already newer than 873 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: oid 420 updating version 875 to 876 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 420 v 876 (ObjNum 1256 snap 0 seq_num 1256) dirty exists 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2799: left oid 420 (ObjNum 1256 snap 0 seq_num 1256) 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2804: got (16) Device or resource busy 2024-09-21T18:09:30.133 INFO:tasks.rados.rados.0.smithi160.stdout:2794: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:09:30.134 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:30.129+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:30.147 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:30.174 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:30.174 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:30.188 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2797: got (2) No such file or directory 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2792: finishing copy_from to smithi16030472-380 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2792: got expected ENOENT (src dne) 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2792: finishing copy_from racing read to smithi16030472-380 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2802: finishing copy_from to smithi16030472-102 2024-09-21T18:09:30.191 INFO:tasks.rados.rados.0.smithi160.stdout:2802: got expected ENOENT (src dne) 2024-09-21T18:09:30.192 INFO:tasks.rados.rados.0.smithi160.stdout:2805: expect (ObjNum 746 snap 0 seq_num 746) 2024-09-21T18:09:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 345916 2024-09-21T18:09:30.235 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:30.232+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:30.252 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2328293 2024-09-21T18:09:30.258 INFO:tasks.rados.rados.0.smithi160.stdout:2796: got (0) Success 2024-09-21T18:09:30.258 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 257 snap 0 seq_num 3132096704) clean dne 2024-09-21T18:09:30.258 INFO:tasks.rados.rados.0.smithi160.stdout:2807: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:30.258 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:30.299 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2750338 2024-09-21T18:09:30.311 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:30.311 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:30.325 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2801: got (0) Success 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 176 v 0 (ObjNum 995 snap 0 seq_num 995) dirty exists 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2802: finishing copy_from racing read to smithi16030472-102 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2806: got (0) Success 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 288 v 0 (ObjNum 872 snap 0 seq_num 872) dirty exists 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2783: done (15 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2784: done (14 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2792: done (13 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2793: done (12 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2794: done (11 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2796: done (10 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2797: done (9 left) 2024-09-21T18:09:30.331 INFO:tasks.rados.rados.0.smithi160.stdout:2799: done (8 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2801: done (7 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2802: done (6 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2804: done (5 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2805: done (4 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2806: done (3 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2807: done (2 left) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2811: cache_evict oid 226 snap -1 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2803: finishing copy_from racing read to smithi16030472-127 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2812: cache_try_flush oid 458 snap -1 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2813: read oid 436 snap -1 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2813: expect (ObjNum 764 snap 0 seq_num 764) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2814: read oid 292 snap -1 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2814: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2815: read oid 136 snap -1 2024-09-21T18:09:30.332 INFO:tasks.rados.rados.0.smithi160.stdout:2815: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:2816: cache_evict oid 283 snap -1 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:2817: read oid 375 snap -1 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:2817: expect (ObjNum 1189 snap 0 seq_num 1189) 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:2818: delete oid 462 current snap is 0 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:2812: got (0) Success 2024-09-21T18:09:30.333 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 458 v 0 (ObjNum -1166044592 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:30.336 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:30.333+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:30.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:09:30.406 INFO:tasks.rados.rados.0.smithi160.stdout:2817: expect (ObjNum 1189 snap 0 seq_num 1189) 2024-09-21T18:09:30.437 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:30.429+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:30.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:30.456 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:30.457 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:30.471 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:30.538 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:30.532+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:30.551 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2194209 2024-09-21T18:09:30.587 INFO:tasks.rados.rados.0.smithi160.stdout:2811: got (0) Success 2024-09-21T18:09:30.587 INFO:tasks.rados.rados.0.smithi160.stdout:2814: expect (ObjNum 640 snap 0 seq_num 640) 2024-09-21T18:09:30.605 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:30.606 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:30.626 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:30.626 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:30.638 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:30.633+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:30.646 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:30.646 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:30.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:30.647 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:30.661 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:30.665 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:30.665 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:30.683 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:30.683 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:30.703 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:30.703 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:30.716 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2556560 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: finishing write tid 1 to smithi16030472-70 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: finishing write tid 2 to smithi16030472-70 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: finishing write tid 3 to smithi16030472-70 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: finishing write tid 4 to smithi16030472-70 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: oid 70 updating version 0 to 865 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: oid 70 updating version 865 to 866 2024-09-21T18:09:30.723 INFO:tasks.rados.rados.0.smithi160.stdout:2810: oid 70 updating version 866 to 867 2024-09-21T18:09:30.724 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 70 v 867 (ObjNum 1257 snap 0 seq_num 1257) dirty exists 2024-09-21T18:09:30.724 INFO:tasks.rados.rados.0.smithi160.stdout:2810: left oid 70 (ObjNum 1257 snap 0 seq_num 1257) 2024-09-21T18:09:30.724 INFO:tasks.rados.rados.0.smithi160.stdout:2803: finishing copy_from to smithi16030472-127 2024-09-21T18:09:30.724 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 127 v 589 (ObjNum 875 snap 0 seq_num 875) dirty exists 2024-09-21T18:09:30.724 INFO:tasks.rados.rados.0.smithi160.stdout:2815: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:09:30.724 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:30.725 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1301187 2024-09-21T18:09:30.740 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:30.736+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:30.745 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:30.745 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:30.760 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1651549 2024-09-21T18:09:30.767 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:30.791 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:30.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:30.813 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:30.815 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:30.816 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:30.816 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:30.816 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:30.816 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:30.820 INFO:tasks.rados.rados.0.smithi160.stdout:2813: expect (ObjNum 764 snap 0 seq_num 764) 2024-09-21T18:09:30.822 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1460720 2024-09-21T18:09:30.829 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:30.831 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:30.832 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:30.840 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:30.836+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:30.854 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:30.877 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:30.890 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:30.897 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:30.903 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:30.903 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2816: got (0) Success 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2803: done (9 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2810: done (8 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2811: done (7 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2812: done (6 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2813: done (5 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2814: done (4 left) 2024-09-21T18:09:30.934 INFO:tasks.rados.rados.0.smithi160.stdout:2815: done (3 left) 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2816: done (2 left) 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2817: done (1 left) 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2818: done (0 left) 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2819: cache_evict oid 426 snap -1 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2820: read oid 69 snap -1 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2820: expect deleted 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2821: cache_try_flush oid 416 snap -1 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2822: cache_evict oid 273 snap -1 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2823: write oid 147 current snap is 0 2024-09-21T18:09:30.935 INFO:tasks.rados.rados.0.smithi160.stdout:2823: seq_num 1259 ranges {760013=736736,2190359=401103} 2024-09-21T18:09:30.941 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:30.936+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:30.944 INFO:tasks.rados.rados.0.smithi160.stdout:2823: writing smithi16030472-147 from 760013 to 1496749 tid 1 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2823: writing smithi16030472-147 from 2190359 to 2591462 tid 2 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2824: cache_evict oid 277 snap -1 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2825: cache_evict oid 76 snap -1 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2826: read oid 452 snap -1 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2826: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:09:30.951 INFO:tasks.rados.rados.0.smithi160.stdout:2827: write oid 394 current snap is 0 2024-09-21T18:09:30.952 INFO:tasks.rados.rados.0.smithi160.stdout:2827: seq_num 1260 ranges {776820=411985,1848841=544761,2918265=258555} 2024-09-21T18:09:30.959 INFO:tasks.rados.rados.0.smithi160.stdout:2827: writing smithi16030472-394 from 776820 to 1188805 tid 1 2024-09-21T18:09:30.968 INFO:tasks.rados.rados.0.smithi160.stdout:2827: writing smithi16030472-394 from 1848841 to 2393602 tid 2 2024-09-21T18:09:30.973 INFO:tasks.rados.rados.0.smithi160.stdout:2827: writing smithi16030472-394 from 2918265 to 3176820 tid 3 2024-09-21T18:09:30.974 INFO:tasks.rados.rados.0.smithi160.stdout:2828: copy_from oid 239 from oid 170 current snap is 0 2024-09-21T18:09:30.974 INFO:tasks.rados.rados.0.smithi160.stdout:2829: write oid 433 current snap is 0 2024-09-21T18:09:30.974 INFO:tasks.rados.rados.0.smithi160.stdout:2829: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-21T18:09:30.982 INFO:tasks.rados.rados.0.smithi160.stdout:2829: writing smithi16030472-433 from 793627 to 1280861 tid 1 2024-09-21T18:09:30.987 INFO:tasks.rados.rados.0.smithi160.stdout:2829: writing smithi16030472-433 from 1790970 to 2195742 tid 2 2024-09-21T18:09:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:2829: writing smithi16030472-433 from 2829335 to 3193627 tid 3 2024-09-21T18:09:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:2830: read oid 179 snap -1 2024-09-21T18:09:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:2830: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:2831: write (excl) oid 30 current snap is 0 2024-09-21T18:09:30.993 INFO:tasks.rados.rados.0.smithi160.stdout:2831: seq_num 1262 ranges {410434=678836,1565805=548430,2573111=637323} 2024-09-21T18:09:30.999 INFO:tasks.rados.rados.0.smithi160.stdout:2831: writing smithi16030472-30 from 410434 to 1089270 tid 1 2024-09-21T18:09:31.004 INFO:tasks.rados.rados.0.smithi160.stdout:2831: writing smithi16030472-30 from 1565805 to 2114235 tid 2 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:31.007 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.85801072999999, 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:09:31.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:09:31.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:09:31.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:09:31.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.013 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.85796442099999, 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:09:31.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:09:31.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.85586850799999, 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.019 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.020 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:09:31.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:09:31.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:09:31.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:09:31.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:09:31.026 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.85520539300001, 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.027 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.030 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.855168894, 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.033 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:31.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:09:31.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:09:31.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.72153072200001, 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.044 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.720818931, 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.045 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 137.72078136799999, 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.051 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:31.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:09:31.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:09:31.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.923395369, 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:09:31.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:09:31.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.923200823, 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.061 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:09:31.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:09:31.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.922914435, 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.065 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.922898355, 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:31.073 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.893971103, 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:31.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:09:31.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:09:31.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.077 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.88775202799999, 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:09:31.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.88265043299999, 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.882316485, 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.087 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.089 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.882296006, 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.090 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:09:31.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.093 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.795967479, 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:31.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:09:31.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:09:31.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.097 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.787507293, 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:31.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:09:31.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:09:31.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.101 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.781579588, 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.102 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:09:31.103 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:09:31.104 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.105 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:31.106 INFO:tasks.rados.rados.0.smithi160.stdout:2831: writing smithi16030472-30 from 2573111 to 3210434 tid 3 2024-09-21T18:09:31.107 INFO:tasks.rados.rados.0.smithi160.stdout:2832: write oid 1 current snap is 0 2024-09-21T18:09:31.107 INFO:tasks.rados.rados.0.smithi160.stdout:2832: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-21T18:09:31.107 INFO:tasks.rados.rados.0.smithi160.stdout:2832: writing smithi16030472-1 from 427241 to 1181326 tid 1 2024-09-21T18:09:31.108 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:31.108 INFO:tasks.rados.rados.0.smithi160.stdout:2832: writing smithi16030472-1 from 1624287 to 2032728 tid 2 2024-09-21T18:09:31.108 INFO:tasks.rados.rados.0.smithi160.stdout:2832: writing smithi16030472-1 from 2600534 to 3150498 tid 3 2024-09-21T18:09:31.108 INFO:tasks.rados.rados.0.smithi160.stdout:2833: delete oid 126 current snap is 0 2024-09-21T18:09:31.108 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:31.104+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:31.130 INFO:tasks.rados.rados.0.smithi160.stdout:2827: finishing write tid 1 to smithi16030472-394 2024-09-21T18:09:31.134 INFO:tasks.rados.rados.0.smithi160.stdout:2827: finishing write tid 2 to smithi16030472-394 2024-09-21T18:09:31.153 INFO:tasks.rados.rados.0.smithi160.stdout:2823: finishing write tid 1 to smithi16030472-147 2024-09-21T18:09:31.153 INFO:tasks.rados.rados.0.smithi160.stdout:2823: finishing write tid 2 to smithi16030472-147 2024-09-21T18:09:31.153 INFO:tasks.rados.rados.0.smithi160.stdout:2823: finishing write tid 3 to smithi16030472-147 2024-09-21T18:09:31.153 INFO:tasks.rados.rados.0.smithi160.stdout:2823: finishing write tid 4 to smithi16030472-147 2024-09-21T18:09:31.153 INFO:tasks.rados.rados.0.smithi160.stdout:2823: oid 147 updating version 0 to 909 2024-09-21T18:09:31.154 INFO:tasks.rados.rados.0.smithi160.stdout:2823: oid 147 updating version 909 to 910 2024-09-21T18:09:31.154 INFO:tasks.rados.rados.0.smithi160.stdout:2823: oid 147 updating version 910 to 911 2024-09-21T18:09:31.154 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 911 (ObjNum 1259 snap 0 seq_num 1259) dirty exists 2024-09-21T18:09:31.154 INFO:tasks.rados.rados.0.smithi160.stdout:2823: left oid 147 (ObjNum 1259 snap 0 seq_num 1259) 2024-09-21T18:09:31.154 INFO:tasks.rados.rados.0.smithi160.stdout:2827: finishing write tid 3 to smithi16030472-394 2024-09-21T18:09:31.157 INFO:tasks.rados.rados.0.smithi160.stdout:2826: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:09:31.208 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:31.207+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:7654 4.2 4.aedb9176 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:31.051370+0000", 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.20161106000000001, 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20164003599999999, 2024-09-21T18:09:31.254 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 7654 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051370+0000", 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051370+0000", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051371+0000", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6600000000000002e-07 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051374+0000", 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8059999999999999e-06 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051379+0000", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7010000000000001e-06 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051383+0000", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9829999999999998e-06 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:7655 4.2 4.aedb9176 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e107)", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:31.051389+0000", 2024-09-21T18:09:31.257 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.20159286000000001, 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201678948, 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 7655 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051389+0000", 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051389+0000", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051389+0000", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9900000000000001e-07 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051391+0000", 2024-09-21T18:09:31.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.776e-06 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051398+0000", 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9310000000000002e-06 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:31.051408+0000", 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0487e-05 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.260 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:31.261 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 2 2024-09-21T18:09:31.261 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:31.273 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:31.285 INFO:tasks.rados.rados.0.smithi160.stdout:2824: got (16) Device or resource busy 2024-09-21T18:09:31.285 INFO:tasks.rados.rados.0.smithi160.stdout:2822: got (16) Device or resource busy 2024-09-21T18:09:31.285 INFO:tasks.rados.rados.0.smithi160.stdout:2819: got (0) Success 2024-09-21T18:09:31.285 INFO:tasks.rados.rados.0.smithi160.stdout:2821: got (0) Success 2024-09-21T18:09:31.285 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 416 v 0 (ObjNum 49 snap 0 seq_num 3111024432) clean dne 2024-09-21T18:09:31.286 INFO:tasks.rados.rados.0.smithi160.stdout:2825: got (16) Device or resource busy 2024-09-21T18:09:31.286 INFO:tasks.rados.rados.0.smithi160.stdout:2830: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:31.288 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1288589 2024-09-21T18:09:31.309 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:31.304+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:31.311 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: finishing write tid 4 to smithi16030472-394 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: finishing write tid 5 to smithi16030472-394 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: oid 394 updating version 0 to 879 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: oid 394 updating version 879 to 881 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: oid 394 version 881 is already newer than 880 2024-09-21T18:09:31.378 INFO:tasks.rados.rados.0.smithi160.stdout:2827: oid 394 version 881 is already newer than 878 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 394 v 881 (ObjNum 1260 snap 0 seq_num 1260) dirty exists 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2827: left oid 394 (ObjNum 1260 snap 0 seq_num 1260) 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: finishing write tid 1 to smithi16030472-30 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: finishing write tid 2 to smithi16030472-30 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2829: finishing write tid 1 to smithi16030472-433 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2829: finishing write tid 2 to smithi16030472-433 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: finishing write tid 3 to smithi16030472-30 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: finishing write tid 4 to smithi16030472-30 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: finishing write tid 5 to smithi16030472-30 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: oid 30 updating version 0 to 822 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: oid 30 version 822 is already newer than 820 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: oid 30 updating version 822 to 823 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:2831: oid 30 version 823 is already newer than 821 2024-09-21T18:09:31.379 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 30 v 823 (ObjNum 1262 snap 0 seq_num 1262) dirty exists 2024-09-21T18:09:31.380 INFO:tasks.rados.rados.0.smithi160.stdout:2831: left oid 30 (ObjNum 1262 snap 0 seq_num 1262) 2024-09-21T18:09:31.386 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:09:31.403 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:31.403 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:31.404 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:31.404 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:31.409 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:31.407+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:31.418 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2819: done (14 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2820: done (13 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2821: done (12 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2822: done (11 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2823: done (10 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2824: done (9 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2825: done (8 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2826: done (7 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2827: done (6 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2830: done (5 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2831: done (4 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2833: done (3 left) 2024-09-21T18:09:31.464 INFO:tasks.rados.rados.0.smithi160.stdout:2834: read oid 414 snap -1 2024-09-21T18:09:31.465 INFO:tasks.rados.rados.0.smithi160.stdout:2834: expect (ObjNum 1053 snap 0 seq_num 1053) 2024-09-21T18:09:31.465 INFO:tasks.rados.rados.0.smithi160.stdout:2835: write oid 379 current snap is 0 2024-09-21T18:09:31.465 INFO:tasks.rados.rados.0.smithi160.stdout:2835: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-09-21T18:09:31.469 INFO:tasks.rados.rados.0.smithi160.stdout:2835: writing smithi16030472-379 from 460855 to 965438 tid 1 2024-09-21T18:09:31.475 INFO:tasks.rados.rados.0.smithi160.stdout:2835: writing smithi16030472-379 from 1624898 to 2037008 tid 2 2024-09-21T18:09:31.481 INFO:tasks.rados.rados.0.smithi160.stdout:2835: writing smithi16030472-379 from 2539027 to 3111535 tid 3 2024-09-21T18:09:31.481 INFO:tasks.rados.rados.0.smithi160.stdout:2836: delete oid 409 current snap is 0 2024-09-21T18:09:31.481 INFO:tasks.rados.rados.0.smithi160.stdout:2835: finishing write tid 1 to smithi16030472-379 2024-09-21T18:09:31.510 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:31.508+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:31.560 INFO:tasks.rados.rados.0.smithi160.stdout:2829: finishing write tid 3 to smithi16030472-433 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: finishing write tid 4 to smithi16030472-433 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: finishing write tid 5 to smithi16030472-433 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: oid 433 updating version 0 to 871 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: oid 433 version 871 is already newer than 870 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: oid 433 version 871 is already newer than 869 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: oid 433 version 871 is already newer than 868 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 871 (ObjNum 1261 snap 0 seq_num 1261) dirty exists 2024-09-21T18:09:31.561 INFO:tasks.rados.rados.0.smithi160.stdout:2829: left oid 433 (ObjNum 1261 snap 0 seq_num 1261) 2024-09-21T18:09:31.568 INFO:tasks.rados.rados.0.smithi160.stdout:2832: finishing write tid 1 to smithi16030472-1 2024-09-21T18:09:31.582 INFO:tasks.rados.rados.0.smithi160.stdout:2835: finishing write tid 2 to smithi16030472-379 2024-09-21T18:09:31.585 INFO:tasks.rados.rados.0.smithi160.stdout:2829: done (5 left) 2024-09-21T18:09:31.585 INFO:tasks.rados.rados.0.smithi160.stdout:2836: done (4 left) 2024-09-21T18:09:31.585 INFO:tasks.rados.rados.0.smithi160.stdout:2837: cache_evict oid 90 snap -1 2024-09-21T18:09:31.585 INFO:tasks.rados.rados.0.smithi160.stdout:2838: write (excl) oid 245 current snap is 0 2024-09-21T18:09:31.586 INFO:tasks.rados.rados.0.smithi160.stdout:2838: seq_num 1267 ranges {494469=655081,1741862=415779,2710226=584243} 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.914654555, 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.591 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.592 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:09:31.593 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:09:31.594 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.914584193, 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:09:31.595 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.902782758, 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.599 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.600 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.89491637500001, 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.603 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:31.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:09:31.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.606 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.894530731, 2024-09-21T18:09:31.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.608 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.611 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 132.89448528200001, 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:31.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:09:31.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:09:31.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:09:31.615 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:09:31.616 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 131.846587365, 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.617 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.618 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.619 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:09:31.620 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 131.84647972299999, 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.621 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.624 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 126.27069710799999, 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:09:31.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.626 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:09:31.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:09:31.628 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:09:31.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 126.27064517300001, 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.631 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:09:31.634 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:09:31.635 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.636 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.503940785, 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.637 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.638 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.639 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.640 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.641 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.642 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.274837527, 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:09:31.643 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:09:31.644 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.645 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.646 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.647 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.648 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.274409537, 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.649 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:31.650 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:09:31.651 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.652 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:31.653 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.654 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:31.655 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 125.27436296800001, 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.656 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:09:31.657 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:09:31.658 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:09:31.659 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:09:31.660 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 120.773693908, 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.661 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.662 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:09:31.663 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:09:31.664 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:09:31.665 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.666 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.766162112, 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.667 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.668 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.669 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:09:31.670 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:09:31.671 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:09:31.672 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.76608779199999, 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.673 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.674 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.675 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.676 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.677 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.678 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.364284547, 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:31.679 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:09:31.680 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:09:31.681 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.682 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 113.261934659, 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.683 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:09:31.684 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:09:31.685 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:09:31.686 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:09:31.687 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6670 4.2 4:574ca510:::smithi16030472-466:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 20.949306261, 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4286385400000001, 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:31.688 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6670 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:31.689 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637674+0000", 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637680+0000", 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6629999999999998e-06 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.690 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637691+0000", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1097000000000001e-05 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637700+0000", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054141+0000", 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41644096200000003 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.691 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054299+0000", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157572 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.065938+0000", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.011639553 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066192+0000", 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253222 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:31.692 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066312+0000", 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120285 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:31.693 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:31.694 INFO:tasks.rados.rados.0.smithi160.stdout:2838: writing smithi16030472-245 from 494469 to 1149550 tid 1 2024-09-21T18:09:31.694 INFO:tasks.rados.rados.0.smithi160.stdout:2838: writing smithi16030472-245 from 1741862 to 2157641 tid 2 2024-09-21T18:09:31.694 INFO:tasks.rados.rados.0.smithi160.stdout:2838: writing smithi16030472-245 from 2710226 to 3294469 tid 3 2024-09-21T18:09:31.694 INFO:tasks.rados.rados.0.smithi160.stdout:2839: write oid 276 current snap is 0 2024-09-21T18:09:31.695 INFO:tasks.rados.rados.0.smithi160.stdout:2839: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-21T18:09:31.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2839: writing smithi16030472-276 from 511276 to 1241606 tid 1 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2839: writing smithi16030472-276 from 1800344 to 2359781 tid 2 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2839: writing smithi16030472-276 from 3021296 to 3311276 tid 3 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2840: read oid 291 snap -1 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2840: expect (ObjNum 683 snap 0 seq_num 683) 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2841: write (excl) oid 398 current snap is 0 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2841: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2841: writing smithi16030472-398 from 528083 to 933662 tid 1 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2841: writing smithi16030472-398 from 1342473 to 1761921 tid 2 2024-09-21T18:09:31.697 INFO:tasks.rados.rados.0.smithi160.stdout:2841: writing smithi16030472-398 from 2248719 to 2982668 tid 3 2024-09-21T18:09:31.698 INFO:tasks.rados.rados.0.smithi160.stdout:2842: read oid 371 snap -1 2024-09-21T18:09:31.698 INFO:tasks.rados.rados.0.smithi160.stdout:2842: expect (ObjNum 1138 snap 0 seq_num 1138) 2024-09-21T18:09:31.698 INFO:tasks.rados.rados.0.smithi160.stdout:2843: copy_from oid 270 from oid 266 current snap is 0 2024-09-21T18:09:31.698 INFO:tasks.rados.rados.0.smithi160.stdout:2838: finishing write tid 1 to smithi16030472-245 2024-09-21T18:09:31.698 INFO:tasks.rados.rados.0.smithi160.stdout:2840: expect (ObjNum 683 snap 0 seq_num 683) 2024-09-21T18:09:31.698 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:31.688+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:31.718 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:31.796 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:31.789+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:31.814 INFO:tasks.rados.rados.0.smithi160.stdout:2834: expect (ObjNum 1053 snap 0 seq_num 1053) 2024-09-21T18:09:31.815 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:31.836 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:31.836 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:31.836 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:31.836 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:31.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:31.897 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:31.891+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:31.921 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:31.972 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:31.972 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:31.972 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:31.972 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:31.972 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:31.980 INFO:tasks.rados.rados.0.smithi160.stdout:2832: finishing write tid 2 to smithi16030472-1 2024-09-21T18:09:31.980 INFO:tasks.rados.rados.0.smithi160.stdout:2832: finishing write tid 3 to smithi16030472-1 2024-09-21T18:09:31.980 INFO:tasks.rados.rados.0.smithi160.stdout:2835: finishing write tid 3 to smithi16030472-379 2024-09-21T18:09:31.980 INFO:tasks.rados.rados.0.smithi160.stdout:2828: finishing copy_from to smithi16030472-239 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2828: got expected ENOENT (src dne) 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: finishing write tid 2 to smithi16030472-245 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: finishing write tid 3 to smithi16030472-245 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: finishing write tid 4 to smithi16030472-245 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: finishing write tid 5 to smithi16030472-245 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: oid 245 updating version 0 to 834 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: oid 245 version 834 is already newer than 831 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: oid 245 version 834 is already newer than 832 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: oid 245 version 834 is already newer than 833 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 245 v 834 (ObjNum 1267 snap 0 seq_num 1267) dirty exists 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2838: left oid 245 (ObjNum 1267 snap 0 seq_num 1267) 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2832: finishing write tid 4 to smithi16030472-1 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2832: finishing write tid 5 to smithi16030472-1 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2832: oid 1 updating version 0 to 885 2024-09-21T18:09:31.981 INFO:tasks.rados.rados.0.smithi160.stdout:2832: oid 1 version 885 is already newer than 882 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2832: oid 1 version 885 is already newer than 884 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2832: oid 1 version 885 is already newer than 883 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 1 v 885 (ObjNum 1263 snap 0 seq_num 1263) dirty exists 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2832: left oid 1 (ObjNum 1263 snap 0 seq_num 1263) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2843: finishing copy_from racing read to smithi16030472-270 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2828: finishing copy_from racing read to smithi16030472-239 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2841: finishing write tid 1 to smithi16030472-398 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2835: finishing write tid 4 to smithi16030472-379 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2841: finishing write tid 2 to smithi16030472-398 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2828: done (10 left) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2832: done (9 left) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2834: done (8 left) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2838: done (7 left) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2840: done (6 left) 2024-09-21T18:09:31.982 INFO:tasks.rados.rados.0.smithi160.stdout:2844: read oid 32 snap -1 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2844: expect (ObjNum 846 snap 0 seq_num 846) 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2845: read oid 68 snap -1 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2845: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2846: cache_evict oid 429 snap -1 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2847: read oid 274 snap -1 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2847: expect (ObjNum 1241 snap 0 seq_num 1241) 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2848: write (excl) oid 51 current snap is 0 2024-09-21T18:09:31.983 INFO:tasks.rados.rados.0.smithi160.stdout:2848: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-21T18:09:31.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:31.989 INFO:tasks.rados.rados.0.smithi160.stdout:2848: writing smithi16030472-51 from 544890 to 1142071 tid 1 2024-09-21T18:09:31.997 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:31.993+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:32.000 INFO:tasks.rados.rados.0.smithi160.stdout:2848: writing smithi16030472-51 from 1917308 to 2480414 tid 2 2024-09-21T18:09:32.007 INFO:tasks.rados.rados.0.smithi160.stdout:2848: writing smithi16030472-51 from 3076142 to 3344890 tid 3 2024-09-21T18:09:32.007 INFO:tasks.rados.rados.0.smithi160.stdout:2849: read oid 209 snap -1 2024-09-21T18:09:32.007 INFO:tasks.rados.rados.0.smithi160.stdout:2849: expect deleted 2024-09-21T18:09:32.007 INFO:tasks.rados.rados.0.smithi160.stdout:2850: write (excl) oid 55 current snap is 0 2024-09-21T18:09:32.007 INFO:tasks.rados.rados.0.smithi160.stdout:2835: finishing write tid 5 to smithi16030472-379 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2835: oid 379 updating version 0 to 913 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2835: oid 379 updating version 913 to 915 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2835: oid 379 version 915 is already newer than 914 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2835: oid 379 version 915 is already newer than 912 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 379 v 915 (ObjNum 1265 snap 0 seq_num 1265) dirty exists 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2835: left oid 379 (ObjNum 1265 snap 0 seq_num 1265) 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2843: finishing copy_from to smithi16030472-270 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 270 v 744 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:2844: expect (ObjNum 846 snap 0 seq_num 846) 2024-09-21T18:09:32.008 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.062 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.098 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:32.093+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:32.116 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:32.116 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:32.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:32.116 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:09:32.116 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:32.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:32.149 INFO:tasks.rados.rados.0.smithi160.stdout:2841: finishing write tid 3 to smithi16030472-398 2024-09-21T18:09:32.149 INFO:tasks.rados.rados.0.smithi160.stdout:2842: expect (ObjNum 1138 snap 0 seq_num 1138) 2024-09-21T18:09:32.179 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.219 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:32.217+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:32.221 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.263 INFO:tasks.rados.rados.0.smithi160.stdout:2841: finishing write tid 4 to smithi16030472-398 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2837: got (0) Success 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: finishing write tid 1 to smithi16030472-276 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: finishing write tid 2 to smithi16030472-276 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: finishing write tid 3 to smithi16030472-276 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: finishing write tid 4 to smithi16030472-276 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: finishing write tid 5 to smithi16030472-276 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: oid 276 updating version 0 to 887 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: oid 276 updating version 887 to 889 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: oid 276 version 889 is already newer than 888 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: oid 276 version 889 is already newer than 886 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 889 (ObjNum 1268 snap 0 seq_num 1268) dirty exists 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2839: left oid 276 (ObjNum 1268 snap 0 seq_num 1268) 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2841: finishing write tid 5 to smithi16030472-398 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2841: oid 398 updating version 0 to 837 2024-09-21T18:09:32.264 INFO:tasks.rados.rados.0.smithi160.stdout:2841: oid 398 version 837 is already newer than 836 2024-09-21T18:09:32.265 INFO:tasks.rados.rados.0.smithi160.stdout:2841: oid 398 updating version 837 to 838 2024-09-21T18:09:32.265 INFO:tasks.rados.rados.0.smithi160.stdout:2841: oid 398 version 838 is already newer than 835 2024-09-21T18:09:32.265 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 398 v 838 (ObjNum 1269 snap 0 seq_num 1269) dirty exists 2024-09-21T18:09:32.265 INFO:tasks.rados.rados.0.smithi160.stdout:2841: left oid 398 (ObjNum 1269 snap 0 seq_num 1269) 2024-09-21T18:09:32.265 INFO:tasks.rados.rados.0.smithi160.stdout:2845: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:32.288 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2225797 2024-09-21T18:09:32.294 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:32.294 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:32.294 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:32.294 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:32.295 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.300 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:32.296+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:32.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:32.323 INFO:tasks.rados.rados.0.smithi160.stdout:2846: got (0) Success 2024-09-21T18:09:32.323 INFO:tasks.rados.rados.0.smithi160.stdout:2847: expect (ObjNum 1241 snap 0 seq_num 1241) 2024-09-21T18:09:32.401 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:32.397+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:32.406 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.450 INFO:tasks.rados.rados.0.smithi160.stdout:2848: finishing write tid 1 to smithi16030472-51 2024-09-21T18:09:32.450 INFO:tasks.rados.rados.0.smithi160.stdout:2848: finishing write tid 2 to smithi16030472-51 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: finishing write tid 3 to smithi16030472-51 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: finishing write tid 4 to smithi16030472-51 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: finishing write tid 5 to smithi16030472-51 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: oid 51 updating version 0 to 874 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: oid 51 version 874 is already newer than 872 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: oid 51 updating version 874 to 875 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: oid 51 version 875 is already newer than 873 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 51 v 875 (ObjNum 1270 snap 0 seq_num 1270) dirty exists 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2848: left oid 51 (ObjNum 1270 snap 0 seq_num 1270) 2024-09-21T18:09:32.451 INFO:tasks.rados.rados.0.smithi160.stdout:2850: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-09-21T18:09:32.470 INFO:tasks.rados.rados.0.smithi160.stdout:2850: writing smithi16030472-55 from 561697 to 1234127 tid 1 2024-09-21T18:09:32.473 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:32.473 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:32.474 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:32.474 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:32.474 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:32.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:32.494 INFO:tasks.rados.rados.0.smithi160.stdout:2850: writing smithi16030472-55 from 1975790 to 2682554 tid 2 2024-09-21T18:09:32.501 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:32.496+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:32.504 INFO:tasks.rados.rados.0.smithi160.stdout:2850: writing smithi16030472-55 from 3103565 to 3361697 tid 3 2024-09-21T18:09:32.504 INFO:tasks.rados.rados.0.smithi160.stdout:2835: done (12 left) 2024-09-21T18:09:32.504 INFO:tasks.rados.rados.0.smithi160.stdout:2837: done (11 left) 2024-09-21T18:09:32.504 INFO:tasks.rados.rados.0.smithi160.stdout:2839: done (10 left) 2024-09-21T18:09:32.504 INFO:tasks.rados.rados.0.smithi160.stdout:2841: done (9 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2842: done (8 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2843: done (7 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2844: done (6 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2845: done (5 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2846: done (4 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2847: done (3 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2848: done (2 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2849: done (1 left) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2851: copy_from oid 161 from oid 473 current snap is 0 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2852: read oid 264 snap -1 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2852: expect (ObjNum 1144 snap 0 seq_num 1144) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2853: read oid 369 snap -1 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2853: expect (ObjNum 1219 snap 0 seq_num 1219) 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2854: copy_from oid 267 from oid 45 current snap is 0 2024-09-21T18:09:32.505 INFO:tasks.rados.rados.0.smithi160.stdout:2855: cache_flush oid 196 snap -1 2024-09-21T18:09:32.506 INFO:tasks.rados.rados.0.smithi160.stdout:2856: write (excl) oid 85 current snap is 0 2024-09-21T18:09:32.506 INFO:tasks.rados.rados.0.smithi160.stdout:2856: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-09-21T18:09:32.519 INFO:tasks.rados.rados.0.smithi160.stdout:2856: writing smithi16030472-85 from 578504 to 1326183 tid 1 2024-09-21T18:09:32.532 INFO:tasks.rados.rados.0.smithi160.stdout:2856: writing smithi16030472-85 from 2034272 to 2601047 tid 2 2024-09-21T18:09:32.538 INFO:tasks.rados.rados.0.smithi160.stdout:2856: writing smithi16030472-85 from 3130988 to 3378504 tid 3 2024-09-21T18:09:32.538 INFO:tasks.rados.rados.0.smithi160.stdout:2857: write oid 109 current snap is 0 2024-09-21T18:09:32.538 INFO:tasks.rados.rados.0.smithi160.stdout:2857: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2024-09-21T18:09:32.546 INFO:tasks.rados.rados.0.smithi160.stdout:2857: writing smithi16030472-109 from 595311 to 1018239 tid 1 2024-09-21T18:09:32.555 INFO:tasks.rados.rados.0.smithi160.stdout:2857: writing smithi16030472-109 from 1576401 to 2286834 tid 2 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2857: writing smithi16030472-109 from 3042058 to 3395311 tid 3 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2858: cache_try_flush oid 384 snap -1 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2859: read oid 273 snap -1 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2859: expect (ObjNum 1110 snap 0 seq_num 1110) 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2860: write oid 355 current snap is 0 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2857: finishing write tid 1 to smithi16030472-109 2024-09-21T18:09:32.562 INFO:tasks.rados.rados.0.smithi160.stdout:2860: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-09-21T18:09:32.568 INFO:tasks.rados.rados.0.smithi160.stdout:2860: writing smithi16030472-355 from 612118 to 1110295 tid 1 2024-09-21T18:09:32.575 INFO:tasks.rados.rados.0.smithi160.stdout:2860: writing smithi16030472-355 from 1634883 to 2205327 tid 2 2024-09-21T18:09:32.582 INFO:tasks.rados.rados.0.smithi160.stdout:2860: writing smithi16030472-355 from 2669481 to 3376143 tid 3 2024-09-21T18:09:32.582 INFO:tasks.rados.rados.0.smithi160.stdout:2861: read oid 381 snap -1 2024-09-21T18:09:32.583 INFO:tasks.rados.rados.0.smithi160.stdout:2861: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:32.583 INFO:tasks.rados.rados.0.smithi160.stdout:2862: read oid 478 snap -1 2024-09-21T18:09:32.583 INFO:tasks.rados.rados.0.smithi160.stdout:2862: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:09:32.583 INFO:tasks.rados.rados.0.smithi160.stdout:2863: write (excl) oid 32 current snap is 0 2024-09-21T18:09:32.583 INFO:tasks.rados.rados.0.smithi160.stdout:2863: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-21T18:09:32.589 INFO:tasks.rados.rados.0.smithi160.stdout:2863: writing smithi16030472-32 from 628925 to 1202351 tid 1 2024-09-21T18:09:32.597 INFO:tasks.rados.rados.0.smithi160.stdout:2863: writing smithi16030472-32 from 1693365 to 2407467 tid 2 2024-09-21T18:09:32.602 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:32.599+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:32.603 INFO:tasks.rados.rados.0.smithi160.stdout:2863: writing smithi16030472-32 from 3096904 to 3428925 tid 3 2024-09-21T18:09:32.603 INFO:tasks.rados.rados.0.smithi160.stdout:2864: delete oid 284 current snap is 0 2024-09-21T18:09:32.612 INFO:tasks.rados.rados.0.smithi160.stdout:2852: expect (ObjNum 1144 snap 0 seq_num 1144) 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 139.44328331200001, 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.658 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.661 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.662 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.664 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.899345763, 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:09:32.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:09:32.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:09:32.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:09:32.669 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.73842229900001, 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.670 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.673 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.737747704, 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.674 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:09:32.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:09:32.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.737721166, 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.678 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.679 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.680 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:09:32.681 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.727887424, 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.682 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.683 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.71547563599999, 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.686 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:32.687 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:09:32.688 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:09:32.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:09:32.690 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:09:32.691 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.706585566, 2024-09-21T18:09:32.692 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.693 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.694 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.696 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.698 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.705773549, 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.699 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.702 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.703 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.705665337, 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.605394191, 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.710 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:32.714 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.604497932, 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:09:32.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:09:32.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.718 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.604016319, 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.719 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.603563894, 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.726 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.603513568, 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:09:32.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:09:32.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:09:32.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.477780788, 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.473502731, 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:09:32.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:09:32.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:09:32.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:09:32.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:09:32.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.473132491, 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.744 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:32.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 121.47306276099999, 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:09:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:09:32.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.753 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 120.881485771, 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:32.754 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:32.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:09:32.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:09:32.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:32.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:09:32.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:09:32.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:09:32.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.760 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.761 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.761 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:32.761 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:32.761 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:32.762 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2072142 2024-09-21T18:09:32.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:32.763 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:32.757+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:32.783 INFO:tasks.rados.rados.0.smithi160.stdout:2853: expect (ObjNum 1219 snap 0 seq_num 1219) 2024-09-21T18:09:32.862 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:32.861+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:32.877 INFO:tasks.rados.rados.0.smithi160.stdout:2851: finishing copy_from to smithi16030472-161 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2851: got expected ENOENT (src dne) 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2856: finishing write tid 1 to smithi16030472-85 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2857: finishing write tid 2 to smithi16030472-109 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2857: finishing write tid 3 to smithi16030472-109 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2850: finishing write tid 1 to smithi16030472-55 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2851: finishing copy_from racing read to smithi16030472-161 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:2862: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:09:32.878 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:32.898 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:32.898 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:32.898 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:32.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:32.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:32.913 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1925551 2024-09-21T18:09:32.942 INFO:tasks.rados.rados.0.smithi160.stdout:2858: got (0) Success 2024-09-21T18:09:32.942 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 384 v 0 (ObjNum 1015 snap 0 seq_num 1015) dirty exists 2024-09-21T18:09:32.942 INFO:tasks.rados.rados.0.smithi160.stdout:2856: finishing write tid 2 to smithi16030472-85 2024-09-21T18:09:32.942 INFO:tasks.rados.rados.0.smithi160.stdout:2856: finishing write tid 3 to smithi16030472-85 2024-09-21T18:09:32.943 INFO:tasks.rados.rados.0.smithi160.stdout:2856: finishing write tid 4 to smithi16030472-85 2024-09-21T18:09:32.943 INFO:tasks.rados.rados.0.smithi160.stdout:2859: expect (ObjNum 1110 snap 0 seq_num 1110) 2024-09-21T18:09:32.965 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:32.960+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:33.016 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1927853 2024-09-21T18:09:33.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:33.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:33.064 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:33.060+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:33.075 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:33.084 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:33.084 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:33.084 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:33.084 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:33.084 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: finishing write tid 4 to smithi16030472-109 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: finishing write tid 5 to smithi16030472-109 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: oid 109 updating version 0 to 840 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: oid 109 updating version 840 to 841 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: oid 109 updating version 841 to 842 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: oid 109 version 842 is already newer than 839 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 842 (ObjNum 1273 snap 0 seq_num 1273) dirty exists 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2857: left oid 109 (ObjNum 1273 snap 0 seq_num 1273) 2024-09-21T18:09:33.085 INFO:tasks.rados.rados.0.smithi160.stdout:2855: got (0) Success 2024-09-21T18:09:33.086 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 196 v 0 (ObjNum 1151 snap 0 seq_num 1151) dirty exists 2024-09-21T18:09:33.086 INFO:tasks.rados.rados.0.smithi160.stdout:2861: expect (ObjNum 4 snap 0 seq_num 4) 2024-09-21T18:09:33.096 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:33.096 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:33.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:33.101 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:33.118 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:33.118 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:33.122 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:33.137 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:33.137 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:33.143 INFO:tasks.rados.rados.0.smithi160.stdout:2860: finishing write tid 1 to smithi16030472-355 2024-09-21T18:09:33.143 INFO:tasks.rados.rados.0.smithi160.stdout:2854: finishing copy_from to smithi16030472-267 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2854: got expected ENOENT (src dne) 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2854: finishing copy_from racing read to smithi16030472-267 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: finishing write tid 5 to smithi16030472-85 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: oid 85 updating version 0 to 877 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: oid 85 version 877 is already newer than 876 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: oid 85 updating version 877 to 878 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: oid 85 updating version 878 to 879 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 879 (ObjNum 1272 snap 0 seq_num 1272) dirty exists 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2856: left oid 85 (ObjNum 1272 snap 0 seq_num 1272) 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2850: finishing write tid 2 to smithi16030472-55 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2850: finishing write tid 3 to smithi16030472-55 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2850: finishing write tid 4 to smithi16030472-55 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2850: finishing write tid 5 to smithi16030472-55 2024-09-21T18:09:33.144 INFO:tasks.rados.rados.0.smithi160.stdout:2850: oid 55 updating version 0 to 916 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2850: oid 55 updating version 916 to 919 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2850: oid 55 version 919 is already newer than 918 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2850: oid 55 version 919 is already newer than 917 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 55 v 919 (ObjNum 1271 snap 0 seq_num 1271) dirty exists 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2850: left oid 55 (ObjNum 1271 snap 0 seq_num 1271) 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2863: finishing write tid 1 to smithi16030472-32 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: finishing write tid 2 to smithi16030472-355 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: finishing write tid 3 to smithi16030472-355 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: finishing write tid 4 to smithi16030472-355 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: finishing write tid 5 to smithi16030472-355 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: oid 355 updating version 0 to 890 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: oid 355 updating version 890 to 892 2024-09-21T18:09:33.145 INFO:tasks.rados.rados.0.smithi160.stdout:2860: oid 355 updating version 892 to 893 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2860: oid 355 version 893 is already newer than 891 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 893 (ObjNum 1274 snap 0 seq_num 1274) dirty exists 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2860: left oid 355 (ObjNum 1274 snap 0 seq_num 1274) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2850: done (14 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2851: done (13 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2852: done (12 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2853: done (11 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2854: done (10 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2855: done (9 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2856: done (8 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2857: done (7 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2858: done (6 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2859: done (5 left) 2024-09-21T18:09:33.146 INFO:tasks.rados.rados.0.smithi160.stdout:2860: done (4 left) 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2861: done (3 left) 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2862: done (2 left) 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2864: done (1 left) 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2865: read oid 199 snap -1 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2865: expect deleted 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2866: read oid 181 snap -1 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2866: expect (ObjNum 1095 snap 0 seq_num 1095) 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2867: cache_evict oid 273 snap -1 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2868: write (excl) oid 200 current snap is 0 2024-09-21T18:09:33.147 INFO:tasks.rados.rados.0.smithi160.stdout:2868: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-21T18:09:33.157 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:33.157 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:33.158 INFO:tasks.rados.rados.0.smithi160.stdout:2868: writing smithi16030472-200 from 662539 to 1386463 tid 1 2024-09-21T18:09:33.165 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:33.163+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:33.173 INFO:tasks.rados.rados.0.smithi160.stdout:2868: writing smithi16030472-200 from 2093976 to 2811747 tid 2 2024-09-21T18:09:33.178 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:33.180 INFO:tasks.rados.rados.0.smithi160.stdout:2868: writing smithi16030472-200 from 3435397 to 3462539 tid 3 2024-09-21T18:09:33.181 INFO:tasks.rados.rados.0.smithi160.stdout:2869: cache_flush oid 89 snap -1 2024-09-21T18:09:33.181 INFO:tasks.rados.rados.0.smithi160.stdout:2867: got (16) Device or resource busy 2024-09-21T18:09:33.181 INFO:tasks.rados.rados.0.smithi160.stdout:2866: expect (ObjNum 1095 snap 0 seq_num 1095) 2024-09-21T18:09:33.195 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:33.195 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:33.216 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:33.216 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:33.220 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.220 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:33.237 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:33.249 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:33.264 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:33.266 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:33.261+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:33.281 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:33.297 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 4929 2024-09-21T18:09:33.302 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:33.324 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:2869: got (0) Success 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 89 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:2863: finishing write tid 2 to smithi16030472-32 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:2863: finishing write tid 3 to smithi16030472-32 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:2865: done (5 left) 2024-09-21T18:09:33.342 INFO:tasks.rados.rados.0.smithi160.stdout:2866: done (4 left) 2024-09-21T18:09:33.343 INFO:tasks.rados.rados.0.smithi160.stdout:2867: done (3 left) 2024-09-21T18:09:33.343 INFO:tasks.rados.rados.0.smithi160.stdout:2869: done (2 left) 2024-09-21T18:09:33.343 INFO:tasks.rados.rados.0.smithi160.stdout:2870: write (excl) oid 211 current snap is 0 2024-09-21T18:09:33.343 INFO:tasks.rados.rados.0.smithi160.stdout:2870: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-21T18:09:33.347 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:33.351 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:33.357 INFO:tasks.rados.rados.0.smithi160.stdout:2870: writing smithi16030472-211 from 679346 to 1194872 tid 1 2024-09-21T18:09:33.368 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:33.364+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:33.372 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.372 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.373 INFO:tasks.rados.rados.0.smithi160.stdout:2870: writing smithi16030472-211 from 1868811 to 2446593 tid 2 2024-09-21T18:09:33.386 INFO:tasks.rados.rados.0.smithi160.stdout:2870: writing smithi16030472-211 from 3179173 to 3479346 tid 3 2024-09-21T18:09:33.386 INFO:tasks.rados.rados.0.smithi160.stdout:2871: read oid 468 snap -1 2024-09-21T18:09:33.386 INFO:tasks.rados.rados.0.smithi160.stdout:2871: expect (ObjNum 467 snap 0 seq_num 467) 2024-09-21T18:09:33.386 INFO:tasks.rados.rados.0.smithi160.stdout:2872: write (excl) oid 409 current snap is 0 2024-09-21T18:09:33.387 INFO:tasks.rados.rados.0.smithi160.stdout:2872: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2024-09-21T18:09:33.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:33.400 INFO:tasks.rados.rados.0.smithi160.stdout:2872: writing smithi16030472-409 from 696153 to 1286928 tid 1 2024-09-21T18:09:33.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:09:33.413 INFO:tasks.rados.rados.0.smithi160.stdout:2872: writing smithi16030472-409 from 1927293 to 2648733 tid 2 2024-09-21T18:09:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:2872: writing smithi16030472-409 from 3206596 to 3496153 tid 3 2024-09-21T18:09:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:2873: copy_from oid 59 from oid 212 current snap is 0 2024-09-21T18:09:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:2863: finishing write tid 4 to smithi16030472-32 2024-09-21T18:09:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:2874: write (excl) oid 123 current snap is 0 2024-09-21T18:09:33.420 INFO:tasks.rados.rados.0.smithi160.stdout:2874: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2024-09-21T18:09:33.429 INFO:tasks.rados.rados.0.smithi160.stdout:2874: writing smithi16030472-123 from 712960 to 1378984 tid 1 2024-09-21T18:09:33.437 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:33.431+0000 7f9bb588f640 -1 osd.10 108 handle_osd_map injecting map crc failure 2024-09-21T18:09:33.438 INFO:tasks.rados.rados.0.smithi160.stdout:2874: writing smithi16030472-123 from 1985775 to 2567226 tid 2 2024-09-21T18:09:33.445 INFO:tasks.rados.rados.0.smithi160.stdout:2874: writing smithi16030472-123 from 3234019 to 3512960 tid 3 2024-09-21T18:09:33.445 INFO:tasks.rados.rados.0.smithi160.stdout:2875: read oid 153 snap -1 2024-09-21T18:09:33.445 INFO:tasks.rados.rados.0.smithi160.stdout:2875: expect deleted 2024-09-21T18:09:33.445 INFO:tasks.rados.rados.0.smithi160.stdout:2876: read oid 412 snap -1 2024-09-21T18:09:33.445 INFO:tasks.rados.rados.0.smithi160.stdout:2876: expect (ObjNum 921 snap 0 seq_num 921) 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2877: delete oid 413 current snap is 0 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: finishing write tid 5 to smithi16030472-32 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: oid 32 updating version 0 to 921 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: oid 32 updating version 921 to 923 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: oid 32 version 923 is already newer than 922 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: oid 32 version 923 is already newer than 920 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 32 v 923 (ObjNum 1275 snap 0 seq_num 1275) dirty exists 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2863: left oid 32 (ObjNum 1275 snap 0 seq_num 1275) 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2868: finishing write tid 1 to smithi16030472-200 2024-09-21T18:09:33.446 INFO:tasks.rados.rados.0.smithi160.stdout:2868: finishing write tid 2 to smithi16030472-200 2024-09-21T18:09:33.469 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:33.464+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:33.541 INFO:tasks.rados.rados.0.smithi160.stdout:2871: expect (ObjNum 467 snap 0 seq_num 467) 2024-09-21T18:09:33.541 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:33.559 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.559 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.569 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:33.561+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:33.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:33.618 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2493899 2024-09-21T18:09:33.631 INFO:tasks.rados.rados.0.smithi160.stdout:2868: finishing write tid 3 to smithi16030472-200 2024-09-21T18:09:33.631 INFO:tasks.rados.rados.0.smithi160.stdout:2870: finishing write tid 1 to smithi16030472-211 2024-09-21T18:09:33.647 INFO:tasks.rados.rados.0.smithi160.stdout:2874: finishing write tid 1 to smithi16030472-123 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: finishing write tid 2 to smithi16030472-123 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: finishing write tid 3 to smithi16030472-123 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: finishing write tid 4 to smithi16030472-123 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: finishing write tid 5 to smithi16030472-123 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: oid 123 updating version 0 to 924 2024-09-21T18:09:33.648 INFO:tasks.rados.rados.0.smithi160.stdout:2874: oid 123 updating version 924 to 925 2024-09-21T18:09:33.649 INFO:tasks.rados.rados.0.smithi160.stdout:2874: oid 123 updating version 925 to 926 2024-09-21T18:09:33.649 INFO:tasks.rados.rados.0.smithi160.stdout:2874: oid 123 updating version 926 to 927 2024-09-21T18:09:33.649 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 927 (ObjNum 1280 snap 0 seq_num 1280) dirty exists 2024-09-21T18:09:33.649 INFO:tasks.rados.rados.0.smithi160.stdout:2874: left oid 123 (ObjNum 1280 snap 0 seq_num 1280) 2024-09-21T18:09:33.660 INFO:tasks.rados.rados.0.smithi160.stdout:2876: expect (ObjNum 921 snap 0 seq_num 921) 2024-09-21T18:09:33.670 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:33.664+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:33.690 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2143107 2024-09-21T18:09:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.712 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:33.758 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:33.771 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:33.767+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2870: finishing write tid 2 to smithi16030472-211 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: finishing write tid 4 to smithi16030472-200 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: finishing write tid 5 to smithi16030472-200 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: oid 200 updating version 0 to 880 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: oid 200 version 880 is already newer than 846 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: oid 200 version 880 is already newer than 848 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: oid 200 version 880 is already newer than 847 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 200 v 880 (ObjNum 1277 snap 0 seq_num 1277) dirty exists 2024-09-21T18:09:33.815 INFO:tasks.rados.rados.0.smithi160.stdout:2868: left oid 200 (ObjNum 1277 snap 0 seq_num 1277) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2863: done (9 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2868: done (8 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2871: done (7 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2874: done (6 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2876: done (5 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2877: done (4 left) 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2878: cache_try_flush oid 387 snap -1 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2879: write (excl) oid 346 current snap is 0 2024-09-21T18:09:33.816 INFO:tasks.rados.rados.0.smithi160.stdout:2879: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-21T18:09:33.826 INFO:tasks.rados.rados.0.smithi160.stdout:2879: writing smithi16030472-346 from 746574 to 1163096 tid 1 2024-09-21T18:09:33.834 INFO:tasks.rados.rados.0.smithi160.stdout:2879: writing smithi16030472-346 from 1586386 to 2055153 tid 2 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2879: writing smithi16030472-346 from 2772512 to 3402056 tid 3 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2880: cache_try_flush oid 327 snap -1 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2881: cache_try_flush oid 367 snap -1 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2882: cache_try_flush oid 86 snap -1 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2883: write (excl) oid 327 current snap is 0 2024-09-21T18:09:33.848 INFO:tasks.rados.rados.0.smithi160.stdout:2883: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-21T18:09:33.850 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.850 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.860 INFO:tasks.rados.rados.0.smithi160.stdout:2883: writing smithi16030472-327 from 763381 to 1255152 tid 1 2024-09-21T18:09:33.865 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:33.871 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:33.865+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:33.875 INFO:tasks.rados.rados.0.smithi160.stdout:2883: writing smithi16030472-327 from 2044868 to 2773646 tid 2 2024-09-21T18:09:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:2883: writing smithi16030472-327 from 3199935 to 3563381 tid 3 2024-09-21T18:09:33.881 INFO:tasks.rados.rados.0.smithi160.stdout:2884: delete oid 402 current snap is 0 2024-09-21T18:09:33.916 INFO:tasks.rados.rados.0.smithi160.stdout:2878: got (0) Success 2024-09-21T18:09:33.916 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 387 v 0 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-21T18:09:33.933 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:33.929+0000 7fb924fb6640 -1 osd.8 108 handle_osd_map injecting map crc failure 2024-09-21T18:09:33.972 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:33.969+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:33.993 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:33.993 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2873: finishing copy_from racing read to smithi16030472-59 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2878: done (10 left) 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2884: done (9 left) 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2885: cache_flush oid 210 snap -1 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2886: read oid 68 snap -1 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2886: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2887: cache_evict oid 251 snap -1 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2888: cache_flush oid 383 snap -1 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2889: copy_from oid 496 from oid 172 current snap is 0 2024-09-21T18:09:33.995 INFO:tasks.rados.rados.0.smithi160.stdout:2890: copy_from oid 183 from oid 76 current snap is 0 2024-09-21T18:09:33.996 INFO:tasks.rados.rados.0.smithi160.stdout:2891: cache_evict oid 174 snap -1 2024-09-21T18:09:33.996 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.008 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:34.057 INFO:tasks.rados.rados.0.smithi160.stdout:2872: finishing write tid 1 to smithi16030472-409 2024-09-21T18:09:34.057 INFO:tasks.rados.rados.0.smithi160.stdout:2872: finishing write tid 2 to smithi16030472-409 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: finishing write tid 3 to smithi16030472-409 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: finishing write tid 4 to smithi16030472-409 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: finishing write tid 5 to smithi16030472-409 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: oid 409 updating version 0 to 844 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: oid 409 updating version 844 to 845 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: oid 409 version 845 is already newer than 843 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: oid 409 updating version 845 to 846 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 409 v 846 (ObjNum 1279 snap 0 seq_num 1279) dirty exists 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: left oid 409 (ObjNum 1279 snap 0 seq_num 1279) 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2888: got (0) Success 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 383 v 0 (ObjNum 1201 snap 0 seq_num 1201) dirty exists 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2872: done (15 left) 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2888: done (14 left) 2024-09-21T18:09:34.058 INFO:tasks.rados.rados.0.smithi160.stdout:2892: cache_try_flush oid 129 snap -1 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout:2893: read oid 224 snap -1 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout:2893: expect deleted 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout:2893: done (15 left) 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout:2894: cache_flush oid 106 snap -1 2024-09-21T18:09:34.059 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.074 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:34.068+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:34.092 INFO:tasks.rados.rados.0.smithi160.stdout:2870: finishing write tid 3 to smithi16030472-211 2024-09-21T18:09:34.092 INFO:tasks.rados.rados.0.smithi160.stdout:2870: finishing write tid 4 to smithi16030472-211 2024-09-21T18:09:34.093 INFO:tasks.rados.rados.0.smithi160.stdout:2870: finishing write tid 5 to smithi16030472-211 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: oid 211 updating version 0 to 850 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: oid 211 version 850 is already newer than 849 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: oid 211 updating version 850 to 883 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: oid 211 version 883 is already newer than 882 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 211 v 883 (ObjNum 1278 snap 0 seq_num 1278) dirty exists 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: left oid 211 (ObjNum 1278 snap 0 seq_num 1278) 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2870: done (15 left) 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout:2895: cache_flush oid 11 snap -1 2024-09-21T18:09:34.094 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.096 INFO:tasks.rados.rados.0.smithi160.stdout:2892: got (0) Success 2024-09-21T18:09:34.096 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:09:34.096 INFO:tasks.rados.rados.0.smithi160.stdout:2892: done (15 left) 2024-09-21T18:09:34.096 INFO:tasks.rados.rados.0.smithi160.stdout:2896: write (excl) oid 14 current snap is 0 2024-09-21T18:09:34.097 INFO:tasks.rados.rados.0.smithi160.stdout:2896: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-21T18:09:34.122 INFO:tasks.rados.rados.0.smithi160.stdout:2896: writing smithi16030472-14 from 796995 to 1555617 tid 1 2024-09-21T18:09:34.135 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.135 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout:2896: writing smithi16030472-14 from 2161832 to 2894279 tid 2 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout:2873: finishing copy_from to smithi16030472-59 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 336 (ObjNum 509 snap 0 seq_num 509) dirty exists 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout:2886: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:34.143 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:34.150 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:34.175 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:34.171+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:34.234 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2414306 2024-09-21T18:09:34.245 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1381955 2024-09-21T18:09:34.258 INFO:tasks.rados.rados.0.smithi160.stdout:2890: finishing copy_from racing read to smithi16030472-183 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2890: finishing copy_from to smithi16030472-183 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 752 (ObjNum 1097 snap 0 seq_num 1097) dirty exists 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2880: got (0) Success 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 327 v 0 (ObjNum 1283 snap 0 seq_num 1283) dirty exists 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2873: done (15 left) 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2875: done (14 left) 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2880: done (13 left) 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2886: done (12 left) 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2890: done (11 left) 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2897: cache_flush oid 36 snap -1 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2898: read oid 284 snap -1 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2898: expect deleted 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2899: write (excl) oid 242 current snap is 0 2024-09-21T18:09:34.259 INFO:tasks.rados.rados.0.smithi160.stdout:2899: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-21T18:09:34.269 INFO:tasks.rados.rados.0.smithi160.stdout:2899: writing smithi16030472-242 from 413802 to 847673 tid 1 2024-09-21T18:09:34.275 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:34.269+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:34.285 INFO:tasks.rados.rados.0.smithi160.stdout:2899: writing smithi16030472-242 from 1420314 to 1896419 tid 2 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2899: writing smithi16030472-242 from 2482204 to 3273189 tid 3 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2900: copy_from oid 495 from oid 347 current snap is 0 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2901: delete oid 264 current snap is 0 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2879: finishing write tid 1 to smithi16030472-346 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2883: finishing write tid 1 to smithi16030472-327 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2887: got (2) No such file or directory 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2891: got (16) Device or resource busy 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2881: got (0) Success 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 0 (ObjNum 968 snap 0 seq_num 968) dirty exists 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:2882: got (0) Success 2024-09-21T18:09:34.308 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 86 v 0 (ObjNum 1139 snap 0 seq_num 1139) dirty exists 2024-09-21T18:09:34.320 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.320 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:34.379 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:34.376+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:34.384 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:09:34.385 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:34.381+0000 7fb260c0e640 -1 osd.11 109 handle_osd_map injecting map crc failure 2024-09-21T18:09:34.414 INFO:tasks.rados.rados.0.smithi160.stdout:2883: finishing write tid 2 to smithi16030472-327 2024-09-21T18:09:34.415 INFO:tasks.rados.rados.0.smithi160.stdout:2879: finishing write tid 2 to smithi16030472-346 2024-09-21T18:09:34.415 INFO:tasks.rados.rados.0.smithi160.stdout:2879: finishing write tid 3 to smithi16030472-346 2024-09-21T18:09:34.415 INFO:tasks.rados.rados.0.smithi160.stdout:2879: finishing write tid 4 to smithi16030472-346 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:2879: finishing write tid 5 to smithi16030472-346 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:2879: oid 346 updating version 0 to 850 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:2879: oid 346 version 850 is already newer than 824 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:2879: oid 346 version 850 is already newer than 849 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:2879: oid 346 updating version 850 to 851 2024-09-21T18:09:34.416 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 346 v 851 (ObjNum 1282 snap 0 seq_num 1282) dirty exists 2024-09-21T18:09:34.417 INFO:tasks.rados.rados.0.smithi160.stdout:2879: left oid 346 (ObjNum 1282 snap 0 seq_num 1282) 2024-09-21T18:09:34.417 INFO:tasks.rados.rados.0.smithi160.stdout:2900: finishing copy_from racing read to smithi16030472-495 2024-09-21T18:09:34.417 INFO:tasks.rados.rados.0.smithi160.stdout:2896: finishing write tid 1 to smithi16030472-14 2024-09-21T18:09:34.421 INFO:tasks.rados.rados.0.smithi160.stdout:2889: finishing copy_from racing read to smithi16030472-496 2024-09-21T18:09:34.421 INFO:tasks.rados.rados.0.smithi160.stdout:2883: finishing write tid 3 to smithi16030472-327 2024-09-21T18:09:34.421 INFO:tasks.rados.rados.0.smithi160.stdout:2883: finishing write tid 4 to smithi16030472-327 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: finishing write tid 5 to smithi16030472-327 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: oid 327 updating version 0 to 853 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: oid 327 version 853 is already newer than 852 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: oid 327 version 853 is already newer than 847 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: oid 327 version 853 is already newer than 848 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 327 v 853 (ObjNum 1283 snap 0 seq_num 1283) dirty exists 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2883: left oid 327 (ObjNum 1283 snap 0 seq_num 1283) 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2899: finishing write tid 1 to smithi16030472-242 2024-09-21T18:09:34.422 INFO:tasks.rados.rados.0.smithi160.stdout:2899: finishing write tid 2 to smithi16030472-242 2024-09-21T18:09:34.460 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.460 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.474 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2879: done (15 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2881: done (14 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2882: done (13 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2883: done (12 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2887: done (11 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2891: done (10 left) 2024-09-21T18:09:34.498 INFO:tasks.rados.rados.0.smithi160.stdout:2898: done (9 left) 2024-09-21T18:09:34.499 INFO:tasks.rados.rados.0.smithi160.stdout:2901: done (8 left) 2024-09-21T18:09:34.499 INFO:tasks.rados.rados.0.smithi160.stdout:2902: read oid 29 snap -1 2024-09-21T18:09:34.499 INFO:tasks.rados.rados.0.smithi160.stdout:2902: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-21T18:09:34.499 INFO:tasks.rados.rados.0.smithi160.stdout:2903: write oid 79 current snap is 0 2024-09-21T18:09:34.499 INFO:tasks.rados.rados.0.smithi160.stdout:2903: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-09-21T18:09:34.510 INFO:tasks.rados.rados.0.smithi160.stdout:2903: writing smithi16030472-79 from 447416 to 1031785 tid 1 2024-09-21T18:09:34.519 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:34.516+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:34.522 INFO:tasks.rados.rados.0.smithi160.stdout:2903: writing smithi16030472-79 from 1537278 to 2017052 tid 2 2024-09-21T18:09:34.534 INFO:tasks.rados.rados.0.smithi160.stdout:2903: writing smithi16030472-79 from 2537050 to 3066932 tid 3 2024-09-21T18:09:34.541 INFO:tasks.rados.rados.0.smithi160.stdout:2903: writing smithi16030472-79 from 3524784 to 3647416 tid 4 2024-09-21T18:09:34.541 INFO:tasks.rados.rados.0.smithi160.stdout:2904: delete oid 490 current snap is 0 2024-09-21T18:09:34.541 INFO:tasks.rados.rados.0.smithi160.stdout:2902: expect (ObjNum 28 snap 0 seq_num 28) 2024-09-21T18:09:34.541 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:34.577 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:34.572+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:34.625 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:34.649 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.649 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.652 INFO:tasks.rados.rados.0.smithi160.stdout:2896: finishing write tid 2 to smithi16030472-14 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: finishing write tid 3 to smithi16030472-14 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: finishing write tid 4 to smithi16030472-14 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: oid 14 updating version 0 to 885 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: oid 14 version 885 is already newer than 884 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: oid 14 updating version 885 to 886 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 886 (ObjNum 1285 snap 0 seq_num 1285) dirty exists 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2896: left oid 14 (ObjNum 1285 snap 0 seq_num 1285) 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2900: finishing copy_from to smithi16030472-495 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 903 (ObjNum 1246 snap 0 seq_num 1246) dirty exists 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2899: finishing write tid 3 to smithi16030472-242 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2899: finishing write tid 4 to smithi16030472-242 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:2895: got (0) Success 2024-09-21T18:09:34.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 11 v 0 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: finishing write tid 5 to smithi16030472-242 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: oid 242 updating version 0 to 857 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: oid 242 version 857 is already newer than 855 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: oid 242 version 857 is already newer than 856 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: oid 242 version 857 is already newer than 854 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 242 v 857 (ObjNum 1286 snap 0 seq_num 1286) dirty exists 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2899: left oid 242 (ObjNum 1286 snap 0 seq_num 1286) 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2885: got (0) Success 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 0 (ObjNum 1037 snap 0 seq_num 1037) dirty exists 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2894: got (0) Success 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 106 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2897: got (0) Success 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 0 (ObjNum 1133 snap 0 seq_num 1133) dirty exists 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:2889: finishing copy_from to smithi16030472-496 2024-09-21T18:09:34.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2885: done (10 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2889: done (9 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2894: done (8 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2895: done (7 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2896: done (6 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2897: done (5 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2899: done (4 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2900: done (3 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2902: done (2 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2904: done (1 left) 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2905: copy_from oid 107 from oid 363 current snap is 0 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2906: write oid 172 current snap is 0 2024-09-21T18:09:34.655 INFO:tasks.rados.rados.0.smithi160.stdout:2906: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-21T18:09:34.663 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:34.670 INFO:tasks.rados.rados.0.smithi160.stdout:2906: writing smithi16030472-172 from 481030 to 1215897 tid 1 2024-09-21T18:09:34.679 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:34.677+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:34.689 INFO:tasks.rados.rados.0.smithi160.stdout:2906: writing smithi16030472-172 from 1937889 to 2421332 tid 2 2024-09-21T18:09:34.702 INFO:tasks.rados.rados.0.smithi160.stdout:2906: writing smithi16030472-172 from 2875543 to 3427969 tid 3 2024-09-21T18:09:34.702 INFO:tasks.rados.rados.0.smithi160.stdout:2907: cache_evict oid 62 snap -1 2024-09-21T18:09:34.703 INFO:tasks.rados.rados.0.smithi160.stdout:2908: write (excl) oid 311 current snap is 0 2024-09-21T18:09:34.703 INFO:tasks.rados.rados.0.smithi160.stdout:2908: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-21T18:09:34.712 INFO:tasks.rados.rados.0.smithi160.stdout:2908: writing smithi16030472-311 from 497837 to 907953 tid 1 2024-09-21T18:09:34.725 INFO:tasks.rados.rados.0.smithi160.stdout:2908: writing smithi16030472-311 from 1596371 to 2339825 tid 2 2024-09-21T18:09:34.736 INFO:tasks.rados.rados.0.smithi160.stdout:2908: writing smithi16030472-311 from 2902966 to 3666664 tid 3 2024-09-21T18:09:34.736 INFO:tasks.rados.rados.0.smithi160.stdout:2909: read oid 275 snap -1 2024-09-21T18:09:34.736 INFO:tasks.rados.rados.0.smithi160.stdout:2909: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2910: cache_evict oid 373 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2907: got (16) Device or resource busy 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 1 to smithi16030472-79 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2907: done (6 left) 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2911: copy_from oid 23 from oid 190 current snap is 0 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2912: cache_evict oid 165 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2913: cache_evict oid 253 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2914: cache_flush oid 380 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2915: read oid 405 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2915: expect (ObjNum 1146 snap 0 seq_num 1146) 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2916: read oid 249 snap -1 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2916: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2917: write (excl) oid 295 current snap is 0 2024-09-21T18:09:34.737 INFO:tasks.rados.rados.0.smithi160.stdout:2917: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-21T18:09:34.744 INFO:tasks.rados.rados.0.smithi160.stdout:2917: writing smithi16030472-295 from 514644 to 1000009 tid 1 2024-09-21T18:09:34.754 INFO:tasks.rados.rados.0.smithi160.stdout:2917: writing smithi16030472-295 from 1654853 to 2141965 tid 2 2024-09-21T18:09:34.766 INFO:tasks.rados.rados.0.smithi160.stdout:2917: writing smithi16030472-295 from 2930389 to 3505359 tid 3 2024-09-21T18:09:34.766 INFO:tasks.rados.rados.0.smithi160.stdout:2918: write (excl) oid 403 current snap is 0 2024-09-21T18:09:34.766 INFO:tasks.rados.rados.0.smithi160.stdout:2918: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-09-21T18:09:34.775 INFO:tasks.rados.rados.0.smithi160.stdout:2918: writing smithi16030472-403 from 531451 to 1208418 tid 1 2024-09-21T18:09:34.780 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:34.775+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:34.784 INFO:tasks.rados.rados.0.smithi160.stdout:2918: writing smithi16030472-403 from 1713335 to 2460458 tid 2 2024-09-21T18:09:34.791 INFO:tasks.rados.rados.0.smithi160.stdout:2918: writing smithi16030472-403 from 2957812 to 3460407 tid 3 2024-09-21T18:09:34.791 INFO:tasks.rados.rados.0.smithi160.stdout:2919: cache_try_flush oid 418 snap -1 2024-09-21T18:09:34.791 INFO:tasks.rados.rados.0.smithi160.stdout:2920: write (excl) oid 455 current snap is 0 2024-09-21T18:09:34.792 INFO:tasks.rados.rados.0.smithi160.stdout:2920: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-09-21T18:09:34.799 INFO:tasks.rados.rados.0.smithi160.stdout:2920: writing smithi16030472-455 from 548258 to 1300474 tid 1 2024-09-21T18:09:34.805 INFO:tasks.rados.rados.0.smithi160.stdout:2920: writing smithi16030472-455 from 1771817 to 2262598 tid 2 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout:2920: writing smithi16030472-455 from 2985235 to 3699102 tid 3 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout:2910: got (16) Device or resource busy 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout:2910: done (15 left) 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout:2921: write oid 237 current snap is 0 2024-09-21T18:09:34.814 INFO:tasks.rados.rados.0.smithi160.stdout:2921: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-09-21T18:09:34.820 INFO:tasks.rados.rados.0.smithi160.stdout:2921: writing smithi16030472-237 from 565065 to 992530 tid 1 2024-09-21T18:09:34.827 INFO:tasks.rados.rados.0.smithi160.stdout:2921: writing smithi16030472-237 from 1430299 to 2064738 tid 2 2024-09-21T18:09:34.832 INFO:tasks.rados.rados.0.smithi160.stdout:2921: writing smithi16030472-237 from 2496305 to 3021444 tid 3 2024-09-21T18:09:34.835 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.835 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.837 INFO:tasks.rados.rados.0.smithi160.stdout:2921: writing smithi16030472-237 from 3472104 to 3765065 tid 4 2024-09-21T18:09:34.837 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:34.850 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:34.870 INFO:tasks.rados.rados.0.smithi160.stdout:2917: finishing write tid 1 to smithi16030472-295 2024-09-21T18:09:34.895 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:34.892+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:34.979 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:34.979 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:34.981 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:34.976+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:34.994 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:35.039 INFO:tasks.rados.rados.0.smithi160.stdout:2906: finishing write tid 1 to smithi16030472-172 2024-09-21T18:09:35.039 INFO:tasks.rados.rados.0.smithi160.stdout:2906: finishing write tid 2 to smithi16030472-172 2024-09-21T18:09:35.040 INFO:tasks.rados.rados.0.smithi160.stdout:2906: finishing write tid 3 to smithi16030472-172 2024-09-21T18:09:35.040 INFO:tasks.rados.rados.0.smithi160.stdout:2906: finishing write tid 4 to smithi16030472-172 2024-09-21T18:09:35.040 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 2 to smithi16030472-79 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: finishing write tid 5 to smithi16030472-172 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: oid 172 updating version 0 to 906 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: oid 172 version 906 is already newer than 904 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: oid 172 version 906 is already newer than 905 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: oid 172 updating version 906 to 907 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 172 v 907 (ObjNum 1290 snap 0 seq_num 1290) dirty exists 2024-09-21T18:09:35.041 INFO:tasks.rados.rados.0.smithi160.stdout:2906: left oid 172 (ObjNum 1290 snap 0 seq_num 1290) 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:2906: done (15 left) 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:2922: read oid 482 snap -1 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:2922: expect deleted 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:2908: finishing write tid 1 to smithi16030472-311 2024-09-21T18:09:35.042 INFO:tasks.rados.rados.0.smithi160.stdout:2908: finishing write tid 2 to smithi16030472-311 2024-09-21T18:09:35.044 INFO:tasks.rados.rados.0.smithi160.stdout:2909: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:09:35.082 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:35.076+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:35.119 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:35.163 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:35.163 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:35.178 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:35.183 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:35.177+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:35.186 INFO:tasks.rados.rados.0.smithi160.stdout:2912: got (2) No such file or directory 2024-09-21T18:09:35.186 INFO:tasks.rados.rados.0.smithi160.stdout:2914: got (0) Success 2024-09-21T18:09:35.186 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 380 v 0 (ObjNum 318 snap 0 seq_num 318) dirty exists 2024-09-21T18:09:35.186 INFO:tasks.rados.rados.0.smithi160.stdout:2915: expect (ObjNum 1146 snap 0 seq_num 1146) 2024-09-21T18:09:35.222 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:35.257 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:35.284 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:35.277+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: finishing write tid 2 to smithi16030472-295 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2919: got (0) Success 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 418 v 0 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: finishing write tid 3 to smithi16030472-295 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: finishing write tid 4 to smithi16030472-295 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: finishing write tid 5 to smithi16030472-295 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: oid 295 updating version 0 to 897 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: oid 295 version 897 is already newer than 896 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: oid 295 updating version 897 to 899 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: oid 295 version 899 is already newer than 898 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 899 (ObjNum 1292 snap 0 seq_num 1292) dirty exists 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2917: left oid 295 (ObjNum 1292 snap 0 seq_num 1292) 2024-09-21T18:09:35.293 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 1 to smithi16030472-237 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2920: finishing write tid 1 to smithi16030472-455 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: finishing write tid 3 to smithi16030472-311 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: finishing write tid 4 to smithi16030472-311 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: finishing write tid 5 to smithi16030472-311 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: oid 311 updating version 0 to 909 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: oid 311 version 909 is already newer than 908 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: oid 311 updating version 909 to 911 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: oid 311 version 911 is already newer than 910 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 311 v 911 (ObjNum 1291 snap 0 seq_num 1291) dirty exists 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2908: left oid 311 (ObjNum 1291 snap 0 seq_num 1291) 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 2 to smithi16030472-237 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 3 to smithi16030472-237 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2920: finishing write tid 2 to smithi16030472-455 2024-09-21T18:09:35.294 INFO:tasks.rados.rados.0.smithi160.stdout:2920: finishing write tid 3 to smithi16030472-455 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: finishing write tid 4 to smithi16030472-455 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: finishing write tid 5 to smithi16030472-455 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: oid 455 updating version 0 to 932 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: oid 455 updating version 932 to 933 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: oid 455 version 933 is already newer than 930 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: oid 455 version 933 is already newer than 931 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 455 v 933 (ObjNum 1294 snap 0 seq_num 1294) dirty exists 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2920: left oid 455 (ObjNum 1294 snap 0 seq_num 1294) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2908: done (15 left) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2909: done (14 left) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2912: done (13 left) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2914: done (12 left) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2915: done (11 left) 2024-09-21T18:09:35.295 INFO:tasks.rados.rados.0.smithi160.stdout:2917: done (10 left) 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2919: done (9 left) 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2920: done (8 left) 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2922: done (7 left) 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2923: cache_evict oid 50 snap -1 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2924: cache_evict oid 321 snap -1 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2925: copy_from oid 339 from oid 492 current snap is 0 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2926: copy_from oid 451 from oid 324 current snap is 0 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2927: cache_try_flush oid 289 snap -1 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2928: copy_from oid 154 from oid 129 current snap is 0 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2929: read oid 467 snap -1 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2929: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:35.296 INFO:tasks.rados.rados.0.smithi160.stdout:2930: delete oid 295 current snap is 0 2024-09-21T18:09:35.348 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:35.348 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:35.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: finishing write tid 1 to smithi16030472-403 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: finishing write tid 2 to smithi16030472-403 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: finishing write tid 3 to smithi16030472-403 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: finishing write tid 4 to smithi16030472-403 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: finishing write tid 5 to smithi16030472-403 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: oid 403 updating version 0 to 913 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: oid 403 version 913 is already newer than 912 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: oid 403 updating version 913 to 914 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: oid 403 updating version 914 to 915 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 403 v 915 (ObjNum 1293 snap 0 seq_num 1293) dirty exists 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2918: left oid 403 (ObjNum 1293 snap 0 seq_num 1293) 2024-09-21T18:09:35.361 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 3 to smithi16030472-79 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 4 to smithi16030472-79 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 5 to smithi16030472-79 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: finishing write tid 6 to smithi16030472-79 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: oid 79 updating version 0 to 862 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: oid 79 version 862 is already newer than 858 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: oid 79 version 862 is already newer than 859 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: oid 79 version 862 is already newer than 860 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: oid 79 version 862 is already newer than 861 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 862 (ObjNum 1288 snap 0 seq_num 1288) dirty exists 2024-09-21T18:09:35.362 INFO:tasks.rados.rados.0.smithi160.stdout:2903: left oid 79 (ObjNum 1288 snap 0 seq_num 1288) 2024-09-21T18:09:35.363 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:35.374 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 4 to smithi16030472-237 2024-09-21T18:09:35.374 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 5 to smithi16030472-237 2024-09-21T18:09:35.374 INFO:tasks.rados.rados.0.smithi160.stdout:2905: finishing copy_from to smithi16030472-107 2024-09-21T18:09:35.374 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 107 v 656 (ObjNum 887 snap 0 seq_num 887) dirty exists 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: finishing write tid 6 to smithi16030472-237 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: oid 237 updating version 0 to 903 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: oid 237 version 903 is already newer than 902 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: oid 237 version 903 is already newer than 900 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: oid 237 updating version 903 to 904 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: oid 237 version 904 is already newer than 901 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 237 v 904 (ObjNum 1295 snap 0 seq_num 1295) dirty exists 2024-09-21T18:09:35.375 INFO:tasks.rados.rados.0.smithi160.stdout:2921: left oid 237 (ObjNum 1295 snap 0 seq_num 1295) 2024-09-21T18:09:35.381 INFO:tasks.rados.rados.0.smithi160.stdout:2905: finishing copy_from racing read to smithi16030472-107 2024-09-21T18:09:35.381 INFO:tasks.rados.rados.0.smithi160.stdout:2911: finishing copy_from to smithi16030472-23 2024-09-21T18:09:35.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 23 v 772 (ObjNum 1150 snap 0 seq_num 1150) dirty exists 2024-09-21T18:09:35.399 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:35.397+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:35.461 INFO:tasks.rados.rados.0.smithi160.stdout:2913: got (0) Success 2024-09-21T18:09:35.475 INFO:tasks.rados.rados.0.smithi160.stdout:2923: got (16) Device or resource busy 2024-09-21T18:09:35.482 INFO:tasks.rados.rados.0.smithi160.stdout:2903: done (14 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2905: done (13 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2913: done (12 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2918: done (11 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2921: done (10 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2923: done (9 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2930: done (8 left) 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2931: write (excl) oid 185 current snap is 0 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2911: finishing copy_from racing read to smithi16030472-23 2024-09-21T18:09:35.483 INFO:tasks.rados.rados.0.smithi160.stdout:2931: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-21T18:09:35.485 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:35.480+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:35.491 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:35.491 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:35.499 INFO:tasks.rados.rados.0.smithi160.stdout:2931: writing smithi16030472-185 from 598679 to 1176642 tid 1 2024-09-21T18:09:35.506 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:35.510 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:35.510 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:35.518 INFO:tasks.rados.rados.0.smithi160.stdout:2931: writing smithi16030472-185 from 1947263 to 2585371 tid 2 2024-09-21T18:09:35.527 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:35.527 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:35.533 INFO:tasks.rados.rados.0.smithi160.stdout:2931: writing smithi16030472-185 from 3351151 to 3798679 tid 3 2024-09-21T18:09:35.533 INFO:tasks.rados.rados.0.smithi160.stdout:2911: done (8 left) 2024-09-21T18:09:35.533 INFO:tasks.rados.rados.0.smithi160.stdout:2932: cache_evict oid 399 snap -1 2024-09-21T18:09:35.533 INFO:tasks.rados.rados.0.smithi160.stdout:2928: finishing copy_from racing read to smithi16030472-154 2024-09-21T18:09:35.534 INFO:tasks.rados.rados.0.smithi160.stdout:2916: expect (ObjNum 590 snap 0 seq_num 590) 2024-09-21T18:09:35.534 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 26865 2024-09-21T18:09:35.548 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:35.548 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:35.570 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:35.587 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:35.582+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:35.587 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:35.587 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:35.606 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:35.607 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:35.628 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:35.628 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:35.628 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:35.634 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:35.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:35.670 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:35.670 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:35.683 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:35.683 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:35.688 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:35.685+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:35.699 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:35.745 INFO:tasks.rados.rados.0.smithi160.stdout:2928: finishing copy_from to smithi16030472-154 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 154 v 488 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2916: done (8 left) 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2928: done (7 left) 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2933: copy_from oid 152 from oid 61 current snap is 0 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2924: got (0) Success 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2932: got (16) Device or resource busy 2024-09-21T18:09:35.746 INFO:tasks.rados.rados.0.smithi160.stdout:2929: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:35.750 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:35.751 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 681591 2024-09-21T18:09:35.789 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:35.782+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:35.832 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:35.832 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:35.832 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:35.832 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:35.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:35.858 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1734108 2024-09-21T18:09:35.877 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:35.877 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:2926: finishing copy_from racing read to smithi16030472-451 2024-09-21T18:09:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:2931: finishing write tid 1 to smithi16030472-185 2024-09-21T18:09:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:2931: finishing write tid 2 to smithi16030472-185 2024-09-21T18:09:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:2931: finishing write tid 3 to smithi16030472-185 2024-09-21T18:09:35.882 INFO:tasks.rados.rados.0.smithi160.stdout:2926: finishing copy_from to smithi16030472-451 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 451 v 79 (ObjNum 119 snap 0 seq_num 119) dirty exists 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2931: finishing write tid 4 to smithi16030472-185 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2924: done (7 left) 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2926: done (6 left) 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2929: done (5 left) 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2932: done (4 left) 2024-09-21T18:09:35.883 INFO:tasks.rados.rados.0.smithi160.stdout:2934: cache_flush oid 464 snap -1 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: finishing write tid 5 to smithi16030472-185 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: oid 185 updating version 0 to 865 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: oid 185 version 865 is already newer than 864 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: oid 185 updating version 865 to 866 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: oid 185 version 866 is already newer than 863 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 185 v 866 (ObjNum 1297 snap 0 seq_num 1297) dirty exists 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2931: left oid 185 (ObjNum 1297 snap 0 seq_num 1297) 2024-09-21T18:09:35.884 INFO:tasks.rados.rados.0.smithi160.stdout:2925: finishing copy_from racing read to smithi16030472-339 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2931: done (4 left) 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2935: read oid 8 snap -1 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2935: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2936: delete oid 185 current snap is 0 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2925: finishing copy_from to smithi16030472-339 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 339 v 323 (ObjNum 491 snap 0 seq_num 491) dirty exists 2024-09-21T18:09:35.885 INFO:tasks.rados.rados.0.smithi160.stdout:2927: got (0) Success 2024-09-21T18:09:35.886 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 0 (ObjNum 1182 snap 0 seq_num 1182) dirty exists 2024-09-21T18:09:35.890 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:35.885+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:35.909 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:35.947 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:35.948 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:35.955 INFO:tasks.rados.rados.0.smithi160.stdout:2935: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:35.969 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:35.991 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:35.988+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:35.992 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:35.992 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:35.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:36.010 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1807825 2024-09-21T18:09:36.012 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.012 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:36.012 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:36.013 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:36.013 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.027 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:36.067 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2418269 2024-09-21T18:09:36.087 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 884922 2024-09-21T18:09:36.092 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:36.085+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:36.115 INFO:tasks.rados.rados.0.smithi160.stdout:2934: got (0) Success 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 464 v 0 (ObjNum 1168 snap 0 seq_num 1168) dirty exists 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:2933: finishing copy_from racing read to smithi16030472-152 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:2933: finishing copy_from to smithi16030472-152 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 152 v 35 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:2925: done (5 left) 2024-09-21T18:09:36.116 INFO:tasks.rados.rados.0.smithi160.stdout:2927: done (4 left) 2024-09-21T18:09:36.117 INFO:tasks.rados.rados.0.smithi160.stdout:2933: done (3 left) 2024-09-21T18:09:36.117 INFO:tasks.rados.rados.0.smithi160.stdout:2934: done (2 left) 2024-09-21T18:09:36.117 INFO:tasks.rados.rados.0.smithi160.stdout:2935: done (1 left) 2024-09-21T18:09:36.117 INFO:tasks.rados.rados.0.smithi160.stdout:2936: done (0 left) 2024-09-21T18:09:36.117 INFO:tasks.rados.rados.0.smithi160.stdout:2937: delete oid 463 current snap is 0 2024-09-21T18:09:36.121 INFO:tasks.rados.rados.0.smithi160.stdout:2937: done (0 left) 2024-09-21T18:09:36.121 INFO:tasks.rados.rados.0.smithi160.stdout:2938: cache_flush oid 309 snap -1 2024-09-21T18:09:36.121 INFO:tasks.rados.rados.0.smithi160.stdout:2939: cache_evict oid 312 snap -1 2024-09-21T18:09:36.121 INFO:tasks.rados.rados.0.smithi160.stdout:2940: read oid 110 snap -1 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2940: expect (ObjNum 109 snap 0 seq_num 109) 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2941: read oid 179 snap -1 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2941: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2942: copy_from oid 308 from oid 293 current snap is 0 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2943: write (excl) oid 143 current snap is 0 2024-09-21T18:09:36.122 INFO:tasks.rados.rados.0.smithi160.stdout:2943: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-21T18:09:36.138 INFO:tasks.rados.rados.0.smithi160.stdout:2943: writing smithi16030472-143 from 649100 to 1052810 tid 1 2024-09-21T18:09:36.155 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.155 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:36.155 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:36.155 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 137.82093478900001, 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.156 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.157 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.158 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.159 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:09:36.160 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.161 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.162 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:09:36.163 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.164 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.98108279300001, 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:09:36.165 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:36.166 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.167 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.168 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.169 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:09:36.170 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.171 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.172 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.87736234499999, 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.173 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.174 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:09:36.175 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:09:36.176 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.177 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.178 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:09:36.179 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.180 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.876587602, 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:09:36.181 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.182 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:36.183 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.184 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:09:36.185 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:09:36.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.87593958299999, 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.187 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.86206788800001, 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:09:36.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.196 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.85912687199999, 2024-09-21T18:09:36.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:36.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:09:36.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:09:36.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:09:36.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:36.205 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.856110024, 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:09:36.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:09:36.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.853898442, 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.213 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:36.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:09:36.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:09:36.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.221 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.74856134999999, 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.222 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:09:36.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:09:36.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.230 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.057138976, 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.231 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:36.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:09:36.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:09:36.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.237 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.05410897600001, 2024-09-21T18:09:36.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 126.087545014, 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.243 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:09:36.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:09:36.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:09:36.247 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 124.420709579, 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:36.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:09:36.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:09:36.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.117218498, 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.11492146499999, 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:09:36.259 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.112871017, 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.110715656, 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:09:36.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.270 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.109952898, 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.271 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:09:36.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:09:36.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:09:36.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.00804785299999, 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:36.280 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:36.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:09:36.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:36.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:09:36.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.288 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:36.289 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:36.289 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:36.294 INFO:tasks.rados.rados.0.smithi160.stdout:2943: writing smithi16030472-143 from 1606356 to 2108144 tid 2 2024-09-21T18:09:36.294 INFO:tasks.rados.rados.0.smithi160.stdout:2943: writing smithi16030472-143 from 2633420 to 3131272 tid 3 2024-09-21T18:09:36.294 INFO:tasks.rados.rados.0.smithi160.stdout:2943: writing smithi16030472-143 from 3636005 to 3849100 tid 4 2024-09-21T18:09:36.294 INFO:tasks.rados.rados.0.smithi160.stdout:2944: read oid 409 snap -1 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2944: expect (ObjNum 1279 snap 0 seq_num 1279) 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2945: write (excl) oid 39 current snap is 0 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2945: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2945: writing smithi16030472-39 from 665907 to 1261219 tid 1 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2945: writing smithi16030472-39 from 1781191 to 2426637 tid 2 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2945: writing smithi16030472-39 from 3177196 to 3865907 tid 3 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2946: cache_evict oid 151 snap -1 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2939: got (16) Device or resource busy 2024-09-21T18:09:36.295 INFO:tasks.rados.rados.0.smithi160.stdout:2939: done (8 left) 2024-09-21T18:09:36.296 INFO:tasks.rados.rados.0.smithi160.stdout:2947: cache_flush oid 65 snap -1 2024-09-21T18:09:36.296 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 1 to smithi16030472-143 2024-09-21T18:09:36.296 INFO:tasks.rados.rados.0.smithi160.stdout:2948: cache_evict oid 168 snap -1 2024-09-21T18:09:36.296 INFO:tasks.rados.rados.0.smithi160.stdout:2949: cache_flush oid 17 snap -1 2024-09-21T18:09:36.296 INFO:tasks.rados.rados.0.smithi160.stdout:2950: delete oid 43 current snap is 0 2024-09-21T18:09:36.296 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:36.285+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:36.331 INFO:tasks.rados.rados.0.smithi160.stdout:2945: finishing write tid 1 to smithi16030472-39 2024-09-21T18:09:36.331 INFO:tasks.rados.rados.0.smithi160.stdout:2945: finishing write tid 2 to smithi16030472-39 2024-09-21T18:09:36.333 INFO:tasks.rados.rados.0.smithi160.stdout:2945: finishing write tid 3 to smithi16030472-39 2024-09-21T18:09:36.333 INFO:tasks.rados.rados.0.smithi160.stdout:2941: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:36.333 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:36.391 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:36.388+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:36.402 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:36.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:09:36.425 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.425 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:36.425 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:36.425 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.439 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:36.458 INFO:tasks.rados.rados.0.smithi160.stdout:2938: got (0) Success 2024-09-21T18:09:36.459 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 309 v 0 (ObjNum 1214 snap 0 seq_num 1214) dirty exists 2024-09-21T18:09:36.459 INFO:tasks.rados.rados.0.smithi160.stdout:2940: expect (ObjNum 109 snap 0 seq_num 109) 2024-09-21T18:09:36.492 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:36.486+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:36.557 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.557 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:36.557 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:36.557 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:36.558 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2946: got (2) No such file or directory 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2948: got (16) Device or resource busy 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2947: got (0) Success 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 65 v 0 (ObjNum 64 snap 0 seq_num 64) dirty exists 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2945: finishing write tid 4 to smithi16030472-39 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2945: finishing write tid 5 to smithi16030472-39 2024-09-21T18:09:36.580 INFO:tasks.rados.rados.0.smithi160.stdout:2945: oid 39 updating version 0 to 935 2024-09-21T18:09:36.581 INFO:tasks.rados.rados.0.smithi160.stdout:2945: oid 39 version 935 is already newer than 934 2024-09-21T18:09:36.581 INFO:tasks.rados.rados.0.smithi160.stdout:2945: oid 39 updating version 935 to 936 2024-09-21T18:09:36.581 INFO:tasks.rados.rados.0.smithi160.stdout:2945: oid 39 updating version 936 to 937 2024-09-21T18:09:36.581 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 937 (ObjNum 1301 snap 0 seq_num 1301) dirty exists 2024-09-21T18:09:36.581 INFO:tasks.rados.rados.0.smithi160.stdout:2945: left oid 39 (ObjNum 1301 snap 0 seq_num 1301) 2024-09-21T18:09:36.593 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:36.588+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:36.674 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 2 to smithi16030472-143 2024-09-21T18:09:36.694 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:36.692+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:36.738 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.738 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:36.738 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:36.738 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:09:36.738 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.752 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:36.774 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 3 to smithi16030472-143 2024-09-21T18:09:36.795 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:36.792+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:36.867 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:09:36.869 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:36.870 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:36.870 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:36.870 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:36.871 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:36.864+0000 7f0b6c216640 -1 osd.9 110 handle_osd_map injecting map crc failure 2024-09-21T18:09:36.875 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 4 to smithi16030472-143 2024-09-21T18:09:36.875 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 5 to smithi16030472-143 2024-09-21T18:09:36.875 INFO:tasks.rados.rados.0.smithi160.stdout:2943: finishing write tid 6 to smithi16030472-143 2024-09-21T18:09:36.875 INFO:tasks.rados.rados.0.smithi160.stdout:2943: oid 143 updating version 0 to 872 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:2943: oid 143 version 872 is already newer than 871 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:2943: oid 143 version 872 is already newer than 870 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:2943: oid 143 version 872 is already newer than 869 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:2943: oid 143 version 872 is already newer than 868 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 143 v 872 (ObjNum 1300 snap 0 seq_num 1300) dirty exists 2024-09-21T18:09:36.876 INFO:tasks.rados.rados.0.smithi160.stdout:2943: left oid 143 (ObjNum 1300 snap 0 seq_num 1300) 2024-09-21T18:09:36.877 INFO:tasks.rados.rados.0.smithi160.stdout:2944: expect (ObjNum 1279 snap 0 seq_num 1279) 2024-09-21T18:09:36.882 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1450559 2024-09-21T18:09:36.885 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:36.916 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 324344 2024-09-21T18:09:36.933 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:36.930+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:36.969 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3012564 2024-09-21T18:09:36.980 INFO:tasks.rados.rados.0.smithi160.stdout:2949: got (0) Success 2024-09-21T18:09:36.980 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 17 v 0 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:09:36.980 INFO:tasks.rados.rados.0.smithi160.stdout:2942: finishing copy_from racing read to smithi16030472-308 2024-09-21T18:09:36.980 INFO:tasks.rados.rados.0.smithi160.stdout:2938: done (11 left) 2024-09-21T18:09:36.980 INFO:tasks.rados.rados.0.smithi160.stdout:2940: done (10 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2941: done (9 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2943: done (8 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2944: done (7 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2945: done (6 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2946: done (5 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2947: done (4 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2948: done (3 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2949: done (2 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2950: done (1 left) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2951: cache_evict oid 390 snap -1 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2952: read oid 272 snap -1 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2952: expect (ObjNum 568 snap 0 seq_num 568) 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2953: read oid 26 snap -1 2024-09-21T18:09:36.981 INFO:tasks.rados.rados.0.smithi160.stdout:2953: expect (ObjNum 881 snap 0 seq_num 881) 2024-09-21T18:09:36.982 INFO:tasks.rados.rados.0.smithi160.stdout:2954: copy_from oid 187 from oid 14 current snap is 0 2024-09-21T18:09:36.982 INFO:tasks.rados.rados.0.smithi160.stdout:2955: write (excl) oid 89 current snap is 0 2024-09-21T18:09:36.982 INFO:tasks.rados.rados.0.smithi160.stdout:2955: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-21T18:09:36.990 INFO:tasks.rados.rados.0.smithi160.stdout:2955: writing smithi16030472-89 from 699521 to 1445331 tid 1 2024-09-21T18:09:36.996 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:36.992+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:37.002 INFO:tasks.rados.rados.0.smithi160.stdout:2955: writing smithi16030472-89 from 2181802 to 2830917 tid 2 2024-09-21T18:09:37.005 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:37.005 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:37.005 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:37.005 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:37.005 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:37.014 INFO:tasks.rados.rados.0.smithi160.stdout:2955: writing smithi16030472-89 from 3515689 to 3899521 tid 3 2024-09-21T18:09:37.014 INFO:tasks.rados.rados.0.smithi160.stdout:2956: read oid 454 snap -1 2024-09-21T18:09:37.014 INFO:tasks.rados.rados.0.smithi160.stdout:2956: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:09:37.014 INFO:tasks.rados.rados.0.smithi160.stdout:2957: cache_evict oid 488 snap -1 2024-09-21T18:09:37.015 INFO:tasks.rados.rados.0.smithi160.stdout:2958: cache_evict oid 91 snap -1 2024-09-21T18:09:37.015 INFO:tasks.rados.rados.0.smithi160.stdout:2952: expect (ObjNum 568 snap 0 seq_num 568) 2024-09-21T18:09:37.019 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:37.110 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:37.106+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:37.148 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.185 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.587924934, 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.186 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.187 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.188 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.189 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.157954987, 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:37.190 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:09:37.191 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.192 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.06043461199999, 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:09:37.193 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.194 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:09:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:09:37.196 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.059750362, 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.197 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:09:37.198 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.199 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.200 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.05891700700001, 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.201 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.202 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.203 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.204 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.056198377, 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.205 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.206 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.207 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.208 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:37.209 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.960389971, 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.210 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.211 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.212 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 102.95808543, 2024-09-21T18:09:37.213 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.214 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.215 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.216 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:37.217 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.980456810000007, 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.218 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:09:37.219 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:09:37.220 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:37.221 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.879301127999994, 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.222 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.223 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.224 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:37.225 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.878505466000007, 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.226 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:09:37.227 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:09:37.229 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.877369646000005, 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.856652053000005, 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:09:37.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:09:37.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.847299368999998, 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.841924720999998, 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.242 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:09:37.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:09:37.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 83.839376446000003, 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.246 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:09:37.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:09:37.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 77.611675908999999, 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.250 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:09:37.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:09:37.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:09:37.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 77.611009616000004, 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:09:37.254 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:37.258 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 77.610917504, 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:37.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:09:37.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:09:37.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.262 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:6894 4.2 e96/33 4:60b99da3:::smithi16030472-227:head v 96'728, mlcod=96'726)", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 21.340277060999998, 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.303226251, 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:37.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843452+0000", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0429999999999999e-06 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844181+0000", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072923900000000002 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844209+0000", 2024-09-21T18:09:37.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7313000000000001e-05 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944352+0000", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100143339 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944508+0000", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015568600000000001 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944543+0000", 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5649999999999999e-05 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146570+0000", 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20202621900000001 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146675+0000", 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010576199999999999 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.266 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:2951: got (2) No such file or directory 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:2951: done (8 left) 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:2952: done (7 left) 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:2959: cache_evict oid 439 snap -1 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:2953: expect (ObjNum 881 snap 0 seq_num 881) 2024-09-21T18:09:37.268 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:37.269 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:37.264+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:37.305 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.357 INFO:tasks.rados.rados.0.smithi160.stdout:2958: got (0) Success 2024-09-21T18:09:37.357 INFO:tasks.rados.rados.0.smithi160.stdout:2942: finishing copy_from to smithi16030472-308 2024-09-21T18:09:37.357 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 308 v 395 (ObjNum 523 snap 0 seq_num 523) dirty exists 2024-09-21T18:09:37.357 INFO:tasks.rados.rados.0.smithi160.stdout:2955: finishing write tid 1 to smithi16030472-89 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2959: got (2) No such file or directory 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2954: finishing copy_from racing read to smithi16030472-187 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2954: finishing copy_from to smithi16030472-187 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 886 (ObjNum 1285 snap 0 seq_num 1285) dirty exists 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2955: finishing write tid 2 to smithi16030472-89 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2955: finishing write tid 3 to smithi16030472-89 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2942: done (7 left) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2953: done (6 left) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2954: done (5 left) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2958: done (4 left) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2959: done (3 left) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2960: read oid 115 snap -1 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2960: expect (ObjNum 895 snap 0 seq_num 895) 2024-09-21T18:09:37.358 INFO:tasks.rados.rados.0.smithi160.stdout:2961: write (excl) oid 459 current snap is 0 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: finishing write tid 4 to smithi16030472-89 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: finishing write tid 5 to smithi16030472-89 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: oid 89 updating version 0 to 938 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: oid 89 updating version 938 to 941 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: oid 89 version 941 is already newer than 939 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: oid 89 version 941 is already newer than 940 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 89 v 941 (ObjNum 1303 snap 0 seq_num 1303) dirty exists 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2955: left oid 89 (ObjNum 1303 snap 0 seq_num 1303) 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2957: got (0) Success 2024-09-21T18:09:37.359 INFO:tasks.rados.rados.0.smithi160.stdout:2961: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2024-09-21T18:09:37.363 INFO:tasks.rados.rados.0.smithi160.stdout:2961: writing smithi16030472-459 from 716328 to 1137387 tid 1 2024-09-21T18:09:37.369 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:37.368+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:37.370 INFO:tasks.rados.rados.0.smithi160.stdout:2961: writing smithi16030472-459 from 1840284 to 2349410 tid 2 2024-09-21T18:09:37.379 INFO:tasks.rados.rados.0.smithi160.stdout:2961: writing smithi16030472-459 from 3143112 to 3802405 tid 3 2024-09-21T18:09:37.379 INFO:tasks.rados.rados.0.smithi160.stdout:2955: done (4 left) 2024-09-21T18:09:37.379 INFO:tasks.rados.rados.0.smithi160.stdout:2957: done (3 left) 2024-09-21T18:09:37.379 INFO:tasks.rados.rados.0.smithi160.stdout:2962: delete oid 369 current snap is 0 2024-09-21T18:09:37.379 INFO:tasks.rados.rados.0.smithi160.stdout:2960: expect (ObjNum 895 snap 0 seq_num 895) 2024-09-21T18:09:37.437 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:37.438 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:37.438 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:37.438 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:37.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:37.473 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.498 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:37.494+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:37.526 INFO:tasks.rados.rados.0.smithi160.stdout:2961: finishing write tid 1 to smithi16030472-459 2024-09-21T18:09:37.526 INFO:tasks.rados.rados.0.smithi160.stdout:2961: finishing write tid 2 to smithi16030472-459 2024-09-21T18:09:37.526 INFO:tasks.rados.rados.0.smithi160.stdout:2961: finishing write tid 3 to smithi16030472-459 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: finishing write tid 4 to smithi16030472-459 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: finishing write tid 5 to smithi16030472-459 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: oid 459 updating version 0 to 908 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: oid 459 version 908 is already newer than 907 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: oid 459 version 908 is already newer than 906 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: oid 459 updating version 908 to 909 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 909 (ObjNum 1304 snap 0 seq_num 1304) dirty exists 2024-09-21T18:09:37.527 INFO:tasks.rados.rados.0.smithi160.stdout:2961: left oid 459 (ObjNum 1304 snap 0 seq_num 1304) 2024-09-21T18:09:37.572 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:37.566+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:37.575 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:37.576 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:37.576 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:37.576 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:37.576 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:37.583 INFO:tasks.rados.rados.0.smithi160.stdout:2960: done (3 left) 2024-09-21T18:09:37.583 INFO:tasks.rados.rados.0.smithi160.stdout:2961: done (2 left) 2024-09-21T18:09:37.584 INFO:tasks.rados.rados.0.smithi160.stdout:2962: done (1 left) 2024-09-21T18:09:37.584 INFO:tasks.rados.rados.0.smithi160.stdout:2963: cache_try_flush oid 418 snap -1 2024-09-21T18:09:37.584 INFO:tasks.rados.rados.0.smithi160.stdout:2964: delete oid 391 current snap is 0 2024-09-21T18:09:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:2963: got (0) Success 2024-09-21T18:09:37.585 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 418 v 0 (ObjNum 417 snap 0 seq_num 417) dirty exists 2024-09-21T18:09:37.591 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:37.605 INFO:tasks.rados.rados.0.smithi160.stdout:2956: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:09:37.605 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.661 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2704725 2024-09-21T18:09:37.675 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:37.672+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:37.713 INFO:tasks.rados.rados.0.smithi160.stdout:2956: done (2 left) 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2963: done (1 left) 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2964: done (0 left) 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2965: cache_flush oid 138 snap -1 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2966: read oid 97 snap -1 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2966: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2967: write (excl) oid 168 current snap is 0 2024-09-21T18:09:37.714 INFO:tasks.rados.rados.0.smithi160.stdout:2967: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-21T18:09:37.721 INFO:tasks.rados.rados.0.smithi160.stdout:2967: writing smithi16030472-168 from 766749 to 1413555 tid 1 2024-09-21T18:09:37.726 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:37.722+0000 7fca4e93e640 -1 osd.2 110 handle_osd_map injecting map crc failure 2024-09-21T18:09:37.728 INFO:tasks.rados.rados.0.smithi160.stdout:2967: writing smithi16030472-168 from 1899377 to 2555830 tid 2 2024-09-21T18:09:37.734 INFO:tasks.rados.rados.0.smithi160.stdout:2967: writing smithi16030472-168 from 3109028 to 3718490 tid 3 2024-09-21T18:09:37.734 INFO:tasks.rados.rados.0.smithi160.stdout:2968: write oid 306 current snap is 0 2024-09-21T18:09:37.735 INFO:tasks.rados.rados.0.smithi160.stdout:2968: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-21T18:09:37.741 INFO:tasks.rados.rados.0.smithi160.stdout:2968: writing smithi16030472-306 from 783556 to 1221964 tid 1 2024-09-21T18:09:37.746 INFO:tasks.rados.rados.0.smithi160.stdout:2968: writing smithi16030472-306 from 1674212 to 2074323 tid 2 2024-09-21T18:09:37.752 INFO:tasks.rados.rados.0.smithi160.stdout:2968: writing smithi16030472-306 from 2736451 to 3157185 tid 3 2024-09-21T18:09:37.757 INFO:tasks.rados.rados.0.smithi160.stdout:2968: writing smithi16030472-306 from 3882270 to 3983556 tid 4 2024-09-21T18:09:37.757 INFO:tasks.rados.rados.0.smithi160.stdout:2969: read oid 243 snap -1 2024-09-21T18:09:37.757 INFO:tasks.rados.rados.0.smithi160.stdout:2969: expect (ObjNum 1227 snap 0 seq_num 1227) 2024-09-21T18:09:37.757 INFO:tasks.rados.rados.0.smithi160.stdout:2970: cache_flush oid 465 snap -1 2024-09-21T18:09:37.757 INFO:tasks.rados.rados.0.smithi160.stdout:2971: delete oid 45 current snap is 0 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 138.91783156, 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.759 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.760 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.761 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.762 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:09:37.763 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.50618926600001, 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:37.764 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.765 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.766 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.767 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.503358394, 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.768 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.769 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.770 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.771 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.49938548599999, 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.772 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.773 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.774 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.775 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:37.776 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.49626853500001, 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.777 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:09:37.778 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.779 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.780 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.49433753100001, 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.781 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.782 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.783 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.784 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.785 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.49123077199999, 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.786 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:09:37.787 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:09:37.788 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.789 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.790 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.48807308799999, 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.791 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.792 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:09:37.793 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.794 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.795 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.485840037, 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.796 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.797 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:09:37.798 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.799 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.800 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.48358836400001, 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.801 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.802 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.803 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.804 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.805 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.391010776, 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.806 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.807 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:09:37.808 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.809 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.810 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.6143414, 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.811 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.812 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:09:37.813 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.814 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.815 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.604833816, 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.816 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:09:37.817 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.818 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:09:37.819 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.820 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.498003837, 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:37.821 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.822 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.823 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.824 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.825 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 118.164632285, 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.826 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:09:37.827 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.828 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:09:37.829 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.830 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 113.392640816, 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.831 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:09:37.832 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:09:37.833 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.834 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.835 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 103.510492361, 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:09:37.836 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:37.837 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.838 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:09:37.839 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.840 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 78.970367913000004, 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.841 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.842 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.843 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 61.660409453, 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.846 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:09:37.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 58.416007141999998, 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:09:37.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:09:37.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:37.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:37.855 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:37.856 INFO:tasks.rados.rados.0.smithi160.stdout:2966: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:37.857 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 517182 2024-09-21T18:09:37.857 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:37.858 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:37.852+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:37.920 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:37.916+0000 7f440780b640 -1 osd.13 110 handle_osd_map injecting map crc failure 2024-09-21T18:09:37.957 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:37.949+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:37.976 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:37.976 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:37.976 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:37.977 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:37.984 INFO:tasks.rados.rados.0.smithi160.stdout:2965: got (0) Success 2024-09-21T18:09:37.984 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 138 v 0 (ObjNum 1016 snap 0 seq_num 1016) dirty exists 2024-09-21T18:09:37.984 INFO:tasks.rados.rados.0.smithi160.stdout:2969: expect (ObjNum 1227 snap 0 seq_num 1227) 2024-09-21T18:09:37.984 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:37.990 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:37.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:38.058 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:38.053+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:38.069 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1647582 2024-09-21T18:09:38.091 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2967: finishing write tid 1 to smithi16030472-168 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2965: done (6 left) 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2966: done (5 left) 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2969: done (4 left) 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2971: done (3 left) 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2972: cache_flush oid 494 snap -1 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 1 to smithi16030472-306 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 2 to smithi16030472-306 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 3 to smithi16030472-306 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2967: finishing write tid 2 to smithi16030472-168 2024-09-21T18:09:38.148 INFO:tasks.rados.rados.0.smithi160.stdout:2967: finishing write tid 3 to smithi16030472-168 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 4 to smithi16030472-306 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 5 to smithi16030472-306 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: finishing write tid 6 to smithi16030472-306 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: oid 306 updating version 0 to 915 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: oid 306 version 915 is already newer than 911 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: oid 306 version 915 is already newer than 913 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: oid 306 version 915 is already newer than 912 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: oid 306 version 915 is already newer than 914 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 306 v 915 (ObjNum 1308 snap 0 seq_num 1308) dirty exists 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: left oid 306 (ObjNum 1308 snap 0 seq_num 1308) 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2970: got (0) Success 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 465 v 0 (ObjNum 1203 snap 0 seq_num 1203) dirty exists 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2967: finishing write tid 4 to smithi16030472-168 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2968: done (3 left) 2024-09-21T18:09:38.149 INFO:tasks.rados.rados.0.smithi160.stdout:2970: done (2 left) 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2973: delete oid 158 current snap is 0 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: finishing write tid 5 to smithi16030472-168 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: oid 168 updating version 0 to 916 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: oid 168 updating version 916 to 919 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: oid 168 version 919 is already newer than 918 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: oid 168 version 919 is already newer than 917 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 168 v 919 (ObjNum 1307 snap 0 seq_num 1307) dirty exists 2024-09-21T18:09:38.150 INFO:tasks.rados.rados.0.smithi160.stdout:2967: left oid 168 (ObjNum 1307 snap 0 seq_num 1307) 2024-09-21T18:09:38.152 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:38.152 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:38.152 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:38.152 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:38.152 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:38.158 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:38.157+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:38.159 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:38.159 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:38.166 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:38.180 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:38.180 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:38.200 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:38.201 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:38.218 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:38.219 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:38.236 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:38.236 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:38.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:38.257 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:38.259 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:38.257+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:38.268 INFO:tasks.rados.rados.0.smithi160.stdout:2972: got (0) Success 2024-09-21T18:09:38.268 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 0 (ObjNum 1142 snap 0 seq_num 1142) dirty exists 2024-09-21T18:09:38.277 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:38.278 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:38.278 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:38.296 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:38.318 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:38.318 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:38.332 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:38.333 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:38.340 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:38.341 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:38.346 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:38.362 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:38.362 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:38.383 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:38.393 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:38.389+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:38.406 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:38.406 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:38.431 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:38.431 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:38.451 INFO:tasks.rados.rados.0.smithi160.stdout:2967: done (2 left) 2024-09-21T18:09:38.451 INFO:tasks.rados.rados.0.smithi160.stdout:2972: done (1 left) 2024-09-21T18:09:38.451 INFO:tasks.rados.rados.0.smithi160.stdout:2973: done (0 left) 2024-09-21T18:09:38.451 INFO:tasks.rados.rados.0.smithi160.stdout:2974: delete oid 449 current snap is 0 2024-09-21T18:09:38.454 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:38.455 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:38.455 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:38.460 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:38.457+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:38.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:38.461 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:38.468 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:38.556 INFO:tasks.rados.rados.0.smithi160.stdout:2974: done (0 left) 2024-09-21T18:09:38.556 INFO:tasks.rados.rados.0.smithi160.stdout:2975: write oid 63 current snap is 0 2024-09-21T18:09:38.556 INFO:tasks.rados.rados.0.smithi160.stdout:2975: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-21T18:09:38.562 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:38.558+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:38.565 INFO:tasks.rados.rados.0.smithi160.stdout:2975: writing smithi16030472-63 from 450784 to 1190188 tid 1 2024-09-21T18:09:38.572 INFO:tasks.rados.rados.0.smithi160.stdout:2975: writing smithi16030472-63 from 1791787 to 2050784 tid 2 2024-09-21T18:09:38.572 INFO:tasks.rados.rados.0.smithi160.stdout:2976: write oid 258 current snap is 0 2024-09-21T18:09:38.572 INFO:tasks.rados.rados.0.smithi160.stdout:2976: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-21T18:09:38.579 INFO:tasks.rados.rados.0.smithi160.stdout:2976: writing smithi16030472-258 from 467591 to 882244 tid 1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2976: writing smithi16030472-258 from 1450269 to 2067591 tid 2 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2977: cache_try_flush oid 210 snap -1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2978: copy_from oid 30 from oid 477 current snap is 0 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2979: cache_try_flush oid 116 snap -1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2980: read oid 182 snap -1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2980: expect (ObjNum 971 snap 0 seq_num 971) 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2981: read oid 108 snap -1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2981: expect (ObjNum 1045 snap 0 seq_num 1045) 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2982: cache_flush oid 249 snap -1 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2983: delete oid 417 current snap is 0 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:2977: got (0) Success 2024-09-21T18:09:38.589 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 0 (ObjNum 1037 snap 0 seq_num 1037) dirty exists 2024-09-21T18:09:38.631 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:38.631 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:38.646 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:38.680 INFO:tasks.rados.rados.0.smithi160.stdout:2979: got (0) Success 2024-09-21T18:09:38.680 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 116 v 0 (ObjNum 869 snap 0 seq_num 869) dirty exists 2024-09-21T18:09:38.680 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:38.673+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:38.690 INFO:tasks.rados.rados.0.smithi160.stdout:2978: finishing copy_from to smithi16030472-30 2024-09-21T18:09:38.690 INFO:tasks.rados.rados.0.smithi160.stdout:2978: got expected ENOENT (src dne) 2024-09-21T18:09:38.694 INFO:tasks.rados.rados.0.smithi160.stdout:2980: expect (ObjNum 971 snap 0 seq_num 971) 2024-09-21T18:09:38.694 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:38.764 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:38.760+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:38.764 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:38.764 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:38.778 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:38.799 INFO:tasks.rados.rados.0.smithi160.stdout:2981: expect (ObjNum 1045 snap 0 seq_num 1045) 2024-09-21T18:09:38.811 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2263023 2024-09-21T18:09:38.865 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:38.860+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:38.886 INFO:tasks.rados.rados.0.smithi160.stdout:2982: got (0) Success 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 249 v 0 (ObjNum 590 snap 0 seq_num 590) dirty exists 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: finishing write tid 1 to smithi16030472-63 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: finishing write tid 2 to smithi16030472-63 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2978: finishing copy_from racing read to smithi16030472-30 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: finishing write tid 3 to smithi16030472-63 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: finishing write tid 4 to smithi16030472-63 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: oid 63 updating version 0 to 922 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: oid 63 version 922 is already newer than 920 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: oid 63 version 922 is already newer than 921 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 63 v 922 (ObjNum 1312 snap 0 seq_num 1312) dirty exists 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: left oid 63 (ObjNum 1312 snap 0 seq_num 1312) 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2975: done (8 left) 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2977: done (7 left) 2024-09-21T18:09:38.887 INFO:tasks.rados.rados.0.smithi160.stdout:2978: done (6 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2979: done (5 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2980: done (4 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2981: done (3 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2982: done (2 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2983: done (1 left) 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2984: cache_evict oid 93 snap -1 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2985: cache_evict oid 456 snap -1 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2986: cache_flush oid 154 snap -1 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2987: cache_try_flush oid 336 snap -1 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2988: cache_evict oid 76 snap -1 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2989: copy_from oid 94 from oid 193 current snap is 0 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2990: write oid 149 current snap is 0 2024-09-21T18:09:38.888 INFO:tasks.rados.rados.0.smithi160.stdout:2990: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-21T18:09:38.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:09:38.903 INFO:tasks.rados.rados.0.smithi160.stdout:2990: writing smithi16030472-149 from 501205 to 1066356 tid 1 2024-09-21T18:09:38.915 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:38.915 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:38.920 INFO:tasks.rados.rados.0.smithi160.stdout:2990: writing smithi16030472-149 from 1850880 to 2101205 tid 2 2024-09-21T18:09:38.920 INFO:tasks.rados.rados.0.smithi160.stdout:2991: write oid 290 current snap is 0 2024-09-21T18:09:38.920 INFO:tasks.rados.rados.0.smithi160.stdout:2991: seq_num 1316 ranges {518012=756753,2025715=92297} 2024-09-21T18:09:38.928 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:38.935 INFO:tasks.rados.rados.0.smithi160.stdout:2991: writing smithi16030472-290 from 518012 to 1274765 tid 1 2024-09-21T18:09:38.943 INFO:tasks.rados.rados.0.smithi160.stdout:2991: writing smithi16030472-290 from 2025715 to 2118012 tid 2 2024-09-21T18:09:38.943 INFO:tasks.rados.rados.0.smithi160.stdout:2992: write oid 305 current snap is 0 2024-09-21T18:09:38.943 INFO:tasks.rados.rados.0.smithi160.stdout:2992: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-21T18:09:38.951 INFO:tasks.rados.rados.0.smithi160.stdout:2992: writing smithi16030472-305 from 534819 to 966821 tid 1 2024-09-21T18:09:38.962 INFO:tasks.rados.rados.0.smithi160.stdout:2992: writing smithi16030472-305 from 1684197 to 2134819 tid 2 2024-09-21T18:09:38.962 INFO:tasks.rados.rados.0.smithi160.stdout:2993: cache_flush oid 234 snap -1 2024-09-21T18:09:38.962 INFO:tasks.rados.rados.0.smithi160.stdout:2994: copy_from oid 113 from oid 142 current snap is 0 2024-09-21T18:09:38.962 INFO:tasks.rados.rados.0.smithi160.stdout:2995: read oid 92 snap -1 2024-09-21T18:09:38.962 INFO:tasks.rados.rados.0.smithi160.stdout:2995: expect (ObjNum 718 snap 0 seq_num 718) 2024-09-21T18:09:38.963 INFO:tasks.rados.rados.0.smithi160.stdout:2996: cache_try_flush oid 192 snap -1 2024-09-21T18:09:38.963 INFO:tasks.rados.rados.0.smithi160.stdout:2997: copy_from oid 456 from oid 27 current snap is 0 2024-09-21T18:09:38.963 INFO:tasks.rados.rados.0.smithi160.stdout:2998: read oid 346 snap -1 2024-09-21T18:09:38.963 INFO:tasks.rados.rados.0.smithi160.stdout:2998: expect (ObjNum 1282 snap 0 seq_num 1282) 2024-09-21T18:09:38.963 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:38.966 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:38.960+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: finishing write tid 1 to smithi16030472-258 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: finishing write tid 2 to smithi16030472-258 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: finishing write tid 3 to smithi16030472-258 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: finishing write tid 4 to smithi16030472-258 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: oid 258 updating version 0 to 923 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: oid 258 updating version 923 to 925 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: oid 258 version 925 is already newer than 924 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 258 v 925 (ObjNum 1313 snap 0 seq_num 1313) dirty exists 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: left oid 258 (ObjNum 1313 snap 0 seq_num 1313) 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2985: got (16) Device or resource busy 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2987: got (0) Success 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 0 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2988: got (16) Device or resource busy 2024-09-21T18:09:38.989 INFO:tasks.rados.rados.0.smithi160.stdout:2976: done (15 left) 2024-09-21T18:09:38.990 INFO:tasks.rados.rados.0.smithi160.stdout:2985: done (14 left) 2024-09-21T18:09:38.990 INFO:tasks.rados.rados.0.smithi160.stdout:2987: done (13 left) 2024-09-21T18:09:38.990 INFO:tasks.rados.rados.0.smithi160.stdout:2988: done (12 left) 2024-09-21T18:09:38.990 INFO:tasks.rados.rados.0.smithi160.stdout:2999: delete oid 111 current snap is 0 2024-09-21T18:09:39.002 INFO:tasks.rados.rados.0.smithi160.stdout:2994: finishing copy_from racing read to smithi16030472-113 2024-09-21T18:09:39.067 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:39.065+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:39.069 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.069 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.083 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:39.089 INFO:tasks.rados.rados.0.smithi160.stdout:2984: got (16) Device or resource busy 2024-09-21T18:09:39.091 INFO:tasks.rados.rados.0.smithi160.stdout:2995: expect (ObjNum 718 snap 0 seq_num 718) 2024-09-21T18:09:39.168 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:39.165+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:39.186 INFO:tasks.rados.rados.0.smithi160.stdout:2993: got (0) Success 2024-09-21T18:09:39.186 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 234 v 0 (ObjNum -1154728096 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:39.187 INFO:tasks.rados.rados.0.smithi160.stdout:2990: finishing write tid 1 to smithi16030472-149 2024-09-21T18:09:39.187 INFO:tasks.rados.rados.0.smithi160.stdout:2991: finishing write tid 1 to smithi16030472-290 2024-09-21T18:09:39.187 INFO:tasks.rados.rados.0.smithi160.stdout:2991: finishing write tid 2 to smithi16030472-290 2024-09-21T18:09:39.187 INFO:tasks.rados.rados.0.smithi160.stdout:2998: expect (ObjNum 1282 snap 0 seq_num 1282) 2024-09-21T18:09:39.213 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.213 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:39.245 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:39.269 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:39.268+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2997: finishing copy_from racing read to smithi16030472-456 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: finishing write tid 3 to smithi16030472-290 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: finishing write tid 4 to smithi16030472-290 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: oid 290 updating version 0 to 928 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: oid 290 version 928 is already newer than 926 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: oid 290 version 928 is already newer than 927 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 290 v 928 (ObjNum 1316 snap 0 seq_num 1316) dirty exists 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: left oid 290 (ObjNum 1316 snap 0 seq_num 1316) 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2992: finishing write tid 1 to smithi16030472-305 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2989: finishing copy_from racing read to smithi16030472-94 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2984: done (12 left) 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2991: done (11 left) 2024-09-21T18:09:39.284 INFO:tasks.rados.rados.0.smithi160.stdout:2993: done (10 left) 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2995: done (9 left) 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2998: done (8 left) 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2999: done (7 left) 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:3000: delete oid 466 current snap is 0 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: finishing write tid 2 to smithi16030472-149 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: finishing write tid 3 to smithi16030472-149 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: finishing write tid 4 to smithi16030472-149 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: oid 149 updating version 0 to 922 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: oid 149 version 922 is already newer than 920 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: oid 149 version 922 is already newer than 921 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 922 (ObjNum 1315 snap 0 seq_num 1315) dirty exists 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2990: left oid 149 (ObjNum 1315 snap 0 seq_num 1315) 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2992: finishing write tid 2 to smithi16030472-305 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2992: finishing write tid 3 to smithi16030472-305 2024-09-21T18:09:39.285 INFO:tasks.rados.rados.0.smithi160.stdout:2994: finishing copy_from to smithi16030472-113 2024-09-21T18:09:39.286 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 113 v 537 (ObjNum 750 snap 0 seq_num 750) dirty exists 2024-09-21T18:09:39.288 INFO:tasks.rados.rados.0.smithi160.stdout:2990: done (7 left) 2024-09-21T18:09:39.288 INFO:tasks.rados.rados.0.smithi160.stdout:2994: done (6 left) 2024-09-21T18:09:39.289 INFO:tasks.rados.rados.0.smithi160.stdout:3000: done (5 left) 2024-09-21T18:09:39.289 INFO:tasks.rados.rados.0.smithi160.stdout:3001: cache_try_flush oid 195 snap -1 2024-09-21T18:09:39.289 INFO:tasks.rados.rados.0.smithi160.stdout:3002: write (excl) oid 396 current snap is 0 2024-09-21T18:09:39.289 INFO:tasks.rados.rados.0.smithi160.stdout:3002: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-21T18:09:39.297 INFO:tasks.rados.rados.0.smithi160.stdout:3002: writing smithi16030472-396 from 585240 to 1242989 tid 1 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:3002: writing smithi16030472-396 from 1743290 to 2165415 tid 2 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:3003: cache_try_flush oid 155 snap -1 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:3004: cache_try_flush oid 78 snap -1 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:3005: delete oid 325 current snap is 0 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:3003: got (0) Success 2024-09-21T18:09:39.305 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 155 v 0 (ObjNum 129 snap 0 seq_num 3126308200) clean dne 2024-09-21T18:09:39.348 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.348 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:39.370 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:39.364+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:39.390 INFO:tasks.rados.rados.0.smithi160.stdout:3004: got (0) Success 2024-09-21T18:09:39.390 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 0 (ObjNum 1102 snap 0 seq_num 1102) dirty exists 2024-09-21T18:09:39.405 INFO:tasks.rados.rados.0.smithi160.stdout:2992: finishing write tid 4 to smithi16030472-305 2024-09-21T18:09:39.405 INFO:tasks.rados.rados.0.smithi160.stdout:2992: oid 305 updating version 0 to 875 2024-09-21T18:09:39.405 INFO:tasks.rados.rados.0.smithi160.stdout:2992: oid 305 version 875 is already newer than 873 2024-09-21T18:09:39.405 INFO:tasks.rados.rados.0.smithi160.stdout:2992: oid 305 version 875 is already newer than 874 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 305 v 875 (ObjNum 1317 snap 0 seq_num 1317) dirty exists 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:2992: left oid 305 (ObjNum 1317 snap 0 seq_num 1317) 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:2997: finishing copy_from to smithi16030472-456 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 832 (ObjNum 1186 snap 0 seq_num 1186) dirty exists 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:2989: finishing copy_from to smithi16030472-94 2024-09-21T18:09:39.406 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 94 v 576 (ObjNum 845 snap 0 seq_num 845) dirty exists 2024-09-21T18:09:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:2986: got (0) Success 2024-09-21T18:09:39.407 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 154 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:09:39.414 INFO:tasks.rados.rados.0.smithi160.stdout:2996: got (0) Success 2024-09-21T18:09:39.414 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 192 v 0 (ObjNum 970 snap 0 seq_num 970) dirty exists 2024-09-21T18:09:39.471 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:39.465+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:39.487 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.487 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:39.572 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:39.569+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:2986: done (9 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:2989: done (8 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:2992: done (7 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:2996: done (6 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:2997: done (5 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:3003: done (4 left) 2024-09-21T18:09:39.610 INFO:tasks.rados.rados.0.smithi160.stdout:3004: done (3 left) 2024-09-21T18:09:39.611 INFO:tasks.rados.rados.0.smithi160.stdout:3005: done (2 left) 2024-09-21T18:09:39.611 INFO:tasks.rados.rados.0.smithi160.stdout:3006: copy_from oid 285 from oid 378 current snap is 0 2024-09-21T18:09:39.611 INFO:tasks.rados.rados.0.smithi160.stdout:3007: cache_flush oid 157 snap -1 2024-09-21T18:09:39.611 INFO:tasks.rados.rados.0.smithi160.stdout:3008: write oid 478 current snap is 0 2024-09-21T18:09:39.611 INFO:tasks.rados.rados.0.smithi160.stdout:3008: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-21T18:09:39.626 INFO:tasks.rados.rados.0.smithi160.stdout:3008: writing smithi16030472-478 from 618854 to 1027101 tid 1 2024-09-21T18:09:39.633 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.633 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.644 INFO:tasks.rados.rados.0.smithi160.stdout:3008: writing smithi16030472-478 from 1743901 to 2169695 tid 2 2024-09-21T18:09:39.644 INFO:tasks.rados.rados.0.smithi160.stdout:3009: delete oid 322 current snap is 0 2024-09-21T18:09:39.644 INFO:tasks.rados.rados.0.smithi160.stdout:3008: finishing write tid 1 to smithi16030472-478 2024-09-21T18:09:39.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:39.652 INFO:tasks.rados.rados.0.smithi160.stdout:3008: finishing write tid 2 to smithi16030472-478 2024-09-21T18:09:39.695 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:39.693+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:39.719 INFO:tasks.rados.rados.0.smithi160.stdout:3002: finishing write tid 1 to smithi16030472-396 2024-09-21T18:09:39.719 INFO:tasks.rados.rados.0.smithi160.stdout:3002: finishing write tid 2 to smithi16030472-396 2024-09-21T18:09:39.720 INFO:tasks.rados.rados.0.smithi160.stdout:3002: finishing write tid 3 to smithi16030472-396 2024-09-21T18:09:39.745 INFO:tasks.rados.rados.0.smithi160.stdout:3008: finishing write tid 3 to smithi16030472-478 2024-09-21T18:09:39.773 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:39.770+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:39.775 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.775 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:3002: finishing write tid 4 to smithi16030472-396 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:3002: oid 396 updating version 0 to 876 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:3002: oid 396 updating version 876 to 878 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:3002: oid 396 version 878 is already newer than 877 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 396 v 878 (ObjNum 1320 snap 0 seq_num 1320) dirty exists 2024-09-21T18:09:39.821 INFO:tasks.rados.rados.0.smithi160.stdout:3002: left oid 396 (ObjNum 1320 snap 0 seq_num 1320) 2024-09-21T18:09:39.828 INFO:tasks.rados.rados.0.smithi160.stdout:3006: finishing copy_from racing read to smithi16030472-285 2024-09-21T18:09:39.829 INFO:tasks.rados.rados.0.smithi160.stdout:3006: finishing copy_from to smithi16030472-285 2024-09-21T18:09:39.829 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 285 v 557 (ObjNum 733 snap 0 seq_num 733) dirty exists 2024-09-21T18:09:39.829 INFO:tasks.rados.rados.0.smithi160.stdout:3001: got (0) Success 2024-09-21T18:09:39.829 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 0 (ObjNum 1048 snap 0 seq_num 1048) dirty exists 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:3008: finishing write tid 4 to smithi16030472-478 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:3008: oid 478 updating version 0 to 931 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:3008: oid 478 version 931 is already newer than 930 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:3008: oid 478 version 931 is already newer than 852 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 478 v 931 (ObjNum 1322 snap 0 seq_num 1322) dirty exists 2024-09-21T18:09:39.846 INFO:tasks.rados.rados.0.smithi160.stdout:3008: left oid 478 (ObjNum 1322 snap 0 seq_num 1322) 2024-09-21T18:09:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:09:39.874 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:39.869+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:39.921 INFO:tasks.rados.rados.0.smithi160.stdout:3001: done (5 left) 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3002: done (4 left) 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3006: done (3 left) 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3008: done (2 left) 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3009: done (1 left) 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3010: cache_evict oid 199 snap -1 2024-09-21T18:09:39.922 INFO:tasks.rados.rados.0.smithi160.stdout:3011: delete oid 212 current snap is 0 2024-09-21T18:09:39.932 INFO:tasks.rados.rados.0.smithi160.stdout:3007: got (0) Success 2024-09-21T18:09:39.932 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 0 (ObjNum 946 snap 0 seq_num 946) dirty exists 2024-09-21T18:09:39.968 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:39.968 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:39.975 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:39.972+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:39.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:40.076 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:40.072+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:40.110 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.110 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3010: got (0) Success 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3007: done (2 left) 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3010: done (1 left) 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3011: done (0 left) 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3012: write oid 384 current snap is 0 2024-09-21T18:09:40.127 INFO:tasks.rados.rados.0.smithi160.stdout:3012: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-21T18:09:40.150 INFO:tasks.rados.rados.0.smithi160.stdout:3012: writing smithi16030472-384 from 669275 to 1419622 tid 1 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3012: writing smithi16030472-384 from 2035700 to 2269275 tid 2 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3013: cache_evict oid 148 snap -1 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3014: cache_evict oid 139 snap -1 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3015: cache_try_flush oid 59 snap -1 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3016: write (excl) oid 49 current snap is 0 2024-09-21T18:09:40.164 INFO:tasks.rados.rados.0.smithi160.stdout:3016: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-21T18:09:40.177 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:40.169+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:40.181 INFO:tasks.rados.rados.0.smithi160.stdout:3016: writing smithi16030472-49 from 686082 to 1111678 tid 1 2024-09-21T18:09:40.191 INFO:tasks.rados.rados.0.smithi160.stdout:3016: writing smithi16030472-49 from 1577829 to 2010961 tid 2 2024-09-21T18:09:40.191 INFO:tasks.rados.rados.0.smithi160.stdout:3017: write (excl) oid 102 current snap is 0 2024-09-21T18:09:40.191 INFO:tasks.rados.rados.0.smithi160.stdout:3017: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-21T18:09:40.203 INFO:tasks.rados.rados.0.smithi160.stdout:3017: writing smithi16030472-102 from 702889 to 1203734 tid 1 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3017: writing smithi16030472-102 from 1636311 to 2213101 tid 2 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3018: read oid 463 snap -1 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3018: expect deleted 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3019: read oid 492 snap -1 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3019: expect (ObjNum 491 snap 0 seq_num 491) 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3020: cache_flush oid 50 snap -1 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3013: got (0) Success 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3013: done (8 left) 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3021: write (excl) oid 331 current snap is 0 2024-09-21T18:09:40.213 INFO:tasks.rados.rados.0.smithi160.stdout:3021: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-21T18:09:40.223 INFO:tasks.rados.rados.0.smithi160.stdout:3021: writing smithi16030472-331 from 719696 to 1295790 tid 1 2024-09-21T18:09:40.231 INFO:tasks.rados.rados.0.smithi160.stdout:3021: writing smithi16030472-331 from 2094793 to 2319696 tid 2 2024-09-21T18:09:40.231 INFO:tasks.rados.rados.0.smithi160.stdout:3022: read oid 421 snap -1 2024-09-21T18:09:40.232 INFO:tasks.rados.rados.0.smithi160.stdout:3022: expect (ObjNum 1119 snap 0 seq_num 1119) 2024-09-21T18:09:40.232 INFO:tasks.rados.rados.0.smithi160.stdout:3023: cache_evict oid 70 snap -1 2024-09-21T18:09:40.232 INFO:tasks.rados.rados.0.smithi160.stdout:3024: write oid 474 current snap is 0 2024-09-21T18:09:40.232 INFO:tasks.rados.rados.0.smithi160.stdout:3024: seq_num 1329 ranges {736503=651343,2153275=183228} 2024-09-21T18:09:40.242 INFO:tasks.rados.rados.0.smithi160.stdout:3024: writing smithi16030472-474 from 736503 to 1387846 tid 1 2024-09-21T18:09:40.250 INFO:tasks.rados.rados.0.smithi160.stdout:3024: writing smithi16030472-474 from 2153275 to 2336503 tid 2 2024-09-21T18:09:40.250 INFO:tasks.rados.rados.0.smithi160.stdout:3025: write oid 298 current snap is 0 2024-09-21T18:09:40.250 INFO:tasks.rados.rados.0.smithi160.stdout:3025: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-21T18:09:40.253 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.254 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.261 INFO:tasks.rados.rados.0.smithi160.stdout:3025: writing smithi16030472-298 from 753310 to 1479902 tid 1 2024-09-21T18:09:40.267 INFO:tasks.rados.rados.0.smithi160.stdout:3025: writing smithi16030472-298 from 2095404 to 2353310 tid 2 2024-09-21T18:09:40.267 INFO:tasks.rados.rados.0.smithi160.stdout:3026: write oid 180 current snap is 0 2024-09-21T18:09:40.267 INFO:tasks.rados.rados.0.smithi160.stdout:3026: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-21T18:09:40.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:40.275 INFO:tasks.rados.rados.0.smithi160.stdout:3026: writing smithi16030472-180 from 770117 to 1288311 tid 1 2024-09-21T18:09:40.278 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:40.272+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3026: writing smithi16030472-180 from 1870239 to 2370117 tid 2 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3027: cache_flush oid 195 snap -1 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3028: read oid 413 snap -1 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3028: expect deleted 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3014: got (0) Success 2024-09-21T18:09:40.282 INFO:tasks.rados.rados.0.smithi160.stdout:3014: done (15 left) 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:3029: cache_flush oid 358 snap -1 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:3027: got (0) Success 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 195 v 0 (ObjNum 1048 snap 0 seq_num 1048) dirty exists 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:3027: done (15 left) 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:3030: read oid 61 snap -1 2024-09-21T18:09:40.283 INFO:tasks.rados.rados.0.smithi160.stdout:3030: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-21T18:09:40.284 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:40.284 INFO:tasks.rados.rados.0.smithi160.stdout:3029: got (0) Success 2024-09-21T18:09:40.284 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 358 v 0 (ObjNum 357 snap 0 seq_num 357) dirty exists 2024-09-21T18:09:40.284 INFO:tasks.rados.rados.0.smithi160.stdout:3029: done (15 left) 2024-09-21T18:09:40.284 INFO:tasks.rados.rados.0.smithi160.stdout:3031: delete oid 254 current snap is 0 2024-09-21T18:09:40.379 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:40.374+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:40.390 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.390 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:40.437 INFO:tasks.rados.rados.0.smithi160.stdout:3020: got (0) Success 2024-09-21T18:09:40.437 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 50 v 0 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-21T18:09:40.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:40.479 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:40.476+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: finishing write tid 1 to smithi16030472-384 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: finishing write tid 2 to smithi16030472-384 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: finishing write tid 3 to smithi16030472-384 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: finishing write tid 4 to smithi16030472-384 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: oid 384 updating version 0 to 948 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: oid 384 updating version 948 to 949 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: oid 384 version 949 is already newer than 947 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 384 v 949 (ObjNum 1325 snap 0 seq_num 1325) dirty exists 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3012: left oid 384 (ObjNum 1325 snap 0 seq_num 1325) 2024-09-21T18:09:40.486 INFO:tasks.rados.rados.0.smithi160.stdout:3016: finishing write tid 1 to smithi16030472-49 2024-09-21T18:09:40.487 INFO:tasks.rados.rados.0.smithi160.stdout:3016: finishing write tid 2 to smithi16030472-49 2024-09-21T18:09:40.487 INFO:tasks.rados.rados.0.smithi160.stdout:3016: finishing write tid 3 to smithi16030472-49 2024-09-21T18:09:40.515 INFO:tasks.rados.rados.0.smithi160.stdout:3017: finishing write tid 1 to smithi16030472-102 2024-09-21T18:09:40.520 INFO:tasks.rados.rados.0.smithi160.stdout:3023: got (16) Device or resource busy 2024-09-21T18:09:40.526 INFO:tasks.rados.rados.0.smithi160.stdout:3022: expect (ObjNum 1119 snap 0 seq_num 1119) 2024-09-21T18:09:40.534 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.534 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.540 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 834561 2024-09-21T18:09:40.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:40.580 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:40.576+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:40.603 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:40.603 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:40.624 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:40.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:40.634 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:40.645 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:40.645 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:3017: finishing write tid 2 to smithi16030472-102 2024-09-21T18:09:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:3017: finishing write tid 3 to smithi16030472-102 2024-09-21T18:09:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:3017: finishing write tid 4 to smithi16030472-102 2024-09-21T18:09:40.665 INFO:tasks.rados.rados.0.smithi160.stdout:3017: oid 102 updating version 0 to 879 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3017: oid 102 updating version 879 to 881 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3017: oid 102 version 881 is already newer than 880 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 102 v 881 (ObjNum 1327 snap 0 seq_num 1327) dirty exists 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3017: left oid 102 (ObjNum 1327 snap 0 seq_num 1327) 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3016: finishing write tid 4 to smithi16030472-49 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3016: oid 49 updating version 0 to 950 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3016: oid 49 updating version 950 to 952 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3016: oid 49 version 952 is already newer than 951 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 49 v 952 (ObjNum 1326 snap 0 seq_num 1326) dirty exists 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3016: left oid 49 (ObjNum 1326 snap 0 seq_num 1326) 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:3019: expect (ObjNum 491 snap 0 seq_num 491) 2024-09-21T18:09:40.666 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:40.667 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:40.667 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:40.681 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:40.678+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:40.683 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.683 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:40.713 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 503719 2024-09-21T18:09:40.782 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:40.776+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:40.785 INFO:tasks.rados.rados.0.smithi160.stdout:3021: finishing write tid 1 to smithi16030472-331 2024-09-21T18:09:40.787 INFO:tasks.rados.rados.0.smithi160.stdout:3015: got (0) Success 2024-09-21T18:09:40.787 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 59 v 0 (ObjNum 509 snap 0 seq_num 509) dirty exists 2024-09-21T18:09:40.792 INFO:tasks.rados.rados.0.smithi160.stdout:3021: finishing write tid 2 to smithi16030472-331 2024-09-21T18:09:40.792 INFO:tasks.rados.rados.0.smithi160.stdout:3021: finishing write tid 3 to smithi16030472-331 2024-09-21T18:09:40.823 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:40.823 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:40.832 INFO:tasks.rados.rados.0.smithi160.stdout:3024: finishing write tid 1 to smithi16030472-474 2024-09-21T18:09:40.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:40.864 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:40.864 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:40.883 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:40.877+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3021: finishing write tid 4 to smithi16030472-331 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3021: oid 331 updating version 0 to 954 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3021: oid 331 updating version 954 to 955 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3021: oid 331 version 955 is already newer than 953 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 331 v 955 (ObjNum 1328 snap 0 seq_num 1328) dirty exists 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3021: left oid 331 (ObjNum 1328 snap 0 seq_num 1328) 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3026: finishing write tid 1 to smithi16030472-180 2024-09-21T18:09:40.893 INFO:tasks.rados.rados.0.smithi160.stdout:3026: finishing write tid 2 to smithi16030472-180 2024-09-21T18:09:40.900 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:40.900 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:40.936 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:40.937 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:40.937 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3012: done (15 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3015: done (14 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3016: done (13 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3017: done (12 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3018: done (11 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3019: done (10 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3020: done (9 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3021: done (8 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3022: done (7 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3023: done (6 left) 2024-09-21T18:09:40.947 INFO:tasks.rados.rados.0.smithi160.stdout:3028: done (5 left) 2024-09-21T18:09:40.948 INFO:tasks.rados.rados.0.smithi160.stdout:3031: done (4 left) 2024-09-21T18:09:40.948 INFO:tasks.rados.rados.0.smithi160.stdout:3032: write (excl) oid 125 current snap is 0 2024-09-21T18:09:40.948 INFO:tasks.rados.rados.0.smithi160.stdout:3032: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-21T18:09:40.954 INFO:tasks.rados.rados.0.smithi160.stdout:3032: writing smithi16030472-125 from 403731 to 1072423 tid 1 2024-09-21T18:09:40.961 INFO:tasks.rados.rados.0.smithi160.stdout:3032: writing smithi16030472-125 from 1587203 to 2175000 tid 2 2024-09-21T18:09:40.961 INFO:tasks.rados.rados.0.smithi160.stdout:3033: copy_from oid 407 from oid 155 current snap is 0 2024-09-21T18:09:40.961 INFO:tasks.rados.rados.0.smithi160.stdout:3024: finishing write tid 2 to smithi16030472-474 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3034: cache_evict oid 337 snap -1 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3035: read oid 68 snap -1 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3035: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3036: cache_flush oid 222 snap -1 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3037: cache_evict oid 65 snap -1 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3038: cache_flush oid 373 snap -1 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3039: write oid 293 current snap is 0 2024-09-21T18:09:40.962 INFO:tasks.rados.rados.0.smithi160.stdout:3039: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-21T18:09:40.969 INFO:tasks.rados.rados.0.smithi160.stdout:3039: writing smithi16030472-293 from 420538 to 1164479 tid 1 2024-09-21T18:09:40.971 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:40.971 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:40.977 INFO:tasks.rados.rados.0.smithi160.stdout:3039: writing smithi16030472-293 from 1929332 to 2420538 tid 2 2024-09-21T18:09:40.977 INFO:tasks.rados.rados.0.smithi160.stdout:3040: write (excl) oid 289 current snap is 0 2024-09-21T18:09:40.977 INFO:tasks.rados.rados.0.smithi160.stdout:3040: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-21T18:09:40.983 INFO:tasks.rados.rados.0.smithi160.stdout:3040: writing smithi16030472-289 from 437345 to 856535 tid 1 2024-09-21T18:09:40.984 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:40.978+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:40.991 INFO:tasks.rados.rados.0.smithi160.stdout:3040: writing smithi16030472-289 from 1587814 to 2179280 tid 2 2024-09-21T18:09:40.992 INFO:tasks.rados.rados.0.smithi160.stdout:3041: cache_try_flush oid 200 snap -1 2024-09-21T18:09:40.992 INFO:tasks.rados.rados.0.smithi160.stdout:3042: delete oid 484 current snap is 0 2024-09-21T18:09:40.992 INFO:tasks.rados.rados.0.smithi160.stdout:3034: got (16) Device or resource busy 2024-09-21T18:09:41.006 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:41.006 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:41.007 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:41.025 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:41.040 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:41.040 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:41.066 INFO:tasks.rados.rados.0.smithi160.stdout:3037: got (0) Success 2024-09-21T18:09:41.084 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:41.077+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: finishing write tid 3 to smithi16030472-180 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: finishing write tid 4 to smithi16030472-180 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: oid 180 updating version 0 to 958 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: oid 180 version 958 is already newer than 956 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: oid 180 version 958 is already newer than 957 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 958 (ObjNum 1331 snap 0 seq_num 1331) dirty exists 2024-09-21T18:09:41.085 INFO:tasks.rados.rados.0.smithi160.stdout:3026: left oid 180 (ObjNum 1331 snap 0 seq_num 1331) 2024-09-21T18:09:41.106 INFO:tasks.rados.rados.0.smithi160.stdout:3035: expect (ObjNum 1101 snap 0 seq_num 1101) 2024-09-21T18:09:41.106 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:971 3.0 3.d67074d4 (undecoded) ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e112)", 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:41.062193+0000", 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.096107764999999998, 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.096129821000000004, 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.159 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:41.062193+0000", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:41.062193+0000", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:41.062198+0000", 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7620000000000003e-06 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.160 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:41.062214+0000", 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5631e-05 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:41.062240+0000", 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6296999999999999e-05 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-09-21T18:09:41.161 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:41.174 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:41.179 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:41.194 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:41.194 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:41.215 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:41.218 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:41.219 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:41.216+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:41.242 INFO:tasks.rados.rados.0.smithi160.stdout:3040: finishing write tid 1 to smithi16030472-289 2024-09-21T18:09:41.242 INFO:tasks.rados.rados.0.smithi160.stdout:3024: finishing write tid 3 to smithi16030472-474 2024-09-21T18:09:41.242 INFO:tasks.rados.rados.0.smithi160.stdout:3024: finishing write tid 4 to smithi16030472-474 2024-09-21T18:09:41.242 INFO:tasks.rados.rados.0.smithi160.stdout:3024: oid 474 updating version 0 to 934 2024-09-21T18:09:41.242 INFO:tasks.rados.rados.0.smithi160.stdout:3024: oid 474 version 934 is already newer than 933 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3024: oid 474 version 934 is already newer than 932 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 474 v 934 (ObjNum 1329 snap 0 seq_num 1329) dirty exists 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3024: left oid 474 (ObjNum 1329 snap 0 seq_num 1329) 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: finishing write tid 1 to smithi16030472-298 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: finishing write tid 2 to smithi16030472-298 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: finishing write tid 3 to smithi16030472-298 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: finishing write tid 4 to smithi16030472-298 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: oid 298 updating version 0 to 935 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: oid 298 updating version 935 to 937 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: oid 298 version 937 is already newer than 936 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 298 v 937 (ObjNum 1330 snap 0 seq_num 1330) dirty exists 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3025: left oid 298 (ObjNum 1330 snap 0 seq_num 1330) 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3032: finishing write tid 1 to smithi16030472-125 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3032: finishing write tid 2 to smithi16030472-125 2024-09-21T18:09:41.243 INFO:tasks.rados.rados.0.smithi160.stdout:3030: expect (ObjNum 60 snap 0 seq_num 60) 2024-09-21T18:09:41.244 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:41.244 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:41.275 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:41.285 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:41.281+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:41.293 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:41.293 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:41.293 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:41.293 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:41.294 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:41.308 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:41.323 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:41.323 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:41.338 INFO:tasks.rados.rados.0.smithi160.stdout:3040: finishing write tid 2 to smithi16030472-289 2024-09-21T18:09:41.338 INFO:tasks.rados.rados.0.smithi160.stdout:3032: finishing write tid 3 to smithi16030472-125 2024-09-21T18:09:41.338 INFO:tasks.rados.rados.0.smithi160.stdout:3032: finishing write tid 4 to smithi16030472-125 2024-09-21T18:09:41.338 INFO:tasks.rados.rados.0.smithi160.stdout:3032: oid 125 updating version 0 to 939 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3032: oid 125 version 939 is already newer than 938 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3032: oid 125 updating version 939 to 940 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 125 v 940 (ObjNum 1333 snap 0 seq_num 1333) dirty exists 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3032: left oid 125 (ObjNum 1333 snap 0 seq_num 1333) 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: finishing write tid 3 to smithi16030472-289 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: finishing write tid 4 to smithi16030472-289 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: oid 289 updating version 0 to 959 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: oid 289 updating version 959 to 960 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: oid 289 updating version 960 to 961 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 961 (ObjNum 1335 snap 0 seq_num 1335) dirty exists 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3040: left oid 289 (ObjNum 1335 snap 0 seq_num 1335) 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3033: finishing copy_from to smithi16030472-407 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3033: got expected ENOENT (src dne) 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3024: done (14 left) 2024-09-21T18:09:41.339 INFO:tasks.rados.rados.0.smithi160.stdout:3025: done (13 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3026: done (12 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3030: done (11 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3032: done (10 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3034: done (9 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3035: done (8 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3037: done (7 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3040: done (6 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3042: done (5 left) 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3043: write (excl) oid 426 current snap is 0 2024-09-21T18:09:41.340 INFO:tasks.rados.rados.0.smithi160.stdout:3043: seq_num 1337 ranges {470959=569688,1704778=595135} 2024-09-21T18:09:41.345 INFO:tasks.rados.rados.0.smithi160.stdout:3043: writing smithi16030472-426 from 470959 to 1040647 tid 1 2024-09-21T18:09:41.350 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:41.350 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3043: writing smithi16030472-426 from 1704778 to 2299913 tid 2 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3044: cache_flush oid 355 snap -1 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3045: read oid 398 snap -1 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3045: expect (ObjNum 1269 snap 0 seq_num 1269) 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3046: read oid 259 snap -1 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3046: expect deleted 2024-09-21T18:09:41.354 INFO:tasks.rados.rados.0.smithi160.stdout:3047: cache_evict oid 437 snap -1 2024-09-21T18:09:41.355 INFO:tasks.rados.rados.0.smithi160.stdout:3048: delete oid 127 current snap is 0 2024-09-21T18:09:41.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:41.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:41.364 INFO:tasks.rados.rados.0.smithi160.stdout:3039: finishing write tid 1 to smithi16030472-293 2024-09-21T18:09:41.364 INFO:tasks.rados.rados.0.smithi160.stdout:3039: finishing write tid 2 to smithi16030472-293 2024-09-21T18:09:41.364 INFO:tasks.rados.rados.0.smithi160.stdout:3039: finishing write tid 3 to smithi16030472-293 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:3039: finishing write tid 4 to smithi16030472-293 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:3039: oid 293 updating version 0 to 883 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:3039: oid 293 version 883 is already newer than 882 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:3039: oid 293 updating version 883 to 884 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 884 (ObjNum 1334 snap 0 seq_num 1334) dirty exists 2024-09-21T18:09:41.365 INFO:tasks.rados.rados.0.smithi160.stdout:3039: left oid 293 (ObjNum 1334 snap 0 seq_num 1334) 2024-09-21T18:09:41.373 INFO:tasks.rados.rados.0.smithi160.stdout:3045: expect (ObjNum 1269 snap 0 seq_num 1269) 2024-09-21T18:09:41.399 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:41.396+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:41.482 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.44112767499999, 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.483 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:09:41.484 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:09:41.485 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:09:41.486 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:09:41.487 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.440888051, 2024-09-21T18:09:41.488 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.489 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:09:41.490 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.491 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.492 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.493 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.44085264399999, 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.494 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:09:41.495 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:09:41.496 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:09:41.497 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.440819471, 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:09:41.498 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.499 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.500 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:09:41.501 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:09:41.502 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.440789911, 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.503 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.504 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.505 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.506 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.507 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.44076025499999, 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:41.508 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.509 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.510 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.511 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.512 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.440273999, 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.513 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.514 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.515 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.516 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:41.517 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.44024040599999, 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:41.518 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:09:41.519 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.520 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.521 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.34002271400001, 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.522 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:09:41.523 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:09:41.524 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:09:41.525 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:09:41.526 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.33942381, 2024-09-21T18:09:41.527 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.528 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:09:41.529 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:09:41.530 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:09:41.531 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.532 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.338709193, 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.533 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.534 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.535 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.536 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.537 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.32945646100001, 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:41.538 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.539 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.540 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.541 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.542 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.32919175999999, 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.543 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.544 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.545 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.546 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.547 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.32909707100001, 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.548 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.549 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:41.550 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:09:41.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.552 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.32905578099999, 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:09:41.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:09:41.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:09:41.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:09:41.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.558 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.317785878, 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.563 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.30866964500001, 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:09:41.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.30760627800001, 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.569 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:09:41.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.573 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.30756596800001, 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:41.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:09:41.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:41.578 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 129.544928126, 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:09:41.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:09:41.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:41.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:41.583 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:41.586 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:41.586 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:41.580+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:41.586 INFO:tasks.rados.rados.0.smithi160.stdout:3033: finishing copy_from racing read to smithi16030472-407 2024-09-21T18:09:41.586 INFO:tasks.rados.rados.0.smithi160.stdout:3047: got (0) Success 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3041: got (0) Success 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 200 v 0 (ObjNum 1277 snap 0 seq_num 1277) dirty exists 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3036: got (0) Success 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 222 v 0 (ObjNum 988 snap 0 seq_num 988) dirty exists 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3038: got (0) Success 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 373 v 0 (ObjNum 1200 snap 0 seq_num 1200) dirty exists 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3043: finishing write tid 1 to smithi16030472-426 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3044: got (0) Success 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 0 (ObjNum 1274 snap 0 seq_num 1274) dirty exists 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3033: done (10 left) 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3036: done (9 left) 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3038: done (8 left) 2024-09-21T18:09:41.587 INFO:tasks.rados.rados.0.smithi160.stdout:3039: done (7 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3041: done (6 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3044: done (5 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3045: done (4 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3046: done (3 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3047: done (2 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3048: done (1 left) 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3049: copy_from oid 299 from oid 70 current snap is 0 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3050: write oid 342 current snap is 0 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3050: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3050: writing smithi16030472-342 from 504573 to 941112 tid 1 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3050: writing smithi16030472-342 from 1421742 to 2020546 tid 2 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3050: writing smithi16030472-342 from 2437505 to 2504573 tid 3 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3051: cache_evict oid 452 snap -1 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3043: finishing write tid 2 to smithi16030472-426 2024-09-21T18:09:41.588 INFO:tasks.rados.rados.0.smithi160.stdout:3043: finishing write tid 3 to smithi16030472-426 2024-09-21T18:09:41.589 INFO:tasks.rados.rados.0.smithi160.stdout:3052: delete oid 384 current snap is 0 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:3043: finishing write tid 4 to smithi16030472-426 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:3043: oid 426 updating version 0 to 942 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:3043: oid 426 version 942 is already newer than 881 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:3043: oid 426 updating version 942 to 943 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 426 v 943 (ObjNum 1337 snap 0 seq_num 1337) dirty exists 2024-09-21T18:09:41.663 INFO:tasks.rados.rados.0.smithi160.stdout:3043: left oid 426 (ObjNum 1337 snap 0 seq_num 1337) 2024-09-21T18:09:41.675 INFO:tasks.rados.rados.0.smithi160.stdout:3049: finishing copy_from racing read to smithi16030472-299 2024-09-21T18:09:41.680 INFO:tasks.rados.rados.0.smithi160.stdout:3050: finishing write tid 1 to smithi16030472-342 2024-09-21T18:09:41.681 INFO:tasks.rados.rados.0.smithi160.stdout:3050: finishing write tid 2 to smithi16030472-342 2024-09-21T18:09:41.686 INFO:tasks.rados.rados.0.smithi160.stdout:3051: got (16) Device or resource busy 2024-09-21T18:09:41.686 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:41.681+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:41.688 INFO:tasks.rados.rados.0.smithi160.stdout:3050: finishing write tid 3 to smithi16030472-342 2024-09-21T18:09:41.757 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:41.757 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:41.757 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:41.757 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:41.771 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:41.786 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:41.785+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:41.882 INFO:tasks.rados.rados.0.smithi160.stdout:3049: finishing copy_from to smithi16030472-299 2024-09-21T18:09:41.882 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 299 v 867 (ObjNum 1257 snap 0 seq_num 1257) dirty exists 2024-09-21T18:09:41.887 INFO:tasks.rados.rados.0.smithi160.stdout:3050: finishing write tid 4 to smithi16030472-342 2024-09-21T18:09:41.887 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:41.882+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: finishing write tid 5 to smithi16030472-342 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: oid 342 updating version 0 to 966 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: oid 342 version 966 is already newer than 963 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: oid 342 version 966 is already newer than 964 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: oid 342 version 966 is already newer than 965 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 966 (ObjNum 1339 snap 0 seq_num 1339) dirty exists 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: left oid 342 (ObjNum 1339 snap 0 seq_num 1339) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3043: done (4 left) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3049: done (3 left) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3050: done (2 left) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3051: done (1 left) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3052: done (0 left) 2024-09-21T18:09:41.888 INFO:tasks.rados.rados.0.smithi160.stdout:3053: delete oid 111 current snap is 0 2024-09-21T18:09:41.900 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:41.900 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:41.900 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:41.900 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:41.900 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:41.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:09:41.915 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:41.988 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:41.981+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:41.991 INFO:tasks.rados.rados.0.smithi160.stdout:3053: done (0 left) 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3054: copy_from oid 65 from oid 200 current snap is 0 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3055: cache_try_flush oid 336 snap -1 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3056: read oid 81 snap -1 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3056: expect deleted 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3057: cache_evict oid 19 snap -1 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3058: delete oid 447 current snap is 0 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:3055: got (0) Success 2024-09-21T18:09:41.992 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 0 (ObjNum 335 snap 0 seq_num 335) dirty exists 2024-09-21T18:09:42.045 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.045 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 177.74304574300001, 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 177.741831684, 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:09:42.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 177.74048076400001, 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:09:42.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 171.45290012699999, 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:83 4.3 e43/33 4:f8b70d40:::smithi16030472-21:head v 43'22, mlcod=42'9)", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 149.35430749700001, 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21191591000000001, 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687790+0000", 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687792+0000", 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3269999999999999e-06 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687807+0000", 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4932999999999999e-05 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687927+0000", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000119672 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.687939+0000", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1656e-05 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792069+0000", 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.104130573 2024-09-21T18:09:42.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792117+0000", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7858999999999997e-05 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899646+0000", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.107528915 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899706+0000", 2024-09-21T18:09:42.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9975000000000001e-05 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:66 4.3 e43/33 4:fd62100b:::smithi16030472-17:head v 43'23, mlcod=42'9)", 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 149.35383028699999, 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211478269, 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688267+0000", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688268+0000", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5600000000000004e-07 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688555+0000", 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00028650600000000001 2024-09-21T18:09:42.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688671+0000", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000116618 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.688683+0000", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1270000000000001e-05 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792601+0000", 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.103918187 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.792656+0000", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4659e-05 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899713+0000", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10705722099999999 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:12.899746+0000", 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2852000000000003e-05 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 131.05366088700001, 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:09:42.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:09:42.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:42.073 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.953091495, 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:09:42.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.95273342799999, 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.077 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:09:42.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.95269659499999, 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:42.081 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.082 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.083 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.084 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.952236863, 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.085 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.086 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:09:42.087 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.088 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.95180533600001, 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.089 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.090 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.091 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.092 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.95086872100001, 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.093 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.094 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.095 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:09:42.096 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:09:42.097 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.95083003799999, 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.098 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.099 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.100 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.101 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.94954338599999, 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.102 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.103 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.104 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:09:42.105 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.847398922, 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.106 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:09:42.107 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.108 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.109 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:09:42.110 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.846675129, 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.111 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:09:42.112 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.113 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:42.114 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 122.72096489, 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.115 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.116 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:09:42.117 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:09:42.118 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 98.081170403000002, 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.119 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.120 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:09:42.121 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:7174 4.3 e101/33 4:fbf38165:::smithi16030472-24:head v 101'482, mlcod=100'479)", 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 19.239840570999998, 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21150047199999999, 2024-09-21T18:09:42.122 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802257+0000", 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.123 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.802260+0000", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8399999999999999e-06 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803007+0000", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00074713700000000004 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803040+0000", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3009999999999997e-05 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.124 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.803061+0000", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0676e-05 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.904316+0000", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101255181 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:22.904349+0000", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3226999999999999e-05 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:23.013660+0000", 2024-09-21T18:09:42.125 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109310465 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:23.013758+0000", 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7936000000000006e-05 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.126 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3057: got (16) Device or resource busy 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3055: done (4 left) 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3056: done (3 left) 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3057: done (2 left) 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3058: done (1 left) 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3059: cache_flush oid 244 snap -1 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3060: write oid 215 current snap is 0 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3060: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-21T18:09:42.128 INFO:tasks.rados.rados.0.smithi160.stdout:3060: writing smithi16030472-215 from 571801 to 1309336 tid 1 2024-09-21T18:09:42.129 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:42.129 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:42.125+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:42.136 INFO:tasks.rados.rados.0.smithi160.stdout:3060: writing smithi16030472-215 from 1939317 to 2429106 tid 2 2024-09-21T18:09:42.137 INFO:tasks.rados.rados.0.smithi160.stdout:3061: write (excl) oid 495 current snap is 0 2024-09-21T18:09:42.137 INFO:tasks.rados.rados.0.smithi160.stdout:3061: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-21T18:09:42.152 INFO:tasks.rados.rados.0.smithi160.stdout:3061: writing smithi16030472-495 from 588608 to 1001392 tid 1 2024-09-21T18:09:42.167 INFO:tasks.rados.rados.0.smithi160.stdout:3061: writing smithi16030472-495 from 1597799 to 2347599 tid 2 2024-09-21T18:09:42.167 INFO:tasks.rados.rados.0.smithi160.stdout:3062: read oid 97 snap -1 2024-09-21T18:09:42.167 INFO:tasks.rados.rados.0.smithi160.stdout:3062: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3063: read oid 323 snap -1 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3063: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3064: copy_from oid 109 from oid 452 current snap is 0 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3065: cache_flush oid 114 snap -1 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3066: write (excl) oid 141 current snap is 0 2024-09-21T18:09:42.168 INFO:tasks.rados.rados.0.smithi160.stdout:3066: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-21T18:09:42.177 INFO:tasks.rados.rados.0.smithi160.stdout:3066: writing smithi16030472-141 from 605415 to 1093448 tid 1 2024-09-21T18:09:42.186 INFO:tasks.rados.rados.0.smithi160.stdout:3066: writing smithi16030472-141 from 1656281 to 2149739 tid 2 2024-09-21T18:09:42.186 INFO:tasks.rados.rados.0.smithi160.stdout:3067: read oid 8 snap -1 2024-09-21T18:09:42.187 INFO:tasks.rados.rados.0.smithi160.stdout:3067: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:42.187 INFO:tasks.rados.rados.0.smithi160.stdout:3068: write (excl) oid 483 current snap is 0 2024-09-21T18:09:42.187 INFO:tasks.rados.rados.0.smithi160.stdout:3068: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-21T18:09:42.197 INFO:tasks.rados.rados.0.smithi160.stdout:3068: writing smithi16030472-483 from 622222 to 1301857 tid 1 2024-09-21T18:09:42.207 INFO:tasks.rados.rados.0.smithi160.stdout:3068: writing smithi16030472-483 from 1831116 to 2584585 tid 2 2024-09-21T18:09:42.207 INFO:tasks.rados.rados.0.smithi160.stdout:3069: cache_evict oid 471 snap -1 2024-09-21T18:09:42.207 INFO:tasks.rados.rados.0.smithi160.stdout:3070: delete oid 497 current snap is 0 2024-09-21T18:09:42.207 INFO:tasks.rados.rados.0.smithi160.stdout:3063: expect (ObjNum 8 snap 0 seq_num 8) 2024-09-21T18:09:42.228 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:42.225+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:42.256 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "pg_update_log_missing(3.0 epoch 112/31 rep_tid 451 entries 112'452 (0'0) error 3:139c9f0d:::smithi16030472-497:head by client.4876.0:8240 0.000000 -2 [r=-2+0b, r=0+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 112'449)", 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:42.223949+0000", 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.044348421999999998, 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.044422595000000002, 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:42.269 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:42.223949+0000", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:42.223949+0000", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:42.223952+0000", 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.3989999999999998e-06 2024-09-21T18:09:42.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:42.223968+0000", 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6276000000000001e-05 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:42.224017+0000", 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8950999999999999e-05 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-09-21T18:09:42.271 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.283 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:42.318 INFO:tasks.rados.rados.0.smithi160.stdout:3062: expect (ObjNum 1052 snap 0 seq_num 1052) 2024-09-21T18:09:42.322 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 861139 2024-09-21T18:09:42.329 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:42.324+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:42.403 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.403 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:42.403 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:42.403 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:42.404 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.418 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:42.459 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:42.456+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:42.473 INFO:tasks.rados.rados.0.smithi160.stdout:3064: finishing copy_from racing read to smithi16030472-109 2024-09-21T18:09:42.473 INFO:tasks.rados.rados.0.smithi160.stdout:3059: got (0) Success 2024-09-21T18:09:42.473 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 244 v 0 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-09-21T18:09:42.473 INFO:tasks.rados.rados.0.smithi160.stdout:3069: got (0) Success 2024-09-21T18:09:42.473 INFO:tasks.rados.rados.0.smithi160.stdout:3054: finishing copy_from to smithi16030472-65 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 65 v 880 (ObjNum 1277 snap 0 seq_num 1277) dirty exists 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3054: finishing copy_from racing read to smithi16030472-65 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3066: finishing write tid 1 to smithi16030472-141 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3061: finishing write tid 1 to smithi16030472-495 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3068: finishing write tid 1 to smithi16030472-483 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3065: got (0) Success 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 0 (ObjNum 841 snap 0 seq_num 841) dirty exists 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3054: done (12 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3059: done (11 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3062: done (10 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3063: done (9 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3065: done (8 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3069: done (7 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3070: done (6 left) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3071: read oid 441 snap -1 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3071: expect (ObjNum 928 snap 0 seq_num 928) 2024-09-21T18:09:42.474 INFO:tasks.rados.rados.0.smithi160.stdout:3072: cache_flush oid 66 snap -1 2024-09-21T18:09:42.475 INFO:tasks.rados.rados.0.smithi160.stdout:3060: finishing write tid 1 to smithi16030472-215 2024-09-21T18:09:42.475 INFO:tasks.rados.rados.0.smithi160.stdout:3060: finishing write tid 2 to smithi16030472-215 2024-09-21T18:09:42.475 INFO:tasks.rados.rados.0.smithi160.stdout:3067: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:42.531 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:42.529+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:42.588 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 131.55401518599999, 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.592 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 131.55040463500001, 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:09:42.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:09:42.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:09:42.596 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 131.547598096, 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:09:42.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:09:42.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.600 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 131.54384989600001, 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:42.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:09:42.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:09:42.604 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.922999092, 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:09:42.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.82214501999999, 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:09:42.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.82138874, 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:09:42.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:09:42.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.820997183, 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:09:42.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:09:42.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.820696059, 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:42.625 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.81997220900001, 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:42.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.629 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.71907537000001, 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:42.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:09:42.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:09:42.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.633 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.718981087, 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.634 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:09:42.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:09:42.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:09:42.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.71806789199999, 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:09:42.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.71749820400001, 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:09:42.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.644 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.645 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.71677072, 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:42.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:09:42.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 130.65058881100001, 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:09:42.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:09:42.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:09:42.653 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 129.39454300599999, 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:42.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:09:42.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:09:42.656 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:09:42.657 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 129.39426020799999, 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:42.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:09:42.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 129.394181223, 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.663 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 129.391562892, 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:42.666 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:42.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:42.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:42.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:42.670 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3061: finishing write tid 2 to smithi16030472-495 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3072: got (2) No such file or directory 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3060: finishing write tid 3 to smithi16030472-215 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3060: finishing write tid 4 to smithi16030472-215 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3060: oid 215 updating version 0 to 886 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3060: oid 215 version 886 is already newer than 885 2024-09-21T18:09:42.671 INFO:tasks.rados.rados.0.smithi160.stdout:3060: oid 215 updating version 886 to 887 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 215 v 887 (ObjNum 1343 snap 0 seq_num 1343) dirty exists 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:3060: left oid 215 (ObjNum 1343 snap 0 seq_num 1343) 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:3064: finishing copy_from to smithi16030472-109 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 811 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:3068: finishing write tid 2 to smithi16030472-483 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:3071: expect (ObjNum 928 snap 0 seq_num 928) 2024-09-21T18:09:42.672 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:42.673 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:42.673 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:42.665+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:42.674 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2144364 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: finishing write tid 2 to smithi16030472-141 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: finishing write tid 3 to smithi16030472-141 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: finishing write tid 4 to smithi16030472-141 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: oid 141 updating version 0 to 970 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: oid 141 version 970 is already newer than 968 2024-09-21T18:09:42.754 INFO:tasks.rados.rados.0.smithi160.stdout:3066: oid 141 version 970 is already newer than 969 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 141 v 970 (ObjNum 1345 snap 0 seq_num 1345) dirty exists 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3066: left oid 141 (ObjNum 1345 snap 0 seq_num 1345) 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: finishing write tid 3 to smithi16030472-495 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: finishing write tid 4 to smithi16030472-495 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: oid 495 updating version 0 to 945 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: oid 495 version 945 is already newer than 944 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: oid 495 updating version 945 to 946 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 946 (ObjNum 1344 snap 0 seq_num 1344) dirty exists 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3061: left oid 495 (ObjNum 1344 snap 0 seq_num 1344) 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3068: finishing write tid 3 to smithi16030472-483 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3068: finishing write tid 4 to smithi16030472-483 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3068: oid 483 updating version 0 to 828 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3068: oid 483 updating version 828 to 889 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:3068: oid 483 version 889 is already newer than 888 2024-09-21T18:09:42.755 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 483 v 889 (ObjNum 1346 snap 0 seq_num 1346) dirty exists 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3068: left oid 483 (ObjNum 1346 snap 0 seq_num 1346) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3060: done (7 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3061: done (6 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3064: done (5 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3066: done (4 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3067: done (3 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3068: done (2 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3071: done (1 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3072: done (0 left) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3073: cache_flush oid 60 snap -1 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3074: read oid 134 snap -1 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3074: expect (ObjNum 650 snap 0 seq_num 650) 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3075: read oid 251 snap -1 2024-09-21T18:09:42.756 INFO:tasks.rados.rados.0.smithi160.stdout:3075: expect (ObjNum 1129 snap 0 seq_num 1129) 2024-09-21T18:09:42.757 INFO:tasks.rados.rados.0.smithi160.stdout:3076: read oid 373 snap -1 2024-09-21T18:09:42.757 INFO:tasks.rados.rados.0.smithi160.stdout:3076: expect (ObjNum 1200 snap 0 seq_num 1200) 2024-09-21T18:09:42.757 INFO:tasks.rados.rados.0.smithi160.stdout:3077: delete oid 174 current snap is 0 2024-09-21T18:09:42.772 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:42.769+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:42.802 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.802 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:42.802 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:42.803 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:42.861 INFO:tasks.rados.rados.0.smithi160.stdout:3077: done (4 left) 2024-09-21T18:09:42.861 INFO:tasks.rados.rados.0.smithi160.stdout:3078: cache_try_flush oid 57 snap -1 2024-09-21T18:09:42.861 INFO:tasks.rados.rados.0.smithi160.stdout:3079: write (excl) oid 396 current snap is 0 2024-09-21T18:09:42.861 INFO:tasks.rados.rados.0.smithi160.stdout:3079: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-21T18:09:42.873 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:42.870+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:42.880 INFO:tasks.rados.rados.0.smithi160.stdout:3079: writing smithi16030472-396 from 672643 to 1178025 tid 1 2024-09-21T18:09:42.892 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:09:42.901 INFO:tasks.rados.rados.0.smithi160.stdout:3079: writing smithi16030472-396 from 1890209 to 2391005 tid 2 2024-09-21T18:09:42.901 INFO:tasks.rados.rados.0.smithi160.stdout:3080: cache_flush oid 429 snap -1 2024-09-21T18:09:42.901 INFO:tasks.rados.rados.0.smithi160.stdout:3074: expect (ObjNum 650 snap 0 seq_num 650) 2024-09-21T18:09:42.930 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1229509 2024-09-21T18:09:42.947 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:42.947 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:42.947 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:42.947 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:42.947 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:42.961 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:42.997 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:43.007 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:43.004+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:43.015 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:43.010+0000 7f7283507640 -1 osd.14 112 handle_osd_map injecting map crc failure 2024-09-21T18:09:43.076 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:43.070+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:43.092 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:43.092 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:43.092 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:43.092 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:09:43.092 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:43.101 INFO:tasks.rados.rados.0.smithi160.stdout:3076: expect (ObjNum 1200 snap 0 seq_num 1200) 2024-09-21T18:09:43.107 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:43.111 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1480242 2024-09-21T18:09:43.123 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:43.176 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:43.170+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:3078: got (0) Success 2024-09-21T18:09:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 57 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:3080: got (0) Success 2024-09-21T18:09:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 0 (ObjNum 428 snap 0 seq_num 428) dirty exists 2024-09-21T18:09:43.178 INFO:tasks.rados.rados.0.smithi160.stdout:3075: expect (ObjNum 1129 snap 0 seq_num 1129) 2024-09-21T18:09:43.228 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1683954 2024-09-21T18:09:43.238 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:43.238 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:43.238 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:43.239 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:43.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:43.277 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:43.273+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:43.289 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2920871 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: finishing write tid 1 to smithi16030472-396 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: finishing write tid 2 to smithi16030472-396 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3073: got (0) Success 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 0 (ObjNum 1161 snap 0 seq_num 1161) dirty exists 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: finishing write tid 3 to smithi16030472-396 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: finishing write tid 4 to smithi16030472-396 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: oid 396 updating version 0 to 893 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: oid 396 version 893 is already newer than 892 2024-09-21T18:09:43.296 INFO:tasks.rados.rados.0.smithi160.stdout:3079: oid 396 version 893 is already newer than 891 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 396 v 893 (ObjNum 1349 snap 0 seq_num 1349) dirty exists 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3079: left oid 396 (ObjNum 1349 snap 0 seq_num 1349) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3073: done (6 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3074: done (5 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3075: done (4 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3076: done (3 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3078: done (2 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3079: done (1 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3080: done (0 left) 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3081: write (excl) oid 79 current snap is 0 2024-09-21T18:09:43.297 INFO:tasks.rados.rados.0.smithi160.stdout:3081: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-09-21T18:09:43.316 INFO:tasks.rados.rados.0.smithi160.stdout:3081: writing smithi16030472-79 from 689450 to 1270081 tid 1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3081: writing smithi16030472-79 from 1948691 to 2689450 tid 2 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3082: read oid 483 snap -1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3082: expect (ObjNum 1346 snap 0 seq_num 1346) 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3083: cache_flush oid 129 snap -1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3084: read oid 260 snap -1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3084: expect (ObjNum 580 snap 0 seq_num 580) 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3085: read oid 209 snap -1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3085: expect deleted 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3086: read oid 319 snap -1 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3086: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3087: copy_from oid 382 from oid 376 current snap is 0 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3088: write (excl) oid 262 current snap is 0 2024-09-21T18:09:43.341 INFO:tasks.rados.rados.0.smithi160.stdout:3088: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-21T18:09:43.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:43.359 INFO:tasks.rados.rados.0.smithi160.stdout:3088: writing smithi16030472-262 from 706257 to 1362137 tid 1 2024-09-21T18:09:43.369 INFO:tasks.rados.rados.0.smithi160.stdout:3088: writing smithi16030472-262 from 1890820 to 2395285 tid 2 2024-09-21T18:09:43.369 INFO:tasks.rados.rados.0.smithi160.stdout:3089: read oid 274 snap -1 2024-09-21T18:09:43.369 INFO:tasks.rados.rados.0.smithi160.stdout:3089: expect (ObjNum 1241 snap 0 seq_num 1241) 2024-09-21T18:09:43.369 INFO:tasks.rados.rados.0.smithi160.stdout:3090: delete oid 404 current snap is 0 2024-09-21T18:09:43.377 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:43.374+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:43.420 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:43.420 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:43.420 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:43.420 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:43.420 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:43.434 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:43.442 INFO:tasks.rados.rados.0.smithi160.stdout:3083: got (0) Success 2024-09-21T18:09:43.442 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 129 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:09:43.470 INFO:tasks.rados.rados.0.smithi160.stdout:3086: expect (ObjNum 318 snap 0 seq_num 318) 2024-09-21T18:09:43.478 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:43.473+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:43.506 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:43.514 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:43.515 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:43.535 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:43.535 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:43.555 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:43.569 INFO:tasks.rados.rados.0.smithi160.stdout:3089: expect (ObjNum 1241 snap 0 seq_num 1241) 2024-09-21T18:09:43.573 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:43.579 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:43.577+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:43.590 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:43.590 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:43.602 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:43.603 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:43.610 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:43.611 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:43.617 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:43.629 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:43.647 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:43.668 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:43.679 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:43.674+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:43.687 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:43.695 INFO:tasks.rados.rados.0.smithi160.stdout:3082: expect (ObjNum 1346 snap 0 seq_num 1346) 2024-09-21T18:09:43.704 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1044773 2024-09-21T18:09:43.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:43.708 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:43.728 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:43.728 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:43.739 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:43.739 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:43.749 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1904316 2024-09-21T18:09:43.749 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:43.749 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:43.754 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:43.762 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:43.767 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:43.767 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:43.780 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:43.776+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:43.789 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:43.789 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:43.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:43.795 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3081: finishing write tid 1 to smithi16030472-79 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3088: finishing write tid 1 to smithi16030472-262 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3082: done (9 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3083: done (8 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3085: done (7 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3086: done (6 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3089: done (5 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3090: done (4 left) 2024-09-21T18:09:43.804 INFO:tasks.rados.rados.0.smithi160.stdout:3091: cache_flush oid 298 snap -1 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: finishing write tid 2 to smithi16030472-79 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: finishing write tid 3 to smithi16030472-79 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: finishing write tid 4 to smithi16030472-79 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: oid 79 updating version 0 to 894 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: oid 79 updating version 894 to 895 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: oid 79 updating version 895 to 896 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 896 (ObjNum 1350 snap 0 seq_num 1350) dirty exists 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: left oid 79 (ObjNum 1350 snap 0 seq_num 1350) 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3081: done (4 left) 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3092: copy_from oid 386 from oid 8 current snap is 0 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3093: write oid 88 current snap is 0 2024-09-21T18:09:43.805 INFO:tasks.rados.rados.0.smithi160.stdout:3093: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-21T18:09:43.810 INFO:tasks.rados.rados.0.smithi160.stdout:3093: writing smithi16030472-88 from 739871 to 1146249 tid 1 2024-09-21T18:09:43.816 INFO:tasks.rados.rados.0.smithi160.stdout:3093: writing smithi16030472-88 from 1607784 to 2115918 tid 2 2024-09-21T18:09:43.816 INFO:tasks.rados.rados.0.smithi160.stdout:3094: cache_try_flush oid 389 snap -1 2024-09-21T18:09:43.816 INFO:tasks.rados.rados.0.smithi160.stdout:3095: delete oid 308 current snap is 0 2024-09-21T18:09:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:3095: done (7 left) 2024-09-21T18:09:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:3096: read oid 87 snap -1 2024-09-21T18:09:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:3096: expect (ObjNum 891 snap 0 seq_num 891) 2024-09-21T18:09:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:3097: cache_flush oid 243 snap -1 2024-09-21T18:09:43.820 INFO:tasks.rados.rados.0.smithi160.stdout:3098: delete oid 277 current snap is 0 2024-09-21T18:09:43.881 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:43.873+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:43.918 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:43.918 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:43.927 INFO:tasks.rados.rados.0.smithi160.stdout:3088: finishing write tid 2 to smithi16030472-262 2024-09-21T18:09:43.930 INFO:tasks.rados.rados.0.smithi160.stdout:3096: expect (ObjNum 891 snap 0 seq_num 891) 2024-09-21T18:09:43.930 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 565781 2024-09-21T18:09:43.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:43.981 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:43.978+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:44.033 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1617085 2024-09-21T18:09:44.052 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.053 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.068 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:44.076 INFO:tasks.rados.rados.0.smithi160.stdout:3088: finishing write tid 3 to smithi16030472-262 2024-09-21T18:09:44.076 INFO:tasks.rados.rados.0.smithi160.stdout:3088: finishing write tid 4 to smithi16030472-262 2024-09-21T18:09:44.077 INFO:tasks.rados.rados.0.smithi160.stdout:3088: oid 262 updating version 0 to 936 2024-09-21T18:09:44.077 INFO:tasks.rados.rados.0.smithi160.stdout:3088: oid 262 version 936 is already newer than 934 2024-09-21T18:09:44.077 INFO:tasks.rados.rados.0.smithi160.stdout:3088: oid 262 version 936 is already newer than 935 2024-09-21T18:09:44.077 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 262 v 936 (ObjNum 1351 snap 0 seq_num 1351) dirty exists 2024-09-21T18:09:44.077 INFO:tasks.rados.rados.0.smithi160.stdout:3088: left oid 262 (ObjNum 1351 snap 0 seq_num 1351) 2024-09-21T18:09:44.107 INFO:tasks.rados.rados.0.smithi160.stdout:3091: got (0) Success 2024-09-21T18:09:44.107 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 298 v 0 (ObjNum 1330 snap 0 seq_num 1330) dirty exists 2024-09-21T18:09:44.107 INFO:tasks.rados.rados.0.smithi160.stdout:3087: finishing copy_from to smithi16030472-382 2024-09-21T18:09:44.107 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 382 v 560 (ObjNum 740 snap 0 seq_num 740) dirty exists 2024-09-21T18:09:44.107 INFO:tasks.rados.rados.0.smithi160.stdout:3087: finishing copy_from racing read to smithi16030472-382 2024-09-21T18:09:44.110 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:44.106+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:44.110 INFO:tasks.rados.rados.0.smithi160.stdout:3094: got (0) Success 2024-09-21T18:09:44.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 389 v 0 (ObjNum 388 snap 0 seq_num 388) dirty exists 2024-09-21T18:09:44.160 INFO:tasks.rados.rados.0.smithi160.stdout:3092: finishing copy_from to smithi16030472-386 2024-09-21T18:09:44.161 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 386 v 762 (ObjNum 1115 snap 0 seq_num 1115) dirty exists 2024-09-21T18:09:44.183 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:44.178+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:44.228 INFO:tasks.rados.rados.0.smithi160.stdout:3084: expect (ObjNum 580 snap 0 seq_num 580) 2024-09-21T18:09:44.231 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.231 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:44.281 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:44.284 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:44.281+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3093: finishing write tid 1 to smithi16030472-88 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3093: finishing write tid 2 to smithi16030472-88 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3097: got (0) Success 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 0 (ObjNum 1227 snap 0 seq_num 1227) dirty exists 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3093: finishing write tid 3 to smithi16030472-88 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3093: finishing write tid 4 to smithi16030472-88 2024-09-21T18:09:44.382 INFO:tasks.rados.rados.0.smithi160.stdout:3093: oid 88 updating version 0 to 899 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3093: oid 88 version 899 is already newer than 897 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3093: oid 88 version 899 is already newer than 898 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 899 (ObjNum 1353 snap 0 seq_num 1353) dirty exists 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3093: left oid 88 (ObjNum 1353 snap 0 seq_num 1353) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3092: finishing copy_from racing read to smithi16030472-386 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3084: done (9 left) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3087: done (8 left) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3088: done (7 left) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3091: done (6 left) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3092: done (5 left) 2024-09-21T18:09:44.383 INFO:tasks.rados.rados.0.smithi160.stdout:3093: done (4 left) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3094: done (3 left) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3096: done (2 left) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3097: done (1 left) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3098: done (0 left) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3099: read oid 151 snap -1 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3099: expect (ObjNum 753 snap 0 seq_num 753) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3100: read oid 3 snap -1 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3100: expect (ObjNum 1051 snap 0 seq_num 1051) 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3101: read oid 341 snap -1 2024-09-21T18:09:44.384 INFO:tasks.rados.rados.0.smithi160.stdout:3101: expect (ObjNum 705 snap 0 seq_num 705) 2024-09-21T18:09:44.385 INFO:tasks.rados.rados.0.smithi160.stdout:3102: cache_try_flush oid 19 snap -1 2024-09-21T18:09:44.385 INFO:tasks.rados.rados.0.smithi160.stdout:3103: read oid 411 snap -1 2024-09-21T18:09:44.385 INFO:tasks.rados.rados.0.smithi160.stdout:3103: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-21T18:09:44.385 INFO:tasks.rados.rados.0.smithi160.stdout:3104: write oid 266 current snap is 0 2024-09-21T18:09:44.385 INFO:tasks.rados.rados.0.smithi160.stdout:3104: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-09-21T18:09:44.386 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:44.381+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:44.406 INFO:tasks.rados.rados.0.smithi160.stdout:3104: writing smithi16030472-266 from 790292 to 1538770 tid 1 2024-09-21T18:09:44.412 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.412 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.427 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:44.429 INFO:tasks.rados.rados.0.smithi160.stdout:3104: writing smithi16030472-266 from 2183230 to 2790292 tid 2 2024-09-21T18:09:44.429 INFO:tasks.rados.rados.0.smithi160.stdout:3105: cache_evict oid 304 snap -1 2024-09-21T18:09:44.430 INFO:tasks.rados.rados.0.smithi160.stdout:3106: cache_try_flush oid 404 snap -1 2024-09-21T18:09:44.430 INFO:tasks.rados.rados.0.smithi160.stdout:3107: write oid 38 current snap is 0 2024-09-21T18:09:44.430 INFO:tasks.rados.rados.0.smithi160.stdout:3107: seq_num 1357 ranges {407099=423727,1441712=515472,2582060=225039} 2024-09-21T18:09:44.438 INFO:tasks.rados.rados.0.smithi160.stdout:3107: writing smithi16030472-38 from 407099 to 830826 tid 1 2024-09-21T18:09:44.450 INFO:tasks.rados.rados.0.smithi160.stdout:3107: writing smithi16030472-38 from 1441712 to 1957184 tid 2 2024-09-21T18:09:44.459 INFO:tasks.rados.rados.0.smithi160.stdout:3107: writing smithi16030472-38 from 2582060 to 2807099 tid 3 2024-09-21T18:09:44.459 INFO:tasks.rados.rados.0.smithi160.stdout:3108: cache_try_flush oid 154 snap -1 2024-09-21T18:09:44.459 INFO:tasks.rados.rados.0.smithi160.stdout:3109: read oid 82 snap -1 2024-09-21T18:09:44.459 INFO:tasks.rados.rados.0.smithi160.stdout:3109: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:09:44.460 INFO:tasks.rados.rados.0.smithi160.stdout:3110: delete oid 71 current snap is 0 2024-09-21T18:09:44.482 INFO:tasks.rados.rados.0.smithi160.stdout:3103: expect (ObjNum 410 snap 0 seq_num 410) 2024-09-21T18:09:44.483 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:44.487 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:44.484+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:44.547 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.547 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.551 INFO:tasks.rados.rados.0.smithi160.stdout:3101: expect (ObjNum 705 snap 0 seq_num 705) 2024-09-21T18:09:44.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:44.588 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:44.585+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:44.674 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1267752 2024-09-21T18:09:44.687 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.687 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.689 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:44.685+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:44.703 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:3106: got (0) Success 2024-09-21T18:09:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 404 v 0 (ObjNum 129 snap 0 seq_num 3139922368) clean dne 2024-09-21T18:09:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:3104: finishing write tid 1 to smithi16030472-266 2024-09-21T18:09:44.717 INFO:tasks.rados.rados.0.smithi160.stdout:3100: expect (ObjNum 1051 snap 0 seq_num 1051) 2024-09-21T18:09:44.725 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1106340 2024-09-21T18:09:44.790 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:44.786+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:44.828 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:44.828 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:44.842 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:44.865 INFO:tasks.rados.rados.0.smithi160.stdout:3099: expect (ObjNum 753 snap 0 seq_num 753) 2024-09-21T18:09:44.888 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2497334 2024-09-21T18:09:44.890 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:44.886+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:44.896 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 786465 2024-09-21T18:09:44.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:09:44.935 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1982902 2024-09-21T18:09:44.947 INFO:tasks.rados.rados.0.smithi160.stdout:3102: got (0) Success 2024-09-21T18:09:44.947 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 19 v 0 (ObjNum 960 snap 0 seq_num 960) dirty exists 2024-09-21T18:09:44.947 INFO:tasks.rados.rados.0.smithi160.stdout:3109: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:09:44.964 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1510593 2024-09-21T18:09:44.983 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:44.991 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:44.986+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:45.014 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.014 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.027 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:45.028 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3099: done (11 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3100: done (10 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3101: done (9 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3102: done (8 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3103: done (7 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3106: done (6 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3109: done (5 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3110: done (4 left) 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3111: copy_from oid 185 from oid 245 current snap is 0 2024-09-21T18:09:45.073 INFO:tasks.rados.rados.0.smithi160.stdout:3108: got (0) Success 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 154 v 0 (ObjNum 651 snap 0 seq_num 651) dirty exists 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3105: got (16) Device or resource busy 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: finishing write tid 2 to smithi16030472-266 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: finishing write tid 3 to smithi16030472-266 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: finishing write tid 4 to smithi16030472-266 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: oid 266 updating version 0 to 938 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: oid 266 version 938 is already newer than 937 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: oid 266 updating version 938 to 939 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 939 (ObjNum 1356 snap 0 seq_num 1356) dirty exists 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3104: left oid 266 (ObjNum 1356 snap 0 seq_num 1356) 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3107: finishing write tid 1 to smithi16030472-38 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3107: finishing write tid 2 to smithi16030472-38 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3107: finishing write tid 3 to smithi16030472-38 2024-09-21T18:09:45.074 INFO:tasks.rados.rados.0.smithi160.stdout:3107: finishing write tid 4 to smithi16030472-38 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: finishing write tid 5 to smithi16030472-38 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: oid 38 updating version 0 to 972 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: oid 38 updating version 972 to 973 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: oid 38 updating version 973 to 974 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: oid 38 updating version 974 to 975 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 38 v 975 (ObjNum 1357 snap 0 seq_num 1357) dirty exists 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: left oid 38 (ObjNum 1357 snap 0 seq_num 1357) 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3104: done (4 left) 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3105: done (3 left) 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3107: done (2 left) 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3108: done (1 left) 2024-09-21T18:09:45.075 INFO:tasks.rados.rados.0.smithi160.stdout:3112: delete oid 253 current snap is 0 2024-09-21T18:09:45.091 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:45.089+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:45.149 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.149 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.164 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:45.178 INFO:tasks.rados.rados.0.smithi160.stdout:3112: done (1 left) 2024-09-21T18:09:45.178 INFO:tasks.rados.rados.0.smithi160.stdout:3113: read oid 378 snap -1 2024-09-21T18:09:45.178 INFO:tasks.rados.rados.0.smithi160.stdout:3113: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:09:45.179 INFO:tasks.rados.rados.0.smithi160.stdout:3114: cache_flush oid 118 snap -1 2024-09-21T18:09:45.179 INFO:tasks.rados.rados.0.smithi160.stdout:3115: write oid 212 current snap is 0 2024-09-21T18:09:45.179 INFO:tasks.rados.rados.0.smithi160.stdout:3115: seq_num 1360 ranges {457520=649474,1900805=662799} 2024-09-21T18:09:45.193 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:45.189+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:45.196 INFO:tasks.rados.rados.0.smithi160.stdout:3115: writing smithi16030472-212 from 457520 to 1106994 tid 1 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3115: writing smithi16030472-212 from 1900805 to 2563604 tid 2 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3116: read oid 71 snap -1 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3116: expect deleted 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3117: delete oid 326 current snap is 0 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3114: got (0) Success 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 118 v 0 (ObjNum 4241 snap 0 seq_num 3126278624) clean dne 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3111: finishing copy_from racing read to smithi16030472-185 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3111: finishing copy_from to smithi16030472-185 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 185 v 834 (ObjNum 1267 snap 0 seq_num 1267) dirty exists 2024-09-21T18:09:45.221 INFO:tasks.rados.rados.0.smithi160.stdout:3115: finishing write tid 1 to smithi16030472-212 2024-09-21T18:09:45.284 INFO:tasks.rados.rados.0.smithi160.stdout:3113: expect (ObjNum 733 snap 0 seq_num 733) 2024-09-21T18:09:45.294 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:45.290+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:45.299 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.299 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.313 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:45.324 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:45.395 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:45.392+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:45.423 INFO:tasks.rados.rados.0.smithi160.stdout:3115: finishing write tid 2 to smithi16030472-212 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: finishing write tid 3 to smithi16030472-212 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: finishing write tid 4 to smithi16030472-212 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: oid 212 updating version 0 to 976 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: oid 212 updating version 976 to 977 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: oid 212 updating version 977 to 978 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 212 v 978 (ObjNum 1360 snap 0 seq_num 1360) dirty exists 2024-09-21T18:09:45.425 INFO:tasks.rados.rados.0.smithi160.stdout:3115: left oid 212 (ObjNum 1360 snap 0 seq_num 1360) 2024-09-21T18:09:45.448 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.448 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.462 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:45.495 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:45.493+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:45.525 INFO:tasks.rados.rados.0.smithi160.stdout:3111: done (5 left) 2024-09-21T18:09:45.525 INFO:tasks.rados.rados.0.smithi160.stdout:3113: done (4 left) 2024-09-21T18:09:45.525 INFO:tasks.rados.rados.0.smithi160.stdout:3114: done (3 left) 2024-09-21T18:09:45.525 INFO:tasks.rados.rados.0.smithi160.stdout:3115: done (2 left) 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3116: done (1 left) 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3117: done (0 left) 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3118: read oid 287 snap -1 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3118: expect deleted 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3119: cache_flush oid 4 snap -1 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3120: copy_from oid 480 from oid 440 current snap is 0 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3121: write (excl) oid 207 current snap is 0 2024-09-21T18:09:45.526 INFO:tasks.rados.rados.0.smithi160.stdout:3121: seq_num 1362 ranges {491134=516325,1734122=666468} 2024-09-21T18:09:45.531 INFO:tasks.rados.rados.0.smithi160.stdout:3121: writing smithi16030472-207 from 491134 to 1007459 tid 1 2024-09-21T18:09:45.540 INFO:tasks.rados.rados.0.smithi160.stdout:3121: writing smithi16030472-207 from 1734122 to 2400590 tid 2 2024-09-21T18:09:45.540 INFO:tasks.rados.rados.0.smithi160.stdout:3122: delete oid 206 current snap is 0 2024-09-21T18:09:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:3119: got (0) Success 2024-09-21T18:09:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 4 v 0 (ObjNum 805 snap 0 seq_num 805) dirty exists 2024-09-21T18:09:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:3120: finishing copy_from to smithi16030472-480 2024-09-21T18:09:45.541 INFO:tasks.rados.rados.0.smithi160.stdout:3120: got expected ENOENT (src dne) 2024-09-21T18:09:45.581 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.581 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.594 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:45.595 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:45.593+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:45.627 INFO:tasks.rados.rados.0.smithi160.stdout:3120: finishing copy_from racing read to smithi16030472-480 2024-09-21T18:09:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:3121: finishing write tid 1 to smithi16030472-207 2024-09-21T18:09:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:3121: finishing write tid 2 to smithi16030472-207 2024-09-21T18:09:45.635 INFO:tasks.rados.rados.0.smithi160.stdout:3121: finishing write tid 3 to smithi16030472-207 2024-09-21T18:09:45.696 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:45.692+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:45.703 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.703 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.717 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:45.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:45.805 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:45.801+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3121: finishing write tid 4 to smithi16030472-207 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3121: oid 207 updating version 0 to 948 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3121: oid 207 updating version 948 to 949 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3121: oid 207 version 949 is already newer than 947 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 207 v 949 (ObjNum 1362 snap 0 seq_num 1362) dirty exists 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3121: left oid 207 (ObjNum 1362 snap 0 seq_num 1362) 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3118: done (4 left) 2024-09-21T18:09:45.837 INFO:tasks.rados.rados.0.smithi160.stdout:3119: done (3 left) 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3120: done (2 left) 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3121: done (1 left) 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3122: done (0 left) 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3123: read oid 70 snap -1 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3123: expect (ObjNum 1257 snap 0 seq_num 1257) 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3124: cache_evict oid 207 snap -1 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3125: write oid 359 current snap is 0 2024-09-21T18:09:45.838 INFO:tasks.rados.rados.0.smithi160.stdout:3125: seq_num 1364 ranges {524748=666823,1734733=670137} 2024-09-21T18:09:45.854 INFO:tasks.rados.rados.0.smithi160.stdout:3125: writing smithi16030472-359 from 524748 to 1191571 tid 1 2024-09-21T18:09:45.870 INFO:tasks.rados.rados.0.smithi160.stdout:3125: writing smithi16030472-359 from 1734733 to 2404870 tid 2 2024-09-21T18:09:45.870 INFO:tasks.rados.rados.0.smithi160.stdout:3126: read oid 273 snap -1 2024-09-21T18:09:45.870 INFO:tasks.rados.rados.0.smithi160.stdout:3126: expect (ObjNum 1110 snap 0 seq_num 1110) 2024-09-21T18:09:45.870 INFO:tasks.rados.rados.0.smithi160.stdout:3127: write oid 66 current snap is 0 2024-09-21T18:09:45.870 INFO:tasks.rados.rados.0.smithi160.stdout:3127: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-21T18:09:45.883 INFO:tasks.rados.rados.0.smithi160.stdout:3127: writing smithi16030472-66 from 541555 to 1283627 tid 1 2024-09-21T18:09:45.884 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:45.885 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:45.893 INFO:tasks.rados.rados.0.smithi160.stdout:3127: writing smithi16030472-66 from 1793215 to 2323363 tid 2 2024-09-21T18:09:45.899 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:45.899 INFO:tasks.rados.rados.0.smithi160.stdout:3127: writing smithi16030472-66 from 2801444 to 2941555 tid 3 2024-09-21T18:09:45.899 INFO:tasks.rados.rados.0.smithi160.stdout:3128: cache_evict oid 358 snap -1 2024-09-21T18:09:45.899 INFO:tasks.rados.rados.0.smithi160.stdout:3129: cache_evict oid 142 snap -1 2024-09-21T18:09:45.899 INFO:tasks.rados.rados.0.smithi160.stdout:3123: expect (ObjNum 1257 snap 0 seq_num 1257) 2024-09-21T18:09:45.900 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:45.897+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:45.918 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:09:45.919 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:45.917+0000 7fb926fba640 -1 osd.8 113 handle_osd_map injecting map crc failure 2024-09-21T18:09:45.958 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:45.981 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:46.183 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:46.183 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:46.183 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:46.183 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:46.184 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:46.184 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:46.184 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:46.185 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:46.185 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:46.186 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:46.186 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:46.187 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:46.188 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:46.188 INFO:tasks.rados.rados.0.smithi160.stdout:3124: got (16) Device or resource busy 2024-09-21T18:09:46.188 INFO:tasks.rados.rados.0.smithi160.stdout:3128: got (0) Success 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3123: done (6 left) 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3124: done (5 left) 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3128: done (4 left) 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3130: read oid 149 snap -1 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3130: expect (ObjNum 1315 snap 0 seq_num 1315) 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3131: cache_try_flush oid 56 snap -1 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3129: got (0) Success 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3129: done (5 left) 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3132: cache_evict oid 182 snap -1 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3133: cache_flush oid 491 snap -1 2024-09-21T18:09:46.189 INFO:tasks.rados.rados.0.smithi160.stdout:3134: cache_evict oid 106 snap -1 2024-09-21T18:09:46.190 INFO:tasks.rados.rados.0.smithi160.stdout:3135: write (excl) oid 117 current snap is 0 2024-09-21T18:09:46.190 INFO:tasks.rados.rados.0.smithi160.stdout:3135: seq_num 1366 ranges {558362=417321,1451697=673806,2712514=245848} 2024-09-21T18:09:46.190 INFO:tasks.rados.rados.0.smithi160.stdout:3135: writing smithi16030472-117 from 558362 to 975683 tid 1 2024-09-21T18:09:46.190 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:46.190 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:46.190 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3135: writing smithi16030472-117 from 1451697 to 2125503 tid 2 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3135: writing smithi16030472-117 from 2712514 to 2958362 tid 3 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3136: cache_evict oid 67 snap -1 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3137: read oid 425 snap -1 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3137: expect (ObjNum 1199 snap 0 seq_num 1199) 2024-09-21T18:09:46.191 INFO:tasks.rados.rados.0.smithi160.stdout:3138: read oid 467 snap -1 2024-09-21T18:09:46.192 INFO:tasks.rados.rados.0.smithi160.stdout:3138: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:46.192 INFO:tasks.rados.rados.0.smithi160.stdout:3139: write oid 363 current snap is 0 2024-09-21T18:09:46.192 INFO:tasks.rados.rados.0.smithi160.stdout:3139: seq_num 1367 ranges {575169=492570,1793826=417464,2623584=351585} 2024-09-21T18:09:46.192 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:46.182+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:46.193 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3139: writing smithi16030472-363 from 575169 to 1067739 tid 1 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3139: writing smithi16030472-363 from 1793826 to 2211290 tid 2 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3139: writing smithi16030472-363 from 2623584 to 2975169 tid 3 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3140: cache_evict oid 284 snap -1 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3141: read oid 328 snap -1 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3141: expect (ObjNum 831 snap 0 seq_num 831) 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3142: copy_from oid 61 from oid 124 current snap is 0 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3132: got (16) Device or resource busy 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3132: done (15 left) 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3143: copy_from oid 327 from oid 481 current snap is 0 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:46.195 INFO:tasks.rados.rados.0.smithi160.stdout:3136: got (16) Device or resource busy 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3136: done (15 left) 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3144: write oid 137 current snap is 0 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3144: seq_num 1368 ranges {591976=567819,1852308=677475} 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3144: writing smithi16030472-137 from 591976 to 1159795 tid 1 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3144: writing smithi16030472-137 from 1852308 to 2529783 tid 2 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:46.196 INFO:tasks.rados.rados.0.smithi160.stdout:3130: expect (ObjNum 1315 snap 0 seq_num 1315) 2024-09-21T18:09:46.288 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:46.285+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:46.320 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:46.321 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:46.335 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:46.339 INFO:tasks.rados.rados.0.smithi160.stdout:3134: got (0) Success 2024-09-21T18:09:46.339 INFO:tasks.rados.rados.0.smithi160.stdout:3139: finishing write tid 1 to smithi16030472-363 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3139: finishing write tid 2 to smithi16030472-363 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3139: finishing write tid 3 to smithi16030472-363 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3140: got (16) Device or resource busy 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3130: done (15 left) 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3134: done (14 left) 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3140: done (13 left) 2024-09-21T18:09:46.340 INFO:tasks.rados.rados.0.smithi160.stdout:3145: delete oid 171 current snap is 0 2024-09-21T18:09:46.374 INFO:tasks.rados.rados.0.smithi160.stdout:3139: finishing write tid 4 to smithi16030472-363 2024-09-21T18:09:46.374 INFO:tasks.rados.rados.0.smithi160.stdout:3139: finishing write tid 5 to smithi16030472-363 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:3139: oid 363 updating version 0 to 942 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:3139: oid 363 version 942 is already newer than 941 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:3139: oid 363 updating version 942 to 944 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:3139: oid 363 version 944 is already newer than 943 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 363 v 944 (ObjNum 1367 snap 0 seq_num 1367) dirty exists 2024-09-21T18:09:46.375 INFO:tasks.rados.rados.0.smithi160.stdout:3139: left oid 363 (ObjNum 1367 snap 0 seq_num 1367) 2024-09-21T18:09:46.389 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:46.384+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:46.464 INFO:tasks.rados.rados.0.smithi160.stdout:3126: expect (ObjNum 1110 snap 0 seq_num 1110) 2024-09-21T18:09:46.464 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:46.469 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:46.469 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:46.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:46.489 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:46.482+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:46.545 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:46.590 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:46.585+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:46.612 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1131450 2024-09-21T18:09:46.616 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:46.617 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:46.617 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:46.617 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:46.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:46.652 INFO:tasks.rados.rados.0.smithi160.stdout:3125: finishing write tid 1 to smithi16030472-359 2024-09-21T18:09:46.652 INFO:tasks.rados.rados.0.smithi160.stdout:3127: finishing write tid 1 to smithi16030472-66 2024-09-21T18:09:46.652 INFO:tasks.rados.rados.0.smithi160.stdout:3127: finishing write tid 2 to smithi16030472-66 2024-09-21T18:09:46.652 INFO:tasks.rados.rados.0.smithi160.stdout:3125: finishing write tid 2 to smithi16030472-359 2024-09-21T18:09:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:3131: got (0) Success 2024-09-21T18:09:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 56 v 0 (ObjNum 939 snap 0 seq_num 939) dirty exists 2024-09-21T18:09:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:3133: got (0) Success 2024-09-21T18:09:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 0 (ObjNum 935 snap 0 seq_num 935) dirty exists 2024-09-21T18:09:46.653 INFO:tasks.rados.rados.0.smithi160.stdout:3138: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:09:46.658 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 618780 2024-09-21T18:09:46.684 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:46.684 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:46.690 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:46.688+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:46.708 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:46.709 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:46.736 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:46.736 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:46.761 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:46.761 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:46.771 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:46.774 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2161224 2024-09-21T18:09:46.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:3125: finishing write tid 3 to smithi16030472-359 2024-09-21T18:09:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:3125: finishing write tid 4 to smithi16030472-359 2024-09-21T18:09:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:3125: oid 359 updating version 0 to 952 2024-09-21T18:09:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:3125: oid 359 version 952 is already newer than 950 2024-09-21T18:09:46.791 INFO:tasks.rados.rados.0.smithi160.stdout:3125: oid 359 version 952 is already newer than 951 2024-09-21T18:09:46.792 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 952 (ObjNum 1364 snap 0 seq_num 1364) dirty exists 2024-09-21T18:09:46.792 INFO:tasks.rados.rados.0.smithi160.stdout:3125: left oid 359 (ObjNum 1364 snap 0 seq_num 1364) 2024-09-21T18:09:46.792 INFO:tasks.rados.rados.0.smithi160.stdout:3141: expect (ObjNum 831 snap 0 seq_num 831) 2024-09-21T18:09:46.792 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:46.797 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:46.798 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:46.793+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:46.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:46.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:46.870 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:46.893 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:46.889+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:46.896 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:46.897 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:46.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:46.897 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:09:46.897 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:46.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:46.994 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:46.988+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:47.025 INFO:tasks.rados.rados.0.smithi160.stdout:3137: expect (ObjNum 1199 snap 0 seq_num 1199) 2024-09-21T18:09:47.048 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:47.075 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1513718 2024-09-21T18:09:47.085 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.086 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:47.086 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:47.086 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.088 INFO:tasks.rados.rados.0.smithi160.stdout:3127: finishing write tid 3 to smithi16030472-66 2024-09-21T18:09:47.088 INFO:tasks.rados.rados.0.smithi160.stdout:3142: finishing copy_from racing read to smithi16030472-61 2024-09-21T18:09:47.088 INFO:tasks.rados.rados.0.smithi160.stdout:3135: finishing write tid 1 to smithi16030472-117 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: finishing write tid 2 to smithi16030472-117 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: finishing write tid 3 to smithi16030472-117 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: finishing write tid 4 to smithi16030472-117 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: finishing write tid 5 to smithi16030472-117 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: oid 117 updating version 0 to 903 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: oid 117 version 903 is already newer than 901 2024-09-21T18:09:47.089 INFO:tasks.rados.rados.0.smithi160.stdout:3135: oid 117 version 903 is already newer than 902 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3135: oid 117 updating version 903 to 904 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 117 v 904 (ObjNum 1366 snap 0 seq_num 1366) dirty exists 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3135: left oid 117 (ObjNum 1366 snap 0 seq_num 1366) 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: finishing write tid 1 to smithi16030472-137 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: finishing write tid 2 to smithi16030472-137 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: finishing write tid 3 to smithi16030472-137 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: finishing write tid 4 to smithi16030472-137 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: oid 137 updating version 0 to 955 2024-09-21T18:09:47.090 INFO:tasks.rados.rados.0.smithi160.stdout:3144: oid 137 version 955 is already newer than 954 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:3144: oid 137 version 955 is already newer than 953 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 137 v 955 (ObjNum 1368 snap 0 seq_num 1368) dirty exists 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:3144: left oid 137 (ObjNum 1368 snap 0 seq_num 1368) 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:3142: finishing copy_from to smithi16030472-61 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 61 v 665 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:09:47.091 INFO:tasks.rados.rados.0.smithi160.stdout:3127: finishing write tid 4 to smithi16030472-66 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: finishing write tid 5 to smithi16030472-66 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: oid 66 updating version 0 to 939 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: oid 66 version 939 is already newer than 938 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: oid 66 updating version 939 to 956 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: oid 66 version 956 is already newer than 940 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 66 v 956 (ObjNum 1365 snap 0 seq_num 1365) dirty exists 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3127: left oid 66 (ObjNum 1365 snap 0 seq_num 1365) 2024-09-21T18:09:47.092 INFO:tasks.rados.rados.0.smithi160.stdout:3143: finishing copy_from racing read to smithi16030472-327 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3143: finishing copy_from to smithi16030472-327 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 327 v 815 (ObjNum 1205 snap 0 seq_num 1205) dirty exists 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3125: done (13 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3126: done (12 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3127: done (11 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3131: done (10 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3133: done (9 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3135: done (8 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3137: done (7 left) 2024-09-21T18:09:47.093 INFO:tasks.rados.rados.0.smithi160.stdout:3138: done (6 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3139: done (5 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3141: done (4 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3142: done (3 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3143: done (2 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3144: done (1 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3145: done (0 left) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3146: copy_from oid 369 from oid 92 current snap is 0 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3147: read oid 432 snap -1 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3147: expect (ObjNum 1176 snap 0 seq_num 1176) 2024-09-21T18:09:47.094 INFO:tasks.rados.rados.0.smithi160.stdout:3148: cache_try_flush oid 160 snap -1 2024-09-21T18:09:47.095 INFO:tasks.rados.rados.0.smithi160.stdout:3149: copy_from oid 400 from oid 489 current snap is 0 2024-09-21T18:09:47.095 INFO:tasks.rados.rados.0.smithi160.stdout:3150: cache_flush oid 494 snap -1 2024-09-21T18:09:47.095 INFO:tasks.rados.rados.0.smithi160.stdout:3151: cache_evict oid 361 snap -1 2024-09-21T18:09:47.095 INFO:tasks.rados.rados.0.smithi160.stdout:3152: delete oid 23 current snap is 0 2024-09-21T18:09:47.096 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:47.090+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:47.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:47.116 INFO:tasks.rados.rados.0.smithi160.stdout:3149: finishing copy_from racing read to smithi16030472-400 2024-09-21T18:09:47.188 INFO:tasks.rados.rados.0.smithi160.stdout:3148: got (2) No such file or directory 2024-09-21T18:09:47.195 INFO:tasks.rados.rados.0.smithi160.stdout:3147: expect (ObjNum 1176 snap 0 seq_num 1176) 2024-09-21T18:09:47.197 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:47.196+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:47.228 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.228 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:47.228 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:47.228 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:47.228 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:47.257 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:47.297 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:47.293+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:47.303 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:3150: got (0) Success 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 494 v 0 (ObjNum 1142 snap 0 seq_num 1142) dirty exists 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:3151: got (16) Device or resource busy 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:3149: finishing copy_from to smithi16030472-400 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 400 v 407 (ObjNum 488 snap 0 seq_num 488) dirty exists 2024-09-21T18:09:47.344 INFO:tasks.rados.rados.0.smithi160.stdout:3146: finishing copy_from racing read to smithi16030472-369 2024-09-21T18:09:47.374 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.374 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:47.374 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 154.16058382700001, 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.375 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:09:47.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:09:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:09:47.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:09:47.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:09:47.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.61664627799999, 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:09:47.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:09:47.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:09:47.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.45572281400001, 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.386 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:09:47.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.45504821899999, 2024-09-21T18:09:47.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:09:47.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.394 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.45502168100001, 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:09:47.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:09:47.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.44518793899999, 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:47.402 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.43277615100001, 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:47.408 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.42388608100001, 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:09:47.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.414 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.42307406399999, 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:09:47.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:09:47.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:09:47.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:09:47.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.420 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.422965852, 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:09:47.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:09:47.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:09:47.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.32269470599999, 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.426 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.32179844699999, 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.430 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:09:47.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:09:47.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.32131683399999, 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.434 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:09:47.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.320864409, 2024-09-21T18:09:47.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:09:47.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:09:47.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.32081408299999, 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.19508130299999, 2024-09-21T18:09:47.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.190803246, 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:09:47.452 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:09:47.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:09:47.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:09:47.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.458 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.19043300600001, 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:09:47.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:09:47.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:09:47.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:09:47.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.464 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.190363276, 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:09:47.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:09:47.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 135.59878628600001, 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:47.469 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:47.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:47.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:09:47.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:09:47.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:47.474 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:47.478 INFO:tasks.rados.rados.0.smithi160.stdout:3147: done (6 left) 2024-09-21T18:09:47.478 INFO:tasks.rados.rados.0.smithi160.stdout:3148: done (5 left) 2024-09-21T18:09:47.478 INFO:tasks.rados.rados.0.smithi160.stdout:3149: done (4 left) 2024-09-21T18:09:47.478 INFO:tasks.rados.rados.0.smithi160.stdout:3150: done (3 left) 2024-09-21T18:09:47.478 INFO:tasks.rados.rados.0.smithi160.stdout:3151: done (2 left) 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3152: done (1 left) 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3153: cache_evict oid 324 snap -1 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3154: copy_from oid 452 from oid 118 current snap is 0 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3155: cache_flush oid 295 snap -1 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3156: read oid 238 snap -1 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3156: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3157: write (excl) oid 333 current snap is 0 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3146: finishing copy_from to smithi16030472-369 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 369 v 541 (ObjNum 718 snap 0 seq_num 718) dirty exists 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3157: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3157: writing smithi16030472-333 from 642397 to 1152316 tid 1 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3157: writing smithi16030472-333 from 1627754 to 2052556 tid 2 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3157: writing smithi16030472-333 from 2733276 to 3042397 tid 3 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3146: done (5 left) 2024-09-21T18:09:47.479 INFO:tasks.rados.rados.0.smithi160.stdout:3158: cache_evict oid 102 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3159: read oid 233 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3159: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3160: read oid 367 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3160: expect (ObjNum 968 snap 0 seq_num 968) 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3153: got (0) Success 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3161: read oid 126 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3161: expect deleted 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3153: done (8 left) 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3162: cache_try_flush oid 112 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3163: read oid 326 snap -1 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3163: expect deleted 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3164: copy_from oid 37 from oid 240 current snap is 0 2024-09-21T18:09:47.480 INFO:tasks.rados.rados.0.smithi160.stdout:3165: delete oid 99 current snap is 0 2024-09-21T18:09:47.480 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:47.473+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:47.546 INFO:tasks.rados.rados.0.smithi160.stdout:3160: expect (ObjNum 968 snap 0 seq_num 968) 2024-09-21T18:09:47.546 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:47.577 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:47.573+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:47.611 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.611 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:47.611 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:47.611 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:47.658 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:47.677 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:47.673+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3162: got (0) Success 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 112 v 0 (ObjNum 81 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3154: finishing copy_from to smithi16030472-452 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3154: got expected ENOENT (src dne) 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3154: finishing copy_from racing read to smithi16030472-452 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3157: finishing write tid 1 to smithi16030472-333 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3158: got (16) Device or resource busy 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3164: finishing copy_from to smithi16030472-37 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3164: got expected ENOENT (src dne) 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3164: finishing copy_from racing read to smithi16030472-37 2024-09-21T18:09:47.698 INFO:tasks.rados.rados.0.smithi160.stdout:3157: finishing write tid 2 to smithi16030472-333 2024-09-21T18:09:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:3157: finishing write tid 3 to smithi16030472-333 2024-09-21T18:09:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:3159: expect (ObjNum 232 snap 0 seq_num 232) 2024-09-21T18:09:47.699 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 379767 2024-09-21T18:09:47.743 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1312445 2024-09-21T18:09:47.778 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:47.773+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:47.796 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:47.797 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.800 INFO:tasks.rados.rados.0.smithi160.stdout:3155: got (0) Success 2024-09-21T18:09:47.800 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 257 snap 0 seq_num 3132096704) clean dne 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: finishing write tid 4 to smithi16030472-333 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: finishing write tid 5 to smithi16030472-333 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: oid 333 updating version 0 to 958 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: oid 333 updating version 958 to 959 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: oid 333 version 959 is already newer than 957 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: oid 333 updating version 959 to 960 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 333 v 960 (ObjNum 1371 snap 0 seq_num 1371) dirty exists 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: left oid 333 (ObjNum 1371 snap 0 seq_num 1371) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3154: done (11 left) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3155: done (10 left) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3157: done (9 left) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3158: done (8 left) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3159: done (7 left) 2024-09-21T18:09:47.801 INFO:tasks.rados.rados.0.smithi160.stdout:3160: done (6 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3161: done (5 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3162: done (4 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3163: done (3 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3164: done (2 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3165: done (1 left) 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3166: cache_evict oid 153 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3167: cache_evict oid 385 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3168: cache_flush oid 475 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3169: cache_try_flush oid 47 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3170: cache_flush oid 351 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3171: cache_try_flush oid 276 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3172: cache_try_flush oid 39 snap -1 2024-09-21T18:09:47.802 INFO:tasks.rados.rados.0.smithi160.stdout:3173: delete oid 23 current snap is 0 2024-09-21T18:09:47.803 INFO:tasks.rados.rados.0.smithi160.stdout:3168: got (2) No such file or directory 2024-09-21T18:09:47.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:47.880 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:47.874+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:47.901 INFO:tasks.rados.rados.0.smithi160.stdout:3167: got (16) Device or resource busy 2024-09-21T18:09:47.933 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:47.933 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:47.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:47.933 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:09:47.933 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:47.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:09:47.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:47.981 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:47.976+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:48.003 INFO:tasks.rados.rados.0.smithi160.stdout:3166: got (0) Success 2024-09-21T18:09:48.016 INFO:tasks.rados.rados.0.smithi160.stdout:3156: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:48.058 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2067344 2024-09-21T18:09:48.082 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:48.078+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:48.090 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 493568 2024-09-21T18:09:48.117 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:48.117 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:48.118 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:48.118 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:48.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:48.142 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:48.196 INFO:tasks.rados.rados.0.smithi160.stdout:3169: got (0) Success 2024-09-21T18:09:48.196 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 0 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:09:48.196 INFO:tasks.rados.rados.0.smithi160.stdout:3156: done (8 left) 2024-09-21T18:09:48.196 INFO:tasks.rados.rados.0.smithi160.stdout:3166: done (7 left) 2024-09-21T18:09:48.196 INFO:tasks.rados.rados.0.smithi160.stdout:3167: done (6 left) 2024-09-21T18:09:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:3168: done (5 left) 2024-09-21T18:09:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:3169: done (4 left) 2024-09-21T18:09:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:3173: done (3 left) 2024-09-21T18:09:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:3174: write oid 68 current snap is 0 2024-09-21T18:09:48.197 INFO:tasks.rados.rados.0.smithi160.stdout:3174: seq_num 1374 ranges {692818=735666,2203200=688482} 2024-09-21T18:09:48.219 INFO:tasks.rados.rados.0.smithi160.stdout:3174: writing smithi16030472-68 from 692818 to 1428484 tid 1 2024-09-21T18:09:48.243 INFO:tasks.rados.rados.0.smithi160.stdout:3174: writing smithi16030472-68 from 2203200 to 2891682 tid 2 2024-09-21T18:09:48.243 INFO:tasks.rados.rados.0.smithi160.stdout:3175: cache_try_flush oid 385 snap -1 2024-09-21T18:09:48.243 INFO:tasks.rados.rados.0.smithi160.stdout:3171: got (0) Success 2024-09-21T18:09:48.244 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 276 v 0 (ObjNum 1268 snap 0 seq_num 1268) dirty exists 2024-09-21T18:09:48.244 INFO:tasks.rados.rados.0.smithi160.stdout:3171: done (4 left) 2024-09-21T18:09:48.244 INFO:tasks.rados.rados.0.smithi160.stdout:3176: read oid 77 snap -1 2024-09-21T18:09:48.244 INFO:tasks.rados.rados.0.smithi160.stdout:3176: expect (ObjNum 1240 snap 0 seq_num 1240) 2024-09-21T18:09:48.244 INFO:tasks.rados.rados.0.smithi160.stdout:3177: delete oid 202 current snap is 0 2024-09-21T18:09:48.250 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:09:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:09:48.250 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:48.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:48.284 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:48.278+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:48.322 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:48.181+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:48.322 INFO:tasks.rados.rados.0.smithi160.stdout:3170: got (0) Success 2024-09-21T18:09:48.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 351 v 0 (ObjNum 1166 snap 0 seq_num 1166) dirty exists 2024-09-21T18:09:48.322 INFO:tasks.rados.rados.0.smithi160.stdout:3172: got (0) Success 2024-09-21T18:09:48.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 0 (ObjNum 1301 snap 0 seq_num 1301) dirty exists 2024-09-21T18:09:48.331 INFO:tasks.rados.rados.0.smithi160.stdout:3176: expect (ObjNum 1240 snap 0 seq_num 1240) 2024-09-21T18:09:48.386 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:48.382+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:48.391 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 182.49387375800001, 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:48.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:09:48.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:48.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:48.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 181.40848136899999, 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:09:48.399 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:48.400 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:48.400 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:48.400 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:09:48.400 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:09:48.400 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:48.406 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:48.420 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2742309 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: finishing write tid 1 to smithi16030472-68 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: finishing write tid 2 to smithi16030472-68 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: finishing write tid 3 to smithi16030472-68 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: finishing write tid 4 to smithi16030472-68 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: oid 68 updating version 0 to 982 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: oid 68 updating version 982 to 983 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: oid 68 updating version 983 to 984 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 68 v 984 (ObjNum 1374 snap 0 seq_num 1374) dirty exists 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: left oid 68 (ObjNum 1374 snap 0 seq_num 1374) 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3170: done (5 left) 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3172: done (4 left) 2024-09-21T18:09:48.458 INFO:tasks.rados.rados.0.smithi160.stdout:3174: done (3 left) 2024-09-21T18:09:48.459 INFO:tasks.rados.rados.0.smithi160.stdout:3176: done (2 left) 2024-09-21T18:09:48.459 INFO:tasks.rados.rados.0.smithi160.stdout:3177: done (1 left) 2024-09-21T18:09:48.459 INFO:tasks.rados.rados.0.smithi160.stdout:3178: write oid 488 current snap is 0 2024-09-21T18:09:48.459 INFO:tasks.rados.rados.0.smithi160.stdout:3178: seq_num 1376 ranges {726432=486164,1803811=692151} 2024-09-21T18:09:48.477 INFO:tasks.rados.rados.0.smithi160.stdout:3178: writing smithi16030472-488 from 726432 to 1212596 tid 1 2024-09-21T18:09:48.487 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:48.482+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3178: writing smithi16030472-488 from 1803811 to 2495962 tid 2 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3179: cache_flush oid 465 snap -1 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3175: got (0) Success 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 0 (ObjNum 1213 snap 0 seq_num 1213) dirty exists 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3175: done (2 left) 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3180: cache_try_flush oid 139 snap -1 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3181: copy_from oid 460 from oid 481 current snap is 0 2024-09-21T18:09:48.499 INFO:tasks.rados.rados.0.smithi160.stdout:3182: delete oid 361 current snap is 0 2024-09-21T18:09:48.500 INFO:tasks.rados.rados.0.smithi160.stdout:3179: got (0) Success 2024-09-21T18:09:48.500 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 465 v 0 (ObjNum 1203 snap 0 seq_num 1203) dirty exists 2024-09-21T18:09:48.541 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:48.541 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:48.541 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:48.541 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:48.555 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:48.581 INFO:tasks.rados.rados.0.smithi160.stdout:3180: got (0) Success 2024-09-21T18:09:48.581 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 139 v 0 (ObjNum 978 snap 0 seq_num 978) dirty exists 2024-09-21T18:09:48.588 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:48.585+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:48.683 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:48.683 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:48.684 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:48.684 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:48.685 INFO:tasks.rados.rados.0.smithi160.stdout:3179: done (4 left) 2024-09-21T18:09:48.685 INFO:tasks.rados.rados.0.smithi160.stdout:3180: done (3 left) 2024-09-21T18:09:48.685 INFO:tasks.rados.rados.0.smithi160.stdout:3182: done (2 left) 2024-09-21T18:09:48.685 INFO:tasks.rados.rados.0.smithi160.stdout:3183: write oid 199 current snap is 0 2024-09-21T18:09:48.685 INFO:tasks.rados.rados.0.smithi160.stdout:3183: seq_num 1378 ranges {760046=753015,2037128=695820} 2024-09-21T18:09:48.689 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:48.685+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:48.698 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:48.712 INFO:tasks.rados.rados.0.smithi160.stdout:3183: writing smithi16030472-199 from 760046 to 1513061 tid 1 2024-09-21T18:09:48.733 INFO:tasks.rados.rados.0.smithi160.stdout:3183: writing smithi16030472-199 from 2037128 to 2732948 tid 2 2024-09-21T18:09:48.733 INFO:tasks.rados.rados.0.smithi160.stdout:3184: delete oid 289 current snap is 0 2024-09-21T18:09:48.733 INFO:tasks.rados.rados.0.smithi160.stdout:3178: finishing write tid 1 to smithi16030472-488 2024-09-21T18:09:48.733 INFO:tasks.rados.rados.0.smithi160.stdout:3181: finishing copy_from racing read to smithi16030472-460 2024-09-21T18:09:48.790 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:48.785+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:48.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.829 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.78754403299999, 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.830 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.831 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.832 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.833 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.834 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.787304409, 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.835 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.836 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:09:48.837 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:09:48.838 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:09:48.839 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.78726900199999, 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.840 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:09:48.841 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.842 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.843 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.787235829, 2024-09-21T18:09:48.844 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.845 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.846 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.847 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.848 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.787206269, 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.849 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.850 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.851 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.852 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.78717661300001, 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.853 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.854 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.855 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.856 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.857 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.786690357, 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.858 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:09:48.859 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:09:48.860 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:09:48.861 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.862 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.78665676399999, 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.863 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.864 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.865 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.866 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.68643907200001, 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.867 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:09:48.868 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:09:48.869 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:09:48.870 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.871 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.872 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.685840168, 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.873 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:09:48.874 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:09:48.875 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:09:48.876 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.877 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.685125551, 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.878 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.879 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.880 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.881 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.882 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.67587281900001, 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.883 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.884 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.885 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.886 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.887 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.67560811800001, 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.888 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:09:48.889 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:09:48.890 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:09:48.891 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.67551342900001, 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:09:48.892 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.893 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.894 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.895 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.896 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.67547213899999, 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.897 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:09:48.898 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.899 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.900 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.66420223599999, 2024-09-21T18:09:48.901 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.902 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:09:48.903 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:09:48.904 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.905 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.906 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.65508600300001, 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.907 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.908 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.909 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.910 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.911 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.65402263600001, 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.912 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.913 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.914 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.915 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.916 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 137.653982326, 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.917 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:09:48.918 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:09:48.919 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:09:48.920 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.921 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 136.891344484, 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.922 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:48.923 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:48.924 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:09:48.925 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:48.926 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:3178: finishing write tid 2 to smithi16030472-488 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:3178: finishing write tid 3 to smithi16030472-488 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:3181: finishing copy_from to smithi16030472-460 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 460 v 815 (ObjNum 1205 snap 0 seq_num 1205) dirty exists 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:3178: finishing write tid 4 to smithi16030472-488 2024-09-21T18:09:48.927 INFO:tasks.rados.rados.0.smithi160.stdout:3178: oid 488 updating version 0 to 985 2024-09-21T18:09:48.928 INFO:tasks.rados.rados.0.smithi160.stdout:3178: oid 488 version 985 is already newer than 938 2024-09-21T18:09:48.928 INFO:tasks.rados.rados.0.smithi160.stdout:3178: oid 488 updating version 985 to 986 2024-09-21T18:09:48.928 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 488 v 986 (ObjNum 1376 snap 0 seq_num 1376) dirty exists 2024-09-21T18:09:48.928 INFO:tasks.rados.rados.0.smithi160.stdout:3178: left oid 488 (ObjNum 1376 snap 0 seq_num 1376) 2024-09-21T18:09:48.928 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:48.929 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:48.925+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:48.935 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:09:48.939 INFO:tasks.rados.rados.0.smithi160.stdout:3178: done (3 left) 2024-09-21T18:09:48.939 INFO:tasks.rados.rados.0.smithi160.stdout:3181: done (2 left) 2024-09-21T18:09:48.939 INFO:tasks.rados.rados.0.smithi160.stdout:3184: done (1 left) 2024-09-21T18:09:48.939 INFO:tasks.rados.rados.0.smithi160.stdout:3185: write (excl) oid 404 current snap is 0 2024-09-21T18:09:48.939 INFO:tasks.rados.rados.0.smithi160.stdout:3185: seq_num 1380 ranges {793660=503513,2037739=583136} 2024-09-21T18:09:48.957 INFO:tasks.rados.rados.0.smithi160.stdout:3185: writing smithi16030472-404 from 793660 to 1297173 tid 1 2024-09-21T18:09:48.961 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:48.961 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:48.976 INFO:tasks.rados.rados.0.smithi160.stdout:3185: writing smithi16030472-404 from 2037739 to 2620875 tid 2 2024-09-21T18:09:48.976 INFO:tasks.rados.rados.0.smithi160.stdout:3186: copy_from oid 78 from oid 213 current snap is 0 2024-09-21T18:09:48.976 INFO:tasks.rados.rados.0.smithi160.stdout:3185: finishing write tid 1 to smithi16030472-404 2024-09-21T18:09:48.976 INFO:tasks.rados.rados.0.smithi160.stdout:3187: read oid 67 snap -1 2024-09-21T18:09:48.976 INFO:tasks.rados.rados.0.smithi160.stdout:3187: expect deleted 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3188: read oid 49 snap -1 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3188: expect (ObjNum 1326 snap 0 seq_num 1326) 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3189: copy_from oid 5 from oid 168 current snap is 0 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3190: read oid 326 snap -1 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3190: expect deleted 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3191: read oid 182 snap -1 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3191: expect (ObjNum 971 snap 0 seq_num 971) 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3192: cache_try_flush oid 440 snap -1 2024-09-21T18:09:48.977 INFO:tasks.rados.rados.0.smithi160.stdout:3193: delete oid 247 current snap is 0 2024-09-21T18:09:48.980 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:48.980 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:49.000 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:49.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:49.022 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:49.022 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:49.028 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:49.022+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:49.062 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:49.062 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:49.062 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:49.062 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:49.077 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:49.078 INFO:tasks.rados.rados.0.smithi160.stdout:3192: got (0) Success 2024-09-21T18:09:49.079 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 440 v 0 (ObjNum 129 snap 0 seq_num 3126305256) clean dne 2024-09-21T18:09:49.090 INFO:tasks.rados.rados.0.smithi160.stdout:3188: expect (ObjNum 1326 snap 0 seq_num 1326) 2024-09-21T18:09:49.108 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 597109 2024-09-21T18:09:49.129 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:49.124+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:49.174 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:49.166+0000 7f7283507640 -1 osd.14 114 handle_osd_map injecting map crc failure 2024-09-21T18:09:49.174 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:49.169+0000 7f094d1de640 -1 osd.0 114 handle_osd_map injecting map crc failure 2024-09-21T18:09:49.190 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:49.190 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:49.193 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 152313 2024-09-21T18:09:49.207 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:49.207 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:49.207 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:49.222 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:49.223 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:49.263 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:49.263 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:49.263 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:49.271 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:49.268+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:49.295 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:49.308 INFO:tasks.rados.rados.0.smithi160.stdout:3183: finishing write tid 1 to smithi16030472-199 2024-09-21T18:09:49.308 INFO:tasks.rados.rados.0.smithi160.stdout:3185: finishing write tid 2 to smithi16030472-404 2024-09-21T18:09:49.308 INFO:tasks.rados.rados.0.smithi160.stdout:3185: finishing write tid 3 to smithi16030472-404 2024-09-21T18:09:49.308 INFO:tasks.rados.rados.0.smithi160.stdout:3185: finishing write tid 4 to smithi16030472-404 2024-09-21T18:09:49.308 INFO:tasks.rados.rados.0.smithi160.stdout:3185: oid 404 updating version 0 to 989 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3185: oid 404 version 989 is already newer than 988 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3185: oid 404 updating version 989 to 990 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 404 v 990 (ObjNum 1380 snap 0 seq_num 1380) dirty exists 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3185: left oid 404 (ObjNum 1380 snap 0 seq_num 1380) 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3186: finishing copy_from racing read to smithi16030472-78 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3189: finishing copy_from racing read to smithi16030472-5 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3186: finishing copy_from to smithi16030472-78 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 78 v 150 (ObjNum 212 snap 0 seq_num 212) dirty exists 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3189: finishing copy_from to smithi16030472-5 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 5 v 919 (ObjNum 1307 snap 0 seq_num 1307) dirty exists 2024-09-21T18:09:49.309 INFO:tasks.rados.rados.0.smithi160.stdout:3191: expect (ObjNum 971 snap 0 seq_num 971) 2024-09-21T18:09:49.322 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1964705 2024-09-21T18:09:49.326 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:49.326 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:49.331 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:49.328+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:49.347 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:49.358 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:49.359 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3183: finishing write tid 2 to smithi16030472-199 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3183: finishing write tid 3 to smithi16030472-199 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3185: done (9 left) 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3186: done (8 left) 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3187: done (7 left) 2024-09-21T18:09:49.375 INFO:tasks.rados.rados.0.smithi160.stdout:3188: done (6 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3189: done (5 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3190: done (4 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3191: done (3 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3192: done (2 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3193: done (1 left) 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3194: cache_flush oid 499 snap -1 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3195: write oid 441 current snap is 0 2024-09-21T18:09:49.376 INFO:tasks.rados.rados.0.smithi160.stdout:3195: seq_num 1382 ranges {427274=654011,1754703=586805,2918576=308698} 2024-09-21T18:09:49.381 INFO:tasks.rados.rados.0.smithi160.stdout:3195: writing smithi16030472-441 from 427274 to 1081285 tid 1 2024-09-21T18:09:49.387 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:49.387 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:49.388 INFO:tasks.rados.rados.0.smithi160.stdout:3195: writing smithi16030472-441 from 1754703 to 2341508 tid 2 2024-09-21T18:09:49.393 INFO:tasks.rados.rados.0.smithi160.stdout:3195: writing smithi16030472-441 from 2918576 to 3227274 tid 3 2024-09-21T18:09:49.393 INFO:tasks.rados.rados.0.smithi160.stdout:3196: cache_flush oid 259 snap -1 2024-09-21T18:09:49.393 INFO:tasks.rados.rados.0.smithi160.stdout:3197: write (excl) oid 500 current snap is 0 2024-09-21T18:09:49.393 INFO:tasks.rados.rados.0.smithi160.stdout:3197: seq_num 1383 ranges {444081=729260,1813185=446816,2945999=298082} 2024-09-21T18:09:49.399 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:49.399 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:49.400 INFO:tasks.rados.rados.0.smithi160.stdout:3197: writing smithi16030472-500 from 444081 to 1173341 tid 1 2024-09-21T18:09:49.407 INFO:tasks.rados.rados.0.smithi160.stdout:3197: writing smithi16030472-500 from 1813185 to 2260001 tid 2 2024-09-21T18:09:49.414 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3197: writing smithi16030472-500 from 2945999 to 3244081 tid 3 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3198: delete oid 49 current snap is 0 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3196: got (0) Success 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 259 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3183: finishing write tid 4 to smithi16030472-199 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3183: oid 199 updating version 0 to 991 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3183: oid 199 version 991 is already newer than 939 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3183: oid 199 updating version 991 to 992 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 992 (ObjNum 1378 snap 0 seq_num 1378) dirty exists 2024-09-21T18:09:49.415 INFO:tasks.rados.rados.0.smithi160.stdout:3183: left oid 199 (ObjNum 1378 snap 0 seq_num 1378) 2024-09-21T18:09:49.416 INFO:tasks.rados.rados.0.smithi160.stdout:3194: got (0) Success 2024-09-21T18:09:49.416 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 0 (ObjNum 1049 snap 0 seq_num 1049) dirty exists 2024-09-21T18:09:49.416 INFO:tasks.rados.rados.0.smithi160.stdout:3197: finishing write tid 1 to smithi16030472-500 2024-09-21T18:09:49.416 INFO:tasks.rados.rados.0.smithi160.stdout:3197: finishing write tid 2 to smithi16030472-500 2024-09-21T18:09:49.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:49.417 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:49.432 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:49.429+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:49.445 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:49.446 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:49.474 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:49.476 INFO:tasks.rados.rados.0.smithi160.stdout:3195: finishing write tid 1 to smithi16030472-441 2024-09-21T18:09:49.502 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:49.503 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:49.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:49.518 INFO:tasks.rados.rados.0.smithi160.stdout:3197: finishing write tid 3 to smithi16030472-500 2024-09-21T18:09:49.519 INFO:tasks.rados.rados.0.smithi160.stdout:3197: finishing write tid 4 to smithi16030472-500 2024-09-21T18:09:49.533 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:49.526+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:49.585 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:49.585 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:49.599 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:49.619 INFO:tasks.rados.rados.0.smithi160.stdout:3197: finishing write tid 5 to smithi16030472-500 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:3197: oid 500 updating version 0 to 966 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:3197: oid 500 version 966 is already newer than 964 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:3197: oid 500 version 966 is already newer than 965 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:3197: oid 500 version 966 is already newer than 963 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 500 v 966 (ObjNum 1383 snap 0 seq_num 1383) dirty exists 2024-09-21T18:09:49.620 INFO:tasks.rados.rados.0.smithi160.stdout:3197: left oid 500 (ObjNum 1383 snap 0 seq_num 1383) 2024-09-21T18:09:49.634 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:49.630+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:49.719 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:49.719 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:49.719 INFO:tasks.rados.rados.0.smithi160.stdout:3195: finishing write tid 2 to smithi16030472-441 2024-09-21T18:09:49.719 INFO:tasks.rados.rados.0.smithi160.stdout:3195: finishing write tid 3 to smithi16030472-441 2024-09-21T18:09:49.733 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:49.734 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:49.732+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:49.821 INFO:tasks.rados.rados.0.smithi160.stdout:3195: finishing write tid 4 to smithi16030472-441 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: finishing write tid 5 to smithi16030472-441 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: oid 441 updating version 0 to 993 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: oid 441 updating version 993 to 996 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: oid 441 version 996 is already newer than 995 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: oid 441 version 996 is already newer than 994 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 441 v 996 (ObjNum 1382 snap 0 seq_num 1382) dirty exists 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: left oid 441 (ObjNum 1382 snap 0 seq_num 1382) 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3183: done (5 left) 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3194: done (4 left) 2024-09-21T18:09:49.822 INFO:tasks.rados.rados.0.smithi160.stdout:3195: done (3 left) 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3196: done (2 left) 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3197: done (1 left) 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3198: done (0 left) 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3199: read oid 386 snap -1 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3199: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3200: copy_from oid 498 from oid 278 current snap is 0 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3201: copy_from oid 485 from oid 184 current snap is 0 2024-09-21T18:09:49.823 INFO:tasks.rados.rados.0.smithi160.stdout:3202: delete oid 135 current snap is 0 2024-09-21T18:09:49.836 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:49.833+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:49.850 INFO:tasks.rados.rados.0.smithi160.stdout:3200: finishing copy_from racing read to smithi16030472-498 2024-09-21T18:09:49.851 INFO:tasks.rados.rados.0.smithi160.stdout:3200: finishing copy_from to smithi16030472-498 2024-09-21T18:09:49.851 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 498 v 754 (ObjNum 1043 snap 0 seq_num 1043) dirty exists 2024-09-21T18:09:49.855 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:49.855 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:49.855 INFO:tasks.rados.rados.0.smithi160.stdout:3201: finishing copy_from racing read to smithi16030472-485 2024-09-21T18:09:49.871 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:49.936 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:49.930+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:49.955 INFO:tasks.rados.rados.0.smithi160.stdout:3201: finishing copy_from to smithi16030472-485 2024-09-21T18:09:49.955 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 485 v 496 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:09:49.999 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:49.999 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.014 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:50.027 INFO:tasks.rados.rados.0.smithi160.stdout:3199: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:09:50.037 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:50.033+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:50.081 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2374996 2024-09-21T18:09:50.138 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.138 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.138 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:50.132+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:50.152 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:50.170 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2171478 2024-09-21T18:09:50.181 INFO:tasks.rados.rados.0.smithi160.stdout:3199: done (3 left) 2024-09-21T18:09:50.181 INFO:tasks.rados.rados.0.smithi160.stdout:3200: done (2 left) 2024-09-21T18:09:50.181 INFO:tasks.rados.rados.0.smithi160.stdout:3201: done (1 left) 2024-09-21T18:09:50.181 INFO:tasks.rados.rados.0.smithi160.stdout:3202: done (0 left) 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3203: read oid 484 snap -1 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3203: expect deleted 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3204: read oid 399 snap -1 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3204: expect (ObjNum 962 snap 0 seq_num 962) 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3205: copy_from oid 380 from oid 99 current snap is 0 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3206: write oid 499 current snap is 0 2024-09-21T18:09:50.182 INFO:tasks.rados.rados.0.smithi160.stdout:3206: seq_num 1386 ranges {494502=671360,1588631=594143,2628268=477951} 2024-09-21T18:09:50.197 INFO:tasks.rados.rados.0.smithi160.stdout:3206: writing smithi16030472-499 from 494502 to 1165862 tid 1 2024-09-21T18:09:50.211 INFO:tasks.rados.rados.0.smithi160.stdout:3206: writing smithi16030472-499 from 1588631 to 2182774 tid 2 2024-09-21T18:09:50.225 INFO:tasks.rados.rados.0.smithi160.stdout:3206: writing smithi16030472-499 from 2628268 to 3106219 tid 3 2024-09-21T18:09:50.225 INFO:tasks.rados.rados.0.smithi160.stdout:3207: cache_evict oid 54 snap -1 2024-09-21T18:09:50.225 INFO:tasks.rados.rados.0.smithi160.stdout:3208: write oid 274 current snap is 0 2024-09-21T18:09:50.225 INFO:tasks.rados.rados.0.smithi160.stdout:3208: seq_num 1387 ranges {511309=746609,2047113=454154,3055691=255618} 2024-09-21T18:09:50.240 INFO:tasks.rados.rados.0.smithi160.stdout:3208: writing smithi16030472-274 from 511309 to 1257918 tid 1 2024-09-21T18:09:50.240 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:50.234+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:50.252 INFO:tasks.rados.rados.0.smithi160.stdout:3208: writing smithi16030472-274 from 2047113 to 2501267 tid 2 2024-09-21T18:09:50.259 INFO:tasks.rados.rados.0.smithi160.stdout:3208: writing smithi16030472-274 from 3055691 to 3311309 tid 3 2024-09-21T18:09:50.259 INFO:tasks.rados.rados.0.smithi160.stdout:3209: write (excl) oid 475 current snap is 0 2024-09-21T18:09:50.259 INFO:tasks.rados.rados.0.smithi160.stdout:3209: seq_num 1388 ranges {528116=421858,1589242=597812,2966761=361355} 2024-09-21T18:09:50.266 INFO:tasks.rados.rados.0.smithi160.stdout:3209: writing smithi16030472-475 from 528116 to 949974 tid 1 2024-09-21T18:09:50.275 INFO:tasks.rados.rados.0.smithi160.stdout:3209: writing smithi16030472-475 from 1589242 to 2187054 tid 2 2024-09-21T18:09:50.282 INFO:tasks.rados.rados.0.smithi160.stdout:3209: writing smithi16030472-475 from 2966761 to 3328116 tid 3 2024-09-21T18:09:50.283 INFO:tasks.rados.rados.0.smithi160.stdout:3210: cache_try_flush oid 65 snap -1 2024-09-21T18:09:50.283 INFO:tasks.rados.rados.0.smithi160.stdout:3211: cache_flush oid 152 snap -1 2024-09-21T18:09:50.283 INFO:tasks.rados.rados.0.smithi160.stdout:3212: write oid 446 current snap is 0 2024-09-21T18:09:50.283 INFO:tasks.rados.rados.0.smithi160.stdout:3212: seq_num 1389 ranges {544923=497107,1647724=457823,2710537=428120} 2024-09-21T18:09:50.288 INFO:tasks.rados.rados.0.smithi160.stdout:3212: writing smithi16030472-446 from 544923 to 1042030 tid 1 2024-09-21T18:09:50.295 INFO:tasks.rados.rados.0.smithi160.stdout:3212: writing smithi16030472-446 from 1647724 to 2105547 tid 2 2024-09-21T18:09:50.301 INFO:tasks.rados.rados.0.smithi160.stdout:3212: writing smithi16030472-446 from 2710537 to 3138657 tid 3 2024-09-21T18:09:50.301 INFO:tasks.rados.rados.0.smithi160.stdout:3213: copy_from oid 112 from oid 323 current snap is 0 2024-09-21T18:09:50.301 INFO:tasks.rados.rados.0.smithi160.stdout:3214: copy_from oid 169 from oid 109 current snap is 0 2024-09-21T18:09:50.301 INFO:tasks.rados.rados.0.smithi160.stdout:3204: expect (ObjNum 962 snap 0 seq_num 962) 2024-09-21T18:09:50.301 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:50.323 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.323 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.337 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:50.342 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:50.338+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3206: finishing write tid 1 to smithi16030472-499 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3207: got (16) Device or resource busy 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3210: got (0) Success 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 65 v 0 (ObjNum 1277 snap 0 seq_num 1277) dirty exists 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3203: done (11 left) 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3204: done (10 left) 2024-09-21T18:09:50.439 INFO:tasks.rados.rados.0.smithi160.stdout:3207: done (9 left) 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3210: done (8 left) 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3215: cache_flush oid 150 snap -1 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3216: cache_try_flush oid 54 snap -1 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3217: copy_from oid 343 from oid 193 current snap is 0 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3218: read oid 282 snap -1 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3218: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3219: write (excl) oid 474 current snap is 0 2024-09-21T18:09:50.440 INFO:tasks.rados.rados.0.smithi160.stdout:3219: seq_num 1390 ranges {561730=572356,1706206=601481,3021607=340123} 2024-09-21T18:09:50.443 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:50.438+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:50.445 INFO:tasks.rados.rados.0.smithi160.stdout:3219: writing smithi16030472-474 from 561730 to 1134086 tid 1 2024-09-21T18:09:50.451 INFO:tasks.rados.rados.0.smithi160.stdout:3219: writing smithi16030472-474 from 1706206 to 2307687 tid 2 2024-09-21T18:09:50.457 INFO:tasks.rados.rados.0.smithi160.stdout:3219: writing smithi16030472-474 from 3021607 to 3361730 tid 3 2024-09-21T18:09:50.458 INFO:tasks.rados.rados.0.smithi160.stdout:3220: cache_try_flush oid 122 snap -1 2024-09-21T18:09:50.458 INFO:tasks.rados.rados.0.smithi160.stdout:3221: copy_from oid 336 from oid 250 current snap is 0 2024-09-21T18:09:50.458 INFO:tasks.rados.rados.0.smithi160.stdout:3222: write (excl) oid 385 current snap is 0 2024-09-21T18:09:50.458 INFO:tasks.rados.rados.0.smithi160.stdout:3222: seq_num 1391 ranges {578537=647605,1764688=745139,3049030=329507} 2024-09-21T18:09:50.464 INFO:tasks.rados.rados.0.smithi160.stdout:3222: writing smithi16030472-385 from 578537 to 1226142 tid 1 2024-09-21T18:09:50.472 INFO:tasks.rados.rados.0.smithi160.stdout:3222: writing smithi16030472-385 from 1764688 to 2509827 tid 2 2024-09-21T18:09:50.474 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.474 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3222: writing smithi16030472-385 from 3049030 to 3378537 tid 3 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3205: finishing copy_from to smithi16030472-380 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3205: got expected ENOENT (src dne) 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3205: finishing copy_from racing read to smithi16030472-380 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3205: done (15 left) 2024-09-21T18:09:50.478 INFO:tasks.rados.rados.0.smithi160.stdout:3223: delete oid 164 current snap is 0 2024-09-21T18:09:50.490 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:50.503 INFO:tasks.rados.rados.0.smithi160.stdout:3212: finishing write tid 1 to smithi16030472-446 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3213: finishing copy_from racing read to smithi16030472-112 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: finishing write tid 2 to smithi16030472-499 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: finishing write tid 3 to smithi16030472-499 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: finishing write tid 4 to smithi16030472-499 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: finishing write tid 5 to smithi16030472-499 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: oid 499 updating version 0 to 1000 2024-09-21T18:09:50.524 INFO:tasks.rados.rados.0.smithi160.stdout:3206: oid 499 updating version 1000 to 1001 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3206: oid 499 version 1001 is already newer than 999 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3206: oid 499 version 1001 is already newer than 998 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 499 v 1001 (ObjNum 1386 snap 0 seq_num 1386) dirty exists 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3206: left oid 499 (ObjNum 1386 snap 0 seq_num 1386) 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3208: finishing write tid 1 to smithi16030472-274 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3208: finishing write tid 2 to smithi16030472-274 2024-09-21T18:09:50.525 INFO:tasks.rados.rados.0.smithi160.stdout:3208: finishing write tid 3 to smithi16030472-274 2024-09-21T18:09:50.544 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:50.540+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:50.575 INFO:tasks.rados.rados.0.smithi160.stdout:3209: finishing write tid 1 to smithi16030472-475 2024-09-21T18:09:50.618 INFO:tasks.rados.rados.0.smithi160.stdout:3218: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T18:09:50.645 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:50.641+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:50.660 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.661 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.663 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1061874 2024-09-21T18:09:50.674 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:50.690 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: finishing write tid 4 to smithi16030472-274 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: finishing write tid 5 to smithi16030472-274 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: oid 274 updating version 0 to 1002 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: oid 274 updating version 1002 to 1004 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: oid 274 version 1004 is already newer than 1003 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: oid 274 updating version 1004 to 1005 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 274 v 1005 (ObjNum 1387 snap 0 seq_num 1387) dirty exists 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3208: left oid 274 (ObjNum 1387 snap 0 seq_num 1387) 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3213: finishing copy_from to smithi16030472-112 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 112 v 9 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3217: finishing copy_from racing read to smithi16030472-343 2024-09-21T18:09:50.719 INFO:tasks.rados.rados.0.smithi160.stdout:3219: finishing write tid 1 to smithi16030472-474 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3219: finishing write tid 2 to smithi16030472-474 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3219: finishing write tid 3 to smithi16030472-474 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3214: finishing copy_from racing read to smithi16030472-169 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3220: got (0) Success 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 122 v 0 (ObjNum 742 snap 0 seq_num 742) dirty exists 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3212: finishing write tid 2 to smithi16030472-446 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3212: finishing write tid 3 to smithi16030472-446 2024-09-21T18:09:50.720 INFO:tasks.rados.rados.0.smithi160.stdout:3212: finishing write tid 4 to smithi16030472-446 2024-09-21T18:09:50.745 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:50.742+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:50.747 INFO:tasks.rados.rados.0.smithi160.stdout:3221: finishing copy_from racing read to smithi16030472-336 2024-09-21T18:09:50.747 INFO:tasks.rados.rados.0.smithi160.stdout:3219: finishing write tid 4 to smithi16030472-474 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: finishing write tid 5 to smithi16030472-474 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: oid 474 updating version 0 to 970 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: oid 474 version 970 is already newer than 968 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: oid 474 version 970 is already newer than 969 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: oid 474 version 970 is already newer than 967 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 474 v 970 (ObjNum 1390 snap 0 seq_num 1390) dirty exists 2024-09-21T18:09:50.748 INFO:tasks.rados.rados.0.smithi160.stdout:3219: left oid 474 (ObjNum 1390 snap 0 seq_num 1390) 2024-09-21T18:09:50.798 INFO:tasks.rados.rados.0.smithi160.stdout:3214: finishing copy_from to smithi16030472-169 2024-09-21T18:09:50.798 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 169 v 811 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:09:50.799 INFO:tasks.rados.rados.0.smithi160.stdout:3211: got (0) Success 2024-09-21T18:09:50.800 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 152 v 0 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-21T18:09:50.803 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.803 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: finishing write tid 5 to smithi16030472-446 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: oid 446 updating version 0 to 952 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: oid 446 updating version 952 to 954 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: oid 446 updating version 954 to 955 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: oid 446 version 955 is already newer than 953 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 446 v 955 (ObjNum 1389 snap 0 seq_num 1389) dirty exists 2024-09-21T18:09:50.806 INFO:tasks.rados.rados.0.smithi160.stdout:3212: left oid 446 (ObjNum 1389 snap 0 seq_num 1389) 2024-09-21T18:09:50.807 INFO:tasks.rados.rados.0.smithi160.stdout:3217: finishing copy_from to smithi16030472-343 2024-09-21T18:09:50.807 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 576 (ObjNum 845 snap 0 seq_num 845) dirty exists 2024-09-21T18:09:50.817 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:50.846 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:50.841+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:50.893 INFO:tasks.rados.rados.0.smithi160.stdout:3215: got (0) Success 2024-09-21T18:09:50.893 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 150 v 0 (ObjNum 631 snap 0 seq_num 631) dirty exists 2024-09-21T18:09:50.901 INFO:tasks.rados.rados.0.smithi160.stdout:3222: finishing write tid 1 to smithi16030472-385 2024-09-21T18:09:50.901 INFO:tasks.rados.rados.0.smithi160.stdout:3222: finishing write tid 2 to smithi16030472-385 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: finishing write tid 3 to smithi16030472-385 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: finishing write tid 4 to smithi16030472-385 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3221: finishing copy_from to smithi16030472-336 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 605 (ObjNum 813 snap 0 seq_num 813) dirty exists 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: finishing write tid 5 to smithi16030472-385 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: oid 385 updating version 0 to 973 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: oid 385 version 973 is already newer than 972 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: oid 385 version 973 is already newer than 971 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: oid 385 updating version 973 to 974 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 974 (ObjNum 1391 snap 0 seq_num 1391) dirty exists 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3222: left oid 385 (ObjNum 1391 snap 0 seq_num 1391) 2024-09-21T18:09:50.902 INFO:tasks.rados.rados.0.smithi160.stdout:3209: finishing write tid 2 to smithi16030472-475 2024-09-21T18:09:50.947 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:50.944+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:50.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:09:50.989 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:50.989 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:51.004 INFO:tasks.rados.rados.0.smithi160.stdout:3209: finishing write tid 3 to smithi16030472-475 2024-09-21T18:09:51.050 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:51.046+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:51.104 INFO:tasks.rados.rados.0.smithi160.stdout:3216: got (0) Success 2024-09-21T18:09:51.105 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 0 (ObjNum 1236 snap 0 seq_num 1236) dirty exists 2024-09-21T18:09:51.107 INFO:tasks.rados.rados.0.smithi160.stdout:3209: finishing write tid 4 to smithi16030472-475 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: finishing write tid 5 to smithi16030472-475 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: oid 475 updating version 0 to 829 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: oid 475 updating version 829 to 920 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: oid 475 updating version 920 to 922 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: oid 475 version 922 is already newer than 921 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 475 v 922 (ObjNum 1388 snap 0 seq_num 1388) dirty exists 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: left oid 475 (ObjNum 1388 snap 0 seq_num 1388) 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3206: done (15 left) 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3208: done (14 left) 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3209: done (13 left) 2024-09-21T18:09:51.109 INFO:tasks.rados.rados.0.smithi160.stdout:3211: done (12 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3212: done (11 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3213: done (10 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3214: done (9 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3215: done (8 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3216: done (7 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3217: done (6 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3218: done (5 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3219: done (4 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3220: done (3 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3221: done (2 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3222: done (1 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3223: done (0 left) 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3224: write (excl) oid 247 current snap is 0 2024-09-21T18:09:51.110 INFO:tasks.rados.rados.0.smithi160.stdout:3224: seq_num 1393 ranges {612151=514456,1881652=748808,3103876=308275} 2024-09-21T18:09:51.116 INFO:tasks.rados.rados.0.smithi160.stdout:3224: writing smithi16030472-247 from 612151 to 1126607 tid 1 2024-09-21T18:09:51.125 INFO:tasks.rados.rados.0.smithi160.stdout:3224: writing smithi16030472-247 from 1881652 to 2630460 tid 2 2024-09-21T18:09:51.126 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:51.127 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.130 INFO:tasks.rados.rados.0.smithi160.stdout:3224: writing smithi16030472-247 from 3103876 to 3412151 tid 3 2024-09-21T18:09:51.130 INFO:tasks.rados.rados.0.smithi160.stdout:3225: read oid 162 snap -1 2024-09-21T18:09:51.130 INFO:tasks.rados.rados.0.smithi160.stdout:3225: expect (ObjNum 946 snap 0 seq_num 946) 2024-09-21T18:09:51.130 INFO:tasks.rados.rados.0.smithi160.stdout:3226: delete oid 97 current snap is 0 2024-09-21T18:09:51.141 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:51.148 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:51.145+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:51.226 INFO:tasks.rados.rados.0.smithi160.stdout:3224: finishing write tid 1 to smithi16030472-247 2024-09-21T18:09:51.234 INFO:tasks.rados.rados.0.smithi160.stdout:3226: done (2 left) 2024-09-21T18:09:51.234 INFO:tasks.rados.rados.0.smithi160.stdout:3227: write (excl) oid 142 current snap is 0 2024-09-21T18:09:51.234 INFO:tasks.rados.rados.0.smithi160.stdout:3227: seq_num 1395 ranges {645765=664954,1998616=752477,3158722=287043} 2024-09-21T18:09:51.245 INFO:tasks.rados.rados.0.smithi160.stdout:3227: writing smithi16030472-142 from 645765 to 1310719 tid 1 2024-09-21T18:09:51.249 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:51.242+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:51.255 INFO:tasks.rados.rados.0.smithi160.stdout:3227: writing smithi16030472-142 from 1998616 to 2751093 tid 2 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3227: writing smithi16030472-142 from 3158722 to 3445765 tid 3 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3228: cache_evict oid 272 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3229: copy_from oid 33 from oid 353 current snap is 0 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3230: cache_flush oid 128 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3231: cache_try_flush oid 495 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3232: cache_evict oid 496 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3233: read oid 474 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3233: expect (ObjNum 1390 snap 0 seq_num 1390) 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3234: read oid 107 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3234: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3235: read oid 393 snap -1 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3235: expect (ObjNum 1141 snap 0 seq_num 1141) 2024-09-21T18:09:51.261 INFO:tasks.rados.rados.0.smithi160.stdout:3236: delete oid 451 current snap is 0 2024-09-21T18:09:51.307 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:51.307 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:51.327 INFO:tasks.rados.rados.0.smithi160.stdout:3224: finishing write tid 2 to smithi16030472-247 2024-09-21T18:09:51.332 INFO:tasks.rados.rados.0.smithi160.stdout:3224: finishing write tid 3 to smithi16030472-247 2024-09-21T18:09:51.340 INFO:tasks.rados.rados.0.smithi160.stdout:3225: expect (ObjNum 946 snap 0 seq_num 946) 2024-09-21T18:09:51.350 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:51.345+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:51.439 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2348406 2024-09-21T18:09:51.451 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:51.448+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:51.490 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:51.491 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.496 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1362240 2024-09-21T18:09:51.504 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:51.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:51.552 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:51.548+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:51.627 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:51.627 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:51.644 INFO:tasks.rados.rados.0.smithi160.stdout:3234: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:09:51.666 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:51.666 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:51.686 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:51.682+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:51.688 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:51.704 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:51.708 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:51.730 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:51.749 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:51.753 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:51.750+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:51.768 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:51.771 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:51.772 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:51.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:51.792 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:51.801 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2781054 2024-09-21T18:09:51.811 INFO:tasks.rados.rados.0.smithi160.stdout:3235: expect (ObjNum 1141 snap 0 seq_num 1141) 2024-09-21T18:09:51.814 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:51.815 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:51.836 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2745604 2024-09-21T18:09:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:51.836 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:51.855 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:51.852+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:51.856 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:51.856 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:51.879 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:51.879 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:51.885 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:51.900 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:51.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:51.925 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: finishing write tid 4 to smithi16030472-247 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: finishing write tid 5 to smithi16030472-247 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: oid 247 updating version 0 to 957 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: oid 247 updating version 957 to 958 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: oid 247 updating version 958 to 959 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: oid 247 updating version 959 to 960 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 960 (ObjNum 1393 snap 0 seq_num 1393) dirty exists 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3224: left oid 247 (ObjNum 1393 snap 0 seq_num 1393) 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3230: got (0) Success 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 128 v 0 (ObjNum -1858781984 snap 32538 seq_num 304) clean dne 2024-09-21T18:09:51.933 INFO:tasks.rados.rados.0.smithi160.stdout:3228: got (16) Device or resource busy 2024-09-21T18:09:51.934 INFO:tasks.rados.rados.0.smithi160.stdout:3232: got (16) Device or resource busy 2024-09-21T18:09:51.934 INFO:tasks.rados.rados.0.smithi160.stdout:3229: finishing copy_from racing read to smithi16030472-33 2024-09-21T18:09:51.934 INFO:tasks.rados.rados.0.smithi160.stdout:3227: finishing write tid 1 to smithi16030472-142 2024-09-21T18:09:51.934 INFO:tasks.rados.rados.0.smithi160.stdout:3233: expect (ObjNum 1390 snap 0 seq_num 1390) 2024-09-21T18:09:51.935 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 532796 2024-09-21T18:09:51.948 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:51.949 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:51.950 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:51.950 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:51.950 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:09:51.955 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:51.952+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:51.964 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:51.974 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:51.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:51.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:52.031 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 408983 2024-09-21T18:09:52.055 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:52.049+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3229: finishing copy_from to smithi16030472-33 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 33 v 707 (ObjNum 976 snap 0 seq_num 976) dirty exists 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: finishing write tid 2 to smithi16030472-142 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3231: got (0) Success 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 0 (ObjNum 1344 snap 0 seq_num 1344) dirty exists 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: finishing write tid 3 to smithi16030472-142 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: finishing write tid 4 to smithi16030472-142 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: finishing write tid 5 to smithi16030472-142 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: oid 142 updating version 0 to 940 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: oid 142 updating version 940 to 1007 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: oid 142 updating version 1007 to 1008 2024-09-21T18:09:52.084 INFO:tasks.rados.rados.0.smithi160.stdout:3227: oid 142 version 1008 is already newer than 1006 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 142 v 1008 (ObjNum 1395 snap 0 seq_num 1395) dirty exists 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3227: left oid 142 (ObjNum 1395 snap 0 seq_num 1395) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3224: done (11 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3225: done (10 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3227: done (9 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3228: done (8 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3229: done (7 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3230: done (6 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3231: done (5 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3232: done (4 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3233: done (3 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3234: done (2 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3235: done (1 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3236: done (0 left) 2024-09-21T18:09:52.085 INFO:tasks.rados.rados.0.smithi160.stdout:3237: cache_flush oid 360 snap -1 2024-09-21T18:09:52.086 INFO:tasks.rados.rados.0.smithi160.stdout:3238: write oid 86 current snap is 0 2024-09-21T18:09:52.086 INFO:tasks.rados.rados.0.smithi160.stdout:3238: seq_num 1397 ranges {679379=415452,1599227=756146,3097215=382164} 2024-09-21T18:09:52.086 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:52.086 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:52.086 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:52.086 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:52.086 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:52.100 INFO:tasks.rados.rados.0.smithi160.stdout:3238: writing smithi16030472-86 from 679379 to 1094831 tid 1 2024-09-21T18:09:52.102 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:52.121 INFO:tasks.rados.rados.0.smithi160.stdout:3238: writing smithi16030472-86 from 1599227 to 2355373 tid 2 2024-09-21T18:09:52.138 INFO:tasks.rados.rados.0.smithi160.stdout:3238: writing smithi16030472-86 from 3097215 to 3479379 tid 3 2024-09-21T18:09:52.138 INFO:tasks.rados.rados.0.smithi160.stdout:3239: cache_try_flush oid 436 snap -1 2024-09-21T18:09:52.138 INFO:tasks.rados.rados.0.smithi160.stdout:3240: copy_from oid 145 from oid 444 current snap is 0 2024-09-21T18:09:52.138 INFO:tasks.rados.rados.0.smithi160.stdout:3241: write (excl) oid 69 current snap is 0 2024-09-21T18:09:52.139 INFO:tasks.rados.rados.0.smithi160.stdout:3241: seq_num 1398 ranges {696186=490701,1657709=616157,2724638=562274} 2024-09-21T18:09:52.151 INFO:tasks.rados.rados.0.smithi160.stdout:3241: writing smithi16030472-69 from 696186 to 1186887 tid 1 2024-09-21T18:09:52.154 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:52.150+0000 7f9bb588f640 -1 osd.10 115 handle_osd_map injecting map crc failure 2024-09-21T18:09:52.156 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:52.153+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:52.161 INFO:tasks.rados.rados.0.smithi160.stdout:3241: writing smithi16030472-69 from 1657709 to 2273866 tid 2 2024-09-21T18:09:52.170 INFO:tasks.rados.rados.0.smithi160.stdout:3241: writing smithi16030472-69 from 2724638 to 3286912 tid 3 2024-09-21T18:09:52.170 INFO:tasks.rados.rados.0.smithi160.stdout:3242: read oid 348 snap -1 2024-09-21T18:09:52.170 INFO:tasks.rados.rados.0.smithi160.stdout:3242: expect (ObjNum 506 snap 0 seq_num 506) 2024-09-21T18:09:52.170 INFO:tasks.rados.rados.0.smithi160.stdout:3243: delete oid 113 current snap is 0 2024-09-21T18:09:52.170 INFO:tasks.rados.rados.0.smithi160.stdout:3241: finishing write tid 1 to smithi16030472-69 2024-09-21T18:09:52.177 INFO:tasks.rados.rados.0.smithi160.stdout:3243: done (6 left) 2024-09-21T18:09:52.178 INFO:tasks.rados.rados.0.smithi160.stdout:3244: copy_from oid 314 from oid 335 current snap is 0 2024-09-21T18:09:52.178 INFO:tasks.rados.rados.0.smithi160.stdout:3245: copy_from oid 239 from oid 289 current snap is 0 2024-09-21T18:09:52.178 INFO:tasks.rados.rados.0.smithi160.stdout:3246: copy_from oid 389 from oid 236 current snap is 0 2024-09-21T18:09:52.178 INFO:tasks.rados.rados.0.smithi160.stdout:3247: write (excl) oid 285 current snap is 0 2024-09-21T18:09:52.178 INFO:tasks.rados.rados.0.smithi160.stdout:3247: seq_num 1400 ranges {729800=757552,2174673=619826} 2024-09-21T18:09:52.192 INFO:tasks.rados.rados.0.smithi160.stdout:3247: writing smithi16030472-285 from 729800 to 1487352 tid 1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3247: writing smithi16030472-285 from 2174673 to 2794499 tid 2 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3248: cache_flush oid 147 snap -1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3249: cache_evict oid 17 snap -1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3250: copy_from oid 203 from oid 445 current snap is 0 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3251: read oid 128 snap -1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3251: expect deleted 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3252: read oid 144 snap -1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3252: expect (ObjNum 713 snap 0 seq_num 713) 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout:3253: cache_try_flush oid 322 snap -1 2024-09-21T18:09:52.205 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:52.221 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:52.216+0000 7fd79a4fd640 -1 osd.1 115 handle_osd_map injecting map crc failure 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:09:52.228 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.38666379599999, 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.229 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.230 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:09:52.231 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.232 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.975021502, 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.233 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:09:52.234 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.235 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.236 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:52.237 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.97219063, 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:09:52.238 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.239 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.240 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.241 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.96821772199999, 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.242 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.243 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.244 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.245 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.96510077100001, 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.246 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:09:52.247 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:09:52.248 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.249 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.96316976700001, 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.250 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:09:52.251 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:09:52.252 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.253 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.96006300799999, 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:52.254 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:09:52.255 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:09:52.256 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:09:52.257 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.95690532399999, 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.258 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.259 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.260 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.261 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.954672273, 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.95242060000001, 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.266 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:09:52.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:09:52.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:09:52.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.859843012, 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.270 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:52.274 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.083173636, 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:09:52.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:09:52.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.278 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.07366605199999, 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.279 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.966836073, 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:52.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:09:52.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 132.63346452100001, 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:09:52.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 127.86147305199999, 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:52.295 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 117.979324597, 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:09:52.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:09:52.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.299 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 93.439200149000001, 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:09:52.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 76.129241688999997, 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.304 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:09:52.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:09:52.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 72.884839377999995, 2024-09-21T18:09:52.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:52.312 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:52.314 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:52.314 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:52.309+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:52.340 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:52.333+0000 7fb260c0e640 -1 osd.11 115 handle_osd_map injecting map crc failure 2024-09-21T18:09:52.356 INFO:tasks.rados.rados.0.smithi160.stdout:3245: finishing copy_from to smithi16030472-239 2024-09-21T18:09:52.357 INFO:tasks.rados.rados.0.smithi160.stdout:3245: got expected ENOENT (src dne) 2024-09-21T18:09:52.371 INFO:tasks.rados.rados.0.smithi160.stdout:3242: expect (ObjNum 506 snap 0 seq_num 506) 2024-09-21T18:09:52.415 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:52.412+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:52.444 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:52.444 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:52.444 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:09:52.444 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:52.457 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: finishing write tid 2 to smithi16030472-69 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: finishing write tid 3 to smithi16030472-69 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: finishing write tid 4 to smithi16030472-69 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: finishing write tid 5 to smithi16030472-69 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: oid 69 updating version 0 to 965 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: oid 69 version 965 is already newer than 964 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: oid 69 version 965 is already newer than 963 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: oid 69 version 965 is already newer than 962 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 965 (ObjNum 1398 snap 0 seq_num 1398) dirty exists 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3241: left oid 69 (ObjNum 1398 snap 0 seq_num 1398) 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:3237: got (0) Success 2024-09-21T18:09:52.519 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 0 (ObjNum 915 snap 0 seq_num 915) dirty exists 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3238: finishing write tid 1 to smithi16030472-86 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3238: finishing write tid 2 to smithi16030472-86 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3245: finishing copy_from racing read to smithi16030472-239 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3237: done (15 left) 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3241: done (14 left) 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3242: done (13 left) 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3245: done (12 left) 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3251: done (11 left) 2024-09-21T18:09:52.520 INFO:tasks.rados.rados.0.smithi160.stdout:3254: delete oid 487 current snap is 0 2024-09-21T18:09:52.520 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:52.513+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:52.542 INFO:tasks.rados.rados.0.smithi160.stdout:3240: finishing copy_from to smithi16030472-145 2024-09-21T18:09:52.542 INFO:tasks.rados.rados.0.smithi160.stdout:3240: got expected ENOENT (src dne) 2024-09-21T18:09:52.542 INFO:tasks.rados.rados.0.smithi160.stdout:3240: finishing copy_from racing read to smithi16030472-145 2024-09-21T18:09:52.546 INFO:tasks.rados.rados.0.smithi160.stdout:3238: finishing write tid 3 to smithi16030472-86 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: finishing write tid 4 to smithi16030472-86 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: finishing write tid 5 to smithi16030472-86 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: oid 86 updating version 0 to 928 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: oid 86 version 928 is already newer than 927 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: oid 86 version 928 is already newer than 926 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: oid 86 updating version 928 to 929 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 86 v 929 (ObjNum 1397 snap 0 seq_num 1397) dirty exists 2024-09-21T18:09:52.547 INFO:tasks.rados.rados.0.smithi160.stdout:3238: left oid 86 (ObjNum 1397 snap 0 seq_num 1397) 2024-09-21T18:09:52.599 INFO:tasks.rados.rados.0.smithi160.stdout:3252: expect (ObjNum 713 snap 0 seq_num 713) 2024-09-21T18:09:52.599 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:52.613 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:52.614 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:52.614 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:52.616 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:52.613+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:52.626 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:09:52.685 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:52.717 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:52.712+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:52.728 INFO:tasks.rados.rados.0.smithi160.stdout:3247: finishing write tid 1 to smithi16030472-285 2024-09-21T18:09:52.728 INFO:tasks.rados.rados.0.smithi160.stdout:3247: finishing write tid 2 to smithi16030472-285 2024-09-21T18:09:52.728 INFO:tasks.rados.rados.0.smithi160.stdout:3247: finishing write tid 3 to smithi16030472-285 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3249: got (0) Success 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3246: finishing copy_from racing read to smithi16030472-389 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3250: finishing copy_from racing read to smithi16030472-203 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3246: finishing copy_from to smithi16030472-389 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 389 v 209 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3239: got (0) Success 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 436 v 0 (ObjNum 764 snap 0 seq_num 764) dirty exists 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3244: finishing copy_from to smithi16030472-314 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 314 v 225 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3244: finishing copy_from racing read to smithi16030472-314 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3247: finishing write tid 4 to smithi16030472-285 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3247: oid 285 updating version 0 to 966 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3247: oid 285 updating version 966 to 968 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:3247: oid 285 version 968 is already newer than 967 2024-09-21T18:09:52.729 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 285 v 968 (ObjNum 1400 snap 0 seq_num 1400) dirty exists 2024-09-21T18:09:52.730 INFO:tasks.rados.rados.0.smithi160.stdout:3247: left oid 285 (ObjNum 1400 snap 0 seq_num 1400) 2024-09-21T18:09:52.753 INFO:tasks.rados.rados.0.smithi160.stdout:3253: got (0) Success 2024-09-21T18:09:52.753 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 322 v 0 (ObjNum 129 snap 0 seq_num 3138341488) clean dne 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3238: done (11 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3239: done (10 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3240: done (9 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3244: done (8 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3246: done (7 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3247: done (6 left) 2024-09-21T18:09:52.771 INFO:tasks.rados.rados.0.smithi160.stdout:3249: done (5 left) 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3252: done (4 left) 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3253: done (3 left) 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3254: done (2 left) 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3255: copy_from oid 168 from oid 428 current snap is 0 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3256: read oid 338 snap -1 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3256: expect deleted 2024-09-21T18:09:52.772 INFO:tasks.rados.rados.0.smithi160.stdout:3257: delete oid 219 current snap is 0 2024-09-21T18:09:52.795 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:52.795 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:52.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:52.795 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:09:52.795 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:52.809 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:09:52.818 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:52.813+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:52.864 INFO:tasks.rados.rados.0.smithi160.stdout:3250: finishing copy_from to smithi16030472-203 2024-09-21T18:09:52.864 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 203 v 697 (ObjNum 950 snap 0 seq_num 950) dirty exists 2024-09-21T18:09:52.918 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:52.913+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:52.967 INFO:tasks.rados.rados.0.smithi160.stdout:3248: got (0) Success 2024-09-21T18:09:52.967 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 0 (ObjNum 1259 snap 0 seq_num 1259) dirty exists 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8653 4.1 4:9a656cd6:::smithi16030472-168:head [copy-from ver 755 in=56b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:52.864059+0000", 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.112927454, 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11294673700000001, 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:52.978 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8653 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864059+0000", 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864059+0000", 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.979 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864062+0000", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8559999999999998e-06 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864073+0000", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0838e-05 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864088+0000", 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5315000000000002e-05 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.980 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864098+0000", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0151e-05 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.880023+0000", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015924969000000001 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.880105+0000", 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1763999999999998e-05 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.981 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8654 4.1 4:9a656cd6:::smithi16030472-168:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:52.864119+0000", 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.112868074, 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11293069999999999, 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:52.982 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8654 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864119+0000", 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864119+0000", 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:52.983 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864120+0000", 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2589999999999999e-06 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864125+0000", 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8690000000000003e-06 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864136+0000", 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1707e-05 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.984 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.864147+0000", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0222e-05 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.880215+0000", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016068003000000001 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.880259+0000", 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4597e-05 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:52.985 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8658 4.3 4:f8256ac6:::smithi16030472-219:head [stat,delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:52.964427+0000", 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012559565, 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012657396, 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8658 2024-09-21T18:09:52.986 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964427+0000", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964427+0000", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964428+0000", 2024-09-21T18:09:52.987 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.4099999999999998e-07 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964431+0000", 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3119999999999998e-06 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964438+0000", 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7789999999999998e-06 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.988 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964444+0000", 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9750000000000004e-06 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964554+0000", 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011021100000000001 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.989 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:2710 4.1 4.31505eb5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:52.964570+0000", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012416361000000001, 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012543253000000001, 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964570+0000", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:52.990 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964570+0000", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964574+0000", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.777e-06 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964592+0000", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7499999999999998e-05 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:52.991 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964613+0000", 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1486000000000001e-05 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(osd.11.0:1916 4.3 e116/33)", 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:52.964669+0000", 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.012317615000000001, 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012464747999999999, 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:52.992 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964669+0000", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964669+0000", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964670+0000", 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3030000000000001e-06 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.993 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964675+0000", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6650000000000002e-06 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964685+0000", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0023e-05 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:52.964692+0000", 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3250000000000002e-06 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.994 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:52.995 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 5 2024-09-21T18:09:52.995 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:52.996 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:09:53.037 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:53.032+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:53.076 INFO:tasks.rados.rados.0.smithi160.stdout:3255: finishing copy_from racing read to smithi16030472-168 2024-09-21T18:09:53.079 INFO:tasks.rados.rados.0.smithi160.stdout:3255: finishing copy_from to smithi16030472-168 2024-09-21T18:09:53.079 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 168 v 755 (ObjNum 1113 snap 0 seq_num 1113) dirty exists 2024-09-21T18:09:53.119 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:53.116+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:53.158 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:53.158 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:53.158 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:53.158 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:53.158 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:53.166 INFO:tasks.rados.rados.0.smithi160.stdout:3248: done (4 left) 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3250: done (3 left) 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3255: done (2 left) 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3256: done (1 left) 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3257: done (0 left) 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3258: cache_flush oid 104 snap -1 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3259: cache_try_flush oid 132 snap -1 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3260: write (excl) oid 301 current snap is 0 2024-09-21T18:09:53.167 INFO:tasks.rados.rados.0.smithi160.stdout:3260: seq_num 1403 ranges {780221=583299,1950119=767153,3261753=318468} 2024-09-21T18:09:53.172 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:09:53.189 INFO:tasks.rados.rados.0.smithi160.stdout:3260: writing smithi16030472-301 from 780221 to 1363520 tid 1 2024-09-21T18:09:53.210 INFO:tasks.rados.rados.0.smithi160.stdout:3260: writing smithi16030472-301 from 1950119 to 2717272 tid 2 2024-09-21T18:09:53.220 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:53.216+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:53.222 INFO:tasks.rados.rados.0.smithi160.stdout:3260: writing smithi16030472-301 from 3261753 to 3580221 tid 3 2024-09-21T18:09:53.222 INFO:tasks.rados.rados.0.smithi160.stdout:3261: copy_from oid 428 from oid 282 current snap is 0 2024-09-21T18:09:53.222 INFO:tasks.rados.rados.0.smithi160.stdout:3262: write (excl) oid 60 current snap is 0 2024-09-21T18:09:53.222 INFO:tasks.rados.rados.0.smithi160.stdout:3262: seq_num 1404 ranges {797028=658548,1892248=510811,3172823=424205} 2024-09-21T18:09:53.238 INFO:tasks.rados.rados.0.smithi160.stdout:3262: writing smithi16030472-60 from 797028 to 1455576 tid 1 2024-09-21T18:09:53.246 INFO:tasks.rados.rados.0.smithi160.stdout:3262: writing smithi16030472-60 from 1892248 to 2403059 tid 2 2024-09-21T18:09:53.256 INFO:tasks.rados.rados.0.smithi160.stdout:3262: writing smithi16030472-60 from 3172823 to 3597028 tid 3 2024-09-21T18:09:53.256 INFO:tasks.rados.rados.0.smithi160.stdout:3263: read oid 495 snap -1 2024-09-21T18:09:53.256 INFO:tasks.rados.rados.0.smithi160.stdout:3263: expect (ObjNum 1344 snap 0 seq_num 1344) 2024-09-21T18:09:53.257 INFO:tasks.rados.rados.0.smithi160.stdout:3264: cache_evict oid 344 snap -1 2024-09-21T18:09:53.257 INFO:tasks.rados.rados.0.smithi160.stdout:3265: copy_from oid 297 from oid 361 current snap is 0 2024-09-21T18:09:53.257 INFO:tasks.rados.rados.0.smithi160.stdout:3266: write oid 427 current snap is 0 2024-09-21T18:09:53.257 INFO:tasks.rados.rados.0.smithi160.stdout:3266: seq_num 1405 ranges {413835=733797,1550730=770822,2800246=557531} 2024-09-21T18:09:53.265 INFO:tasks.rados.rados.0.smithi160.stdout:3266: writing smithi16030472-427 from 413835 to 1147632 tid 1 2024-09-21T18:09:53.273 INFO:tasks.rados.rados.0.smithi160.stdout:3266: writing smithi16030472-427 from 1550730 to 2321552 tid 2 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3266: writing smithi16030472-427 from 2800246 to 3357777 tid 3 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3267: read oid 14 snap -1 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3267: expect (ObjNum 1285 snap 0 seq_num 1285) 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3268: read oid 135 snap -1 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3268: expect deleted 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3269: read oid 476 snap -1 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3269: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3270: copy_from oid 120 from oid 56 current snap is 0 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3271: copy_from oid 84 from oid 52 current snap is 0 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3272: read oid 448 snap -1 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3272: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:53.282 INFO:tasks.rados.rados.0.smithi160.stdout:3273: write oid 36 current snap is 0 2024-09-21T18:09:53.283 INFO:tasks.rados.rados.0.smithi160.stdout:3273: seq_num 1406 ranges {430642=409046,1609212=514480,2827669=768803} 2024-09-21T18:09:53.287 INFO:tasks.rados.rados.0.smithi160.stdout:3273: writing smithi16030472-36 from 430642 to 839688 tid 1 2024-09-21T18:09:53.296 INFO:tasks.rados.rados.0.smithi160.stdout:3273: writing smithi16030472-36 from 1609212 to 2123692 tid 2 2024-09-21T18:09:53.306 INFO:tasks.rados.rados.0.smithi160.stdout:3273: writing smithi16030472-36 from 2827669 to 3596472 tid 3 2024-09-21T18:09:53.307 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.16212279300001, 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:09:53.311 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:09:53.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:09:53.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:09:53.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:09:53.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.16207648400001, 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:09:53.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:09:53.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:09:53.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.15998057100001, 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.325 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:09:53.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:09:53.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:53.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:53.331 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.159317456, 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:09:53.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:09:53.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.15928095699999, 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.338 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.025642785, 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:09:53.344 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:09:53.345 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:09:53.346 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:09:53.347 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:09:53.348 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:09:53.349 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.02493099399999, 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.350 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.351 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:09:53.352 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:09:53.353 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:09:53.354 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.355 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 160.02489343100001, 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:09:53.356 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.357 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.358 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:53.359 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.360 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.361 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.22750743200001, 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.362 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.363 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.364 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.365 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.22731288599999, 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.366 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.367 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.368 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.369 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.22702649799999, 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:09:53.370 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:09:53.371 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:09:53.372 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:09:53.373 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.22701041799999, 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:09:53.374 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:09:53.375 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:09:53.376 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.377 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.198083166, 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.378 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.379 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.380 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.381 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.19186409100001, 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.382 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.383 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:09:53.384 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:09:53.385 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.386 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.186762496, 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.387 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:09:53.388 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:09:53.389 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.390 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.18642854800001, 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.391 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:09:53.392 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:09:53.393 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.394 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.18640806900001, 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.395 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.396 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.397 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.398 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.100079542, 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:53.399 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:09:53.400 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:09:53.401 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.402 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.091619356, 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.403 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.404 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.405 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.406 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.08569165099999, 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.407 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.408 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:53.409 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.410 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.411 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.411 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:53.414 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:09:53.415 INFO:tasks.rados.rados.0.smithi160.stdout:3269: expect (ObjNum 180 snap 0 seq_num 180) 2024-09-21T18:09:53.415 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1704065 2024-09-21T18:09:53.415 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:53.406+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:53.461 INFO:tasks.rados.rados.0.smithi160.stdout:3271: finishing copy_from to smithi16030472-84 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3271: got expected ENOENT (src dne) 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3271: finishing copy_from racing read to smithi16030472-84 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3269: done (15 left) 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3271: done (14 left) 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3274: copy_from oid 6 from oid 11 current snap is 0 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout:3275: cache_flush oid 151 snap -1 2024-09-21T18:09:53.462 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.467 INFO:tasks.rados.rados.0.smithi160.stdout:3262: finishing write tid 1 to smithi16030472-60 2024-09-21T18:09:53.467 INFO:tasks.rados.rados.0.smithi160.stdout:3262: finishing write tid 2 to smithi16030472-60 2024-09-21T18:09:53.467 INFO:tasks.rados.rados.0.smithi160.stdout:3262: finishing write tid 3 to smithi16030472-60 2024-09-21T18:09:53.485 INFO:tasks.rados.rados.0.smithi160.stdout:3274: finishing copy_from racing read to smithi16030472-6 2024-09-21T18:09:53.513 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:53.508+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:53.542 INFO:tasks.rados.rados.0.smithi160.stdout:3262: finishing write tid 4 to smithi16030472-60 2024-09-21T18:09:53.542 INFO:tasks.rados.rados.0.smithi160.stdout:3262: finishing write tid 5 to smithi16030472-60 2024-09-21T18:09:53.543 INFO:tasks.rados.rados.0.smithi160.stdout:3262: oid 60 updating version 0 to 1010 2024-09-21T18:09:53.543 INFO:tasks.rados.rados.0.smithi160.stdout:3262: oid 60 updating version 1010 to 1013 2024-09-21T18:09:53.543 INFO:tasks.rados.rados.0.smithi160.stdout:3262: oid 60 version 1013 is already newer than 1012 2024-09-21T18:09:53.543 INFO:tasks.rados.rados.0.smithi160.stdout:3262: oid 60 version 1013 is already newer than 1011 2024-09-21T18:09:53.543 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 1013 (ObjNum 1404 snap 0 seq_num 1404) dirty exists 2024-09-21T18:09:53.544 INFO:tasks.rados.rados.0.smithi160.stdout:3262: left oid 60 (ObjNum 1404 snap 0 seq_num 1404) 2024-09-21T18:09:53.544 INFO:tasks.rados.rados.0.smithi160.stdout:3262: done (15 left) 2024-09-21T18:09:53.544 INFO:tasks.rados.rados.0.smithi160.stdout:3276: write (excl) oid 364 current snap is 0 2024-09-21T18:09:53.544 INFO:tasks.rados.rados.0.smithi160.stdout:3276: seq_num 1407 ranges {447449=600648,1784047=774491,2971445=676004} 2024-09-21T18:09:53.552 INFO:tasks.rados.rados.0.smithi160.stdout:3276: writing smithi16030472-364 from 447449 to 1048097 tid 1 2024-09-21T18:09:53.565 INFO:tasks.rados.rados.0.smithi160.stdout:3276: writing smithi16030472-364 from 1784047 to 2558538 tid 2 2024-09-21T18:09:53.574 INFO:tasks.rados.rados.0.smithi160.stdout:3276: writing smithi16030472-364 from 2971445 to 3647449 tid 3 2024-09-21T18:09:53.575 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:3264: got (0) Success 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:3264: done (15 left) 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:3277: cache_flush oid 230 snap -1 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:3274: finishing copy_from to smithi16030472-6 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 333 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:09:53.590 INFO:tasks.rados.rados.0.smithi160.stdout:3274: done (15 left) 2024-09-21T18:09:53.591 INFO:tasks.rados.rados.0.smithi160.stdout:3278: read oid 251 snap -1 2024-09-21T18:09:53.591 INFO:tasks.rados.rados.0.smithi160.stdout:3278: expect (ObjNum 1129 snap 0 seq_num 1129) 2024-09-21T18:09:53.591 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.592 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8708 4.2 4.d25eab1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:53.552096+0000", 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.039111492999999997, 2024-09-21T18:09:53.593 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.039161315000000002, 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.594 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8708 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552096+0000", 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.595 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552096+0000", 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.596 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552100+0000", 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552350+0000", 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000249428 2024-09-21T18:09:53.597 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552384+0000", 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4505999999999998e-05 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.598 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.552402+0000", 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7723999999999998e-05 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.599 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8709 4.2 4.d25eab1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:53.564893+0000", 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.026314271, 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.026498226999999999, 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.600 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8709 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.601 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.564893+0000", 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.564893+0000", 2024-09-21T18:09:53.602 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.564898+0000", 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0829999999999997e-06 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.603 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.565654+0000", 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00075624399999999999 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.604 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.565692+0000", 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7953000000000001e-05 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.565710+0000", 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.84e-05 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.605 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.606 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.606 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.606 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.606 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8710 4.2 4.d25eab1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:53.606 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:53.574116+0000", 2024-09-21T18:09:53.607 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.017091248, 2024-09-21T18:09:53.607 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.017393850999999998, 2024-09-21T18:09:53.607 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.607 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.607 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:53.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.608 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.608 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8710 2024-09-21T18:09:53.608 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.609 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.609 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.609 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.609 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574116+0000", 2024-09-21T18:09:53.609 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.610 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.610 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.610 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.610 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574116+0000", 2024-09-21T18:09:53.610 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.611 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.611 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.611 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.611 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574120+0000", 2024-09-21T18:09:53.612 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4029999999999999e-06 2024-09-21T18:09:53.612 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.612 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.612 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.612 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574780+0000", 2024-09-21T18:09:53.613 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00065997799999999998 2024-09-21T18:09:53.613 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.613 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.613 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.613 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574809+0000", 2024-09-21T18:09:53.614 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8943000000000001e-05 2024-09-21T18:09:53.614 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.614 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.614 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.614 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574825+0000", 2024-09-21T18:09:53.615 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5724000000000001e-05 2024-09-21T18:09:53.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.615 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.615 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.618 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.618 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.618 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8711 4.2 4.d25eab1e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:53.619 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:53.574854+0000", 2024-09-21T18:09:53.619 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.016353607999999999, 2024-09-21T18:09:53.619 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016772307, 2024-09-21T18:09:53.619 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.619 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.620 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:53.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.620 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.620 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8711 2024-09-21T18:09:53.621 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.621 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.621 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.621 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.621 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574854+0000", 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574854+0000", 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.622 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574856+0000", 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4699999999999999e-06 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.623 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574867+0000", 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0967e-05 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574879+0000", 2024-09-21T18:09:53.624 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2480000000000001e-05 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574888+0000", 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8100000000000004e-06 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.625 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.626 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.626 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.626 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.626 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:8712 4.2 4.d25eab1e (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-21T18:09:53.626 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:53.574903+0000", 2024-09-21T18:09:53.627 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.016305180999999998, 2024-09-21T18:09:53.627 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.016844181999999999, 2024-09-21T18:09:53.627 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.627 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.627 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:09:53.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.628 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.628 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 8712 2024-09-21T18:09:53.628 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.629 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.629 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.629 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.629 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574903+0000", 2024-09-21T18:09:53.629 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.630 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.630 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.630 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:53.630 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574903+0000", 2024-09-21T18:09:53.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.631 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.631 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.631 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:53.631 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574903+0000", 2024-09-21T18:09:53.631 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.33e-07 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574907+0000", 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8829999999999999e-06 2024-09-21T18:09:53.632 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.633 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.633 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:53.633 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574919+0000", 2024-09-21T18:09:53.633 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1406999999999999e-05 2024-09-21T18:09:53.633 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:53.574927+0000", 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8529999999999992e-06 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:53.634 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.635 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:53.635 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:09:53.635 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 5 2024-09-21T18:09:53.635 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:53.637 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:09:53.638 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:53.632+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:53.691 INFO:tasks.rados.rados.0.smithi160.stdout:3277: got (0) Success 2024-09-21T18:09:53.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 0 (ObjNum 796 snap 0 seq_num 796) dirty exists 2024-09-21T18:09:53.691 INFO:tasks.rados.rados.0.smithi160.stdout:3277: done (15 left) 2024-09-21T18:09:53.692 INFO:tasks.rados.rados.0.smithi160.stdout:3279: read oid 238 snap -1 2024-09-21T18:09:53.692 INFO:tasks.rados.rados.0.smithi160.stdout:3279: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:53.692 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.738 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:53.736+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:53.773 INFO:tasks.rados.rados.0.smithi160.stdout:3258: got (2) No such file or directory 2024-09-21T18:09:53.773 INFO:tasks.rados.rados.0.smithi160.stdout:3260: finishing write tid 1 to smithi16030472-301 2024-09-21T18:09:53.773 INFO:tasks.rados.rados.0.smithi160.stdout:3258: done (15 left) 2024-09-21T18:09:53.773 INFO:tasks.rados.rados.0.smithi160.stdout:3280: copy_from oid 337 from oid 302 current snap is 0 2024-09-21T18:09:53.773 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.776 INFO:tasks.rados.rados.0.smithi160.stdout:3263: expect (ObjNum 1344 snap 0 seq_num 1344) 2024-09-21T18:09:53.801 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2027496 2024-09-21T18:09:53.813 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:53.813 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:09:53.813 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:09:53.813 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:09:53.813 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:53.828 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:09:53.839 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:53.834+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:53.868 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:53.910 INFO:tasks.rados.rados.0.smithi160.stdout:3261: finishing copy_from racing read to smithi16030472-428 2024-09-21T18:09:53.910 INFO:tasks.rados.rados.0.smithi160.stdout:3259: got (0) Success 2024-09-21T18:09:53.910 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 132 v 0 (ObjNum 1243 snap 0 seq_num 1243) dirty exists 2024-09-21T18:09:53.910 INFO:tasks.rados.rados.0.smithi160.stdout:3260: finishing write tid 2 to smithi16030472-301 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3259: done (15 left) 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3263: done (14 left) 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3281: read oid 216 snap -1 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3281: expect (ObjNum 605 snap 0 seq_num 605) 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3282: write oid 449 current snap is 0 2024-09-21T18:09:53.911 INFO:tasks.rados.rados.0.smithi160.stdout:3282: seq_num 1408 ranges {464256=675897,1726176=518149,2882515=507700} 2024-09-21T18:09:53.917 INFO:tasks.rados.rados.0.smithi160.stdout:3282: writing smithi16030472-449 from 464256 to 1140153 tid 1 2024-09-21T18:09:53.925 INFO:tasks.rados.rados.0.smithi160.stdout:3282: writing smithi16030472-449 from 1726176 to 2244325 tid 2 2024-09-21T18:09:53.932 INFO:tasks.rados.rados.0.smithi160.stdout:3282: writing smithi16030472-449 from 2882515 to 3390215 tid 3 2024-09-21T18:09:53.933 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:53.933 INFO:tasks.rados.rados.0.smithi160.stdout:3276: finishing write tid 1 to smithi16030472-364 2024-09-21T18:09:53.933 INFO:tasks.rados.rados.0.smithi160.stdout:3267: expect (ObjNum 1285 snap 0 seq_num 1285) 2024-09-21T18:09:53.939 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:53.934+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:53.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:09:53.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:53.980 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 511237 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.32044439500001, 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:09:53.998 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:53.999 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.000 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.001 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.002 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.32037403300001, 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.003 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.004 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.005 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:09:54.006 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.30857259800001, 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.007 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.008 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:09:54.009 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:09:54.010 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.30070621499999, 2024-09-21T18:09:54.011 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.012 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.013 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.014 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.015 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.30032057099999, 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:54.016 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:09:54.017 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:09:54.018 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.019 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 155.30027512199999, 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.020 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:54.021 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:09:54.022 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:09:54.023 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:09:54.024 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 154.25237720499999, 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:09:54.025 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.026 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:09:54.027 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:09:54.028 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.029 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 154.252269563, 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.030 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:09:54.031 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:09:54.032 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.033 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 148.67648694799999, 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.034 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.035 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.036 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:09:54.037 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:09:54.038 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.039 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 148.676435013, 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.040 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.041 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:09:54.042 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:09:54.043 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.044 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 147.90973062500001, 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.045 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.046 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:09:54.047 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:09:54.048 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:09:54.049 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:09:54.050 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.051 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 147.680627367, 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.052 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.053 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.054 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.055 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.056 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.057 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 147.68019937700001, 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:09:54.058 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.059 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:09:54.060 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:09:54.061 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:54.062 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:09:54.063 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 147.680152808, 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.064 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.065 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.066 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.067 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.068 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:09:54.069 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 143.179483748, 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:54.070 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.071 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.072 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.073 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:09:54.074 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.171951952, 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.075 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.076 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.077 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.078 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.079 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:09:54.080 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 136.17187763199999, 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.081 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.082 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.083 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.084 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.085 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.086 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 135.77007438699999, 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:09:54.087 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:09:54.088 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:09:54.089 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:09:54.090 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 135.667724499, 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.091 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.092 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.093 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.094 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.095 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6670 4.2 4:574ca510:::smithi16030472-466:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:54.096 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 43.355096101000001, 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4286385400000001, 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6670 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.097 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637674+0000", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637680+0000", 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6629999999999998e-06 2024-09-21T18:09:54.098 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637691+0000", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1097000000000001e-05 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637700+0000", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054141+0000", 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41644096200000003 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.099 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054299+0000", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157572 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.065938+0000", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.011639553 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066192+0000", 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253222 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:09:54.100 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066312+0000", 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120285 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:09:54.101 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:09:54.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:09:54.103 INFO:tasks.rados.rados.0.smithi160.stdout:3260: finishing write tid 3 to smithi16030472-301 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: finishing write tid 4 to smithi16030472-301 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: finishing write tid 5 to smithi16030472-301 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: oid 301 updating version 0 to 977 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: oid 301 version 977 is already newer than 975 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: oid 301 updating version 977 to 978 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: oid 301 version 978 is already newer than 976 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 301 v 978 (ObjNum 1403 snap 0 seq_num 1403) dirty exists 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3260: left oid 301 (ObjNum 1403 snap 0 seq_num 1403) 2024-09-21T18:09:54.104 INFO:tasks.rados.rados.0.smithi160.stdout:3279: expect (ObjNum 684 snap 0 seq_num 684) 2024-09-21T18:09:54.105 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:54.098+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:54.138 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1561417 2024-09-21T18:09:54.203 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:54.200+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:54.226 INFO:tasks.rados.rados.0.smithi160.stdout:3276: finishing write tid 2 to smithi16030472-364 2024-09-21T18:09:54.226 INFO:tasks.rados.rados.0.smithi160.stdout:3272: expect (ObjNum 777 snap 0 seq_num 777) 2024-09-21T18:09:54.240 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:09:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:09:54.240 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:09:54.240 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:09:54.257 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:54.258 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:54.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:09:54.278 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:54.299 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:54.299 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:54.304 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:54.298+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:54.319 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:54.320 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3276: finishing write tid 3 to smithi16030472-364 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3275: got (0) Success 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 151 v 0 (ObjNum 753 snap 0 seq_num 753) dirty exists 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3266: finishing write tid 1 to smithi16030472-427 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3261: finishing copy_from to smithi16030472-428 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 428 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3266: finishing write tid 2 to smithi16030472-427 2024-09-21T18:09:54.336 INFO:tasks.rados.rados.0.smithi160.stdout:3266: finishing write tid 3 to smithi16030472-427 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3265: finishing copy_from to smithi16030472-297 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3265: got expected ENOENT (src dne) 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3265: finishing copy_from racing read to smithi16030472-297 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: finishing write tid 4 to smithi16030472-364 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: finishing write tid 5 to smithi16030472-364 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: oid 364 updating version 0 to 1014 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: oid 364 updating version 1014 to 1016 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: oid 364 updating version 1016 to 1017 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: oid 364 version 1017 is already newer than 1015 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 1017 (ObjNum 1407 snap 0 seq_num 1407) dirty exists 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3276: left oid 364 (ObjNum 1407 snap 0 seq_num 1407) 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3266: finishing write tid 4 to smithi16030472-427 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3266: finishing write tid 5 to smithi16030472-427 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3266: oid 427 updating version 0 to 980 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3266: oid 427 version 980 is already newer than 979 2024-09-21T18:09:54.337 INFO:tasks.rados.rados.0.smithi160.stdout:3266: oid 427 updating version 980 to 981 2024-09-21T18:09:54.338 INFO:tasks.rados.rados.0.smithi160.stdout:3266: oid 427 updating version 981 to 982 2024-09-21T18:09:54.338 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 982 (ObjNum 1405 snap 0 seq_num 1405) dirty exists 2024-09-21T18:09:54.338 INFO:tasks.rados.rados.0.smithi160.stdout:3266: left oid 427 (ObjNum 1405 snap 0 seq_num 1405) 2024-09-21T18:09:54.338 INFO:tasks.rados.rados.0.smithi160.stdout:3278: expect (ObjNum 1129 snap 0 seq_num 1129) 2024-09-21T18:09:54.338 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:54.338 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:54.375 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:54.375 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:54.382 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:54.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:09:54.428 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:54.428 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:54.429 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:54.425+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:54.449 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:54.450 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:54.450 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:54.450 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2428369 2024-09-21T18:09:54.459 INFO:tasks.rados.rados.0.smithi160.stdout:3280: finishing copy_from racing read to smithi16030472-337 2024-09-21T18:09:54.459 INFO:tasks.rados.rados.0.smithi160.stdout:3280: finishing copy_from to smithi16030472-337 2024-09-21T18:09:54.459 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 337 v 440 (ObjNum 592 snap 0 seq_num 592) dirty exists 2024-09-21T18:09:54.459 INFO:tasks.rados.rados.0.smithi160.stdout:3281: expect (ObjNum 605 snap 0 seq_num 605) 2024-09-21T18:09:54.467 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:54.488 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:54.489 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:54.505 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:54.501+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:54.513 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:54.513 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:54.514 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 593165 2024-09-21T18:09:54.535 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:54.536 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3270: finishing copy_from racing read to smithi16030472-120 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: finishing write tid 1 to smithi16030472-36 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: finishing write tid 2 to smithi16030472-36 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: finishing write tid 3 to smithi16030472-36 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: finishing write tid 4 to smithi16030472-36 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: finishing write tid 5 to smithi16030472-36 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: oid 36 updating version 0 to 930 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: oid 36 updating version 930 to 932 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: oid 36 updating version 932 to 933 2024-09-21T18:09:54.537 INFO:tasks.rados.rados.0.smithi160.stdout:3273: oid 36 version 933 is already newer than 931 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 36 v 933 (ObjNum 1406 snap 0 seq_num 1406) dirty exists 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3273: left oid 36 (ObjNum 1406 snap 0 seq_num 1406) 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3270: finishing copy_from to smithi16030472-120 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 120 v 669 (ObjNum 939 snap 0 seq_num 939) dirty exists 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3282: finishing write tid 1 to smithi16030472-449 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3282: finishing write tid 2 to smithi16030472-449 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3260: done (15 left) 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3261: done (14 left) 2024-09-21T18:09:54.538 INFO:tasks.rados.rados.0.smithi160.stdout:3265: done (13 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3266: done (12 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3267: done (11 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3268: done (10 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3270: done (9 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3272: done (8 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3273: done (7 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3275: done (6 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3276: done (5 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3278: done (4 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3279: done (3 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3280: done (2 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3281: done (1 left) 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3283: write oid 323 current snap is 0 2024-09-21T18:09:54.539 INFO:tasks.rados.rados.0.smithi160.stdout:3283: seq_num 1409 ranges {481063=751146,1784658=778160,3309938=371125} 2024-09-21T18:09:54.547 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:54.547 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:54.556 INFO:tasks.rados.rados.0.smithi160.stdout:3283: writing smithi16030472-323 from 481063 to 1232209 tid 1 2024-09-21T18:09:54.557 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:54.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:09:54.577 INFO:tasks.rados.rados.0.smithi160.stdout:3283: writing smithi16030472-323 from 1784658 to 2562818 tid 2 2024-09-21T18:09:54.583 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:54.583 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3283: writing smithi16030472-323 from 3309938 to 3681063 tid 3 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3284: cache_evict oid 177 snap -1 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3285: read oid 493 snap -1 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3285: expect (ObjNum 1220 snap 0 seq_num 1220) 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3286: copy_from oid 68 from oid 34 current snap is 0 2024-09-21T18:09:54.594 INFO:tasks.rados.rados.0.smithi160.stdout:3283: finishing write tid 1 to smithi16030472-323 2024-09-21T18:09:54.595 INFO:tasks.rados.rados.0.smithi160.stdout:3287: delete oid 58 current snap is 0 2024-09-21T18:09:54.606 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:54.600+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:54.608 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:54.612 INFO:tasks.rados.rados.0.smithi160.stdout:3282: finishing write tid 3 to smithi16030472-449 2024-09-21T18:09:54.612 INFO:tasks.rados.rados.0.smithi160.stdout:3282: finishing write tid 4 to smithi16030472-449 2024-09-21T18:09:54.612 INFO:tasks.rados.rados.0.smithi160.stdout:3282: finishing write tid 5 to smithi16030472-449 2024-09-21T18:09:54.612 INFO:tasks.rados.rados.0.smithi160.stdout:3282: oid 449 updating version 0 to 935 2024-09-21T18:09:54.613 INFO:tasks.rados.rados.0.smithi160.stdout:3282: oid 449 version 935 is already newer than 934 2024-09-21T18:09:54.613 INFO:tasks.rados.rados.0.smithi160.stdout:3282: oid 449 updating version 935 to 936 2024-09-21T18:09:54.613 INFO:tasks.rados.rados.0.smithi160.stdout:3282: oid 449 updating version 936 to 937 2024-09-21T18:09:54.613 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 937 (ObjNum 1408 snap 0 seq_num 1408) dirty exists 2024-09-21T18:09:54.613 INFO:tasks.rados.rados.0.smithi160.stdout:3282: left oid 449 (ObjNum 1408 snap 0 seq_num 1408) 2024-09-21T18:09:54.695 INFO:tasks.rados.rados.0.smithi160.stdout:3284: got (16) Device or resource busy 2024-09-21T18:09:54.698 INFO:tasks.rados.rados.0.smithi160.stdout:3285: expect (ObjNum 1220 snap 0 seq_num 1220) 2024-09-21T18:09:54.698 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:54.702 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:54.702 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:54.707 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:54.704+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:54.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:09:54.719 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:54.719 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:54.723 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:54.758 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2338511 2024-09-21T18:09:54.779 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1700290 2024-09-21T18:09:54.794 INFO:tasks.rados.rados.0.smithi160.stdout:3283: finishing write tid 2 to smithi16030472-323 2024-09-21T18:09:54.798 INFO:tasks.rados.rados.0.smithi160.stdout:3282: done (5 left) 2024-09-21T18:09:54.798 INFO:tasks.rados.rados.0.smithi160.stdout:3284: done (4 left) 2024-09-21T18:09:54.799 INFO:tasks.rados.rados.0.smithi160.stdout:3285: done (3 left) 2024-09-21T18:09:54.799 INFO:tasks.rados.rados.0.smithi160.stdout:3287: done (2 left) 2024-09-21T18:09:54.799 INFO:tasks.rados.rados.0.smithi160.stdout:3288: write oid 456 current snap is 0 2024-09-21T18:09:54.799 INFO:tasks.rados.rados.0.smithi160.stdout:3288: seq_num 1411 ranges {514677=501644,1501622=781829,2964784=741516} 2024-09-21T18:09:54.804 INFO:tasks.rados.rados.0.smithi160.stdout:3288: writing smithi16030472-456 from 514677 to 1016321 tid 1 2024-09-21T18:09:54.808 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:54.804+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:54.811 INFO:tasks.rados.rados.0.smithi160.stdout:3288: writing smithi16030472-456 from 1501622 to 2283451 tid 2 2024-09-21T18:09:54.820 INFO:tasks.rados.rados.0.smithi160.stdout:3288: writing smithi16030472-456 from 2964784 to 3706300 tid 3 2024-09-21T18:09:54.820 INFO:tasks.rados.rados.0.smithi160.stdout:3289: copy_from oid 306 from oid 353 current snap is 0 2024-09-21T18:09:54.820 INFO:tasks.rados.rados.0.smithi160.stdout:3283: finishing write tid 3 to smithi16030472-323 2024-09-21T18:09:54.820 INFO:tasks.rados.rados.0.smithi160.stdout:3283: finishing write tid 4 to smithi16030472-323 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: finishing write tid 5 to smithi16030472-323 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: oid 323 updating version 0 to 974 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: oid 323 updating version 974 to 975 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: oid 323 version 975 is already newer than 973 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: oid 323 updating version 975 to 976 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 323 v 976 (ObjNum 1409 snap 0 seq_num 1409) dirty exists 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: left oid 323 (ObjNum 1409 snap 0 seq_num 1409) 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3288: finishing write tid 1 to smithi16030472-456 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3286: finishing copy_from racing read to smithi16030472-68 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3283: done (3 left) 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3290: copy_from oid 76 from oid 24 current snap is 0 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3288: finishing write tid 2 to smithi16030472-456 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3291: cache_evict oid 397 snap -1 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3292: cache_evict oid 119 snap -1 2024-09-21T18:09:54.821 INFO:tasks.rados.rados.0.smithi160.stdout:3293: read oid 200 snap -1 2024-09-21T18:09:54.822 INFO:tasks.rados.rados.0.smithi160.stdout:3293: expect (ObjNum 1277 snap 0 seq_num 1277) 2024-09-21T18:09:54.822 INFO:tasks.rados.rados.0.smithi160.stdout:3294: write oid 468 current snap is 0 2024-09-21T18:09:54.822 INFO:tasks.rados.rados.0.smithi160.stdout:3294: seq_num 1412 ranges {531484=576893,1843751=525487,2875854=669141} 2024-09-21T18:09:54.827 INFO:tasks.rados.rados.0.smithi160.stdout:3294: writing smithi16030472-468 from 531484 to 1108377 tid 1 2024-09-21T18:09:54.834 INFO:tasks.rados.rados.0.smithi160.stdout:3294: writing smithi16030472-468 from 1843751 to 2369238 tid 2 2024-09-21T18:09:54.842 INFO:tasks.rados.rados.0.smithi160.stdout:3294: writing smithi16030472-468 from 2875854 to 3544995 tid 3 2024-09-21T18:09:54.842 INFO:tasks.rados.rados.0.smithi160.stdout:3295: write oid 149 current snap is 0 2024-09-21T18:09:54.842 INFO:tasks.rados.rados.0.smithi160.stdout:3295: seq_num 1413 ranges {548291=652142,1902233=785498,3303277=445014} 2024-09-21T18:09:54.846 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:54.846 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:54.847 INFO:tasks.rados.rados.0.smithi160.stdout:3295: writing smithi16030472-149 from 548291 to 1200433 tid 1 2024-09-21T18:09:54.855 INFO:tasks.rados.rados.0.smithi160.stdout:3295: writing smithi16030472-149 from 1902233 to 2687731 tid 2 2024-09-21T18:09:54.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:09:54.862 INFO:tasks.rados.rados.0.smithi160.stdout:3295: writing smithi16030472-149 from 3303277 to 3748291 tid 3 2024-09-21T18:09:54.862 INFO:tasks.rados.rados.0.smithi160.stdout:3296: read oid 320 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3296: expect deleted 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3297: cache_try_flush oid 35 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3291: got (16) Device or resource busy 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3291: done (10 left) 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3298: cache_flush oid 175 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3299: cache_evict oid 256 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3289: finishing copy_from racing read to smithi16030472-306 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3289: finishing copy_from to smithi16030472-306 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 306 v 707 (ObjNum 976 snap 0 seq_num 976) dirty exists 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3289: done (11 left) 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3300: cache_flush oid 143 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3301: read oid 409 snap -1 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3301: expect (ObjNum 1279 snap 0 seq_num 1279) 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3302: copy_from oid 181 from oid 250 current snap is 0 2024-09-21T18:09:54.863 INFO:tasks.rados.rados.0.smithi160.stdout:3303: delete oid 496 current snap is 0 2024-09-21T18:09:54.909 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:54.908+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:54.941 INFO:tasks.rados.rados.0.smithi160.stdout:3286: finishing copy_from to smithi16030472-68 2024-09-21T18:09:54.942 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 68 v 758 (ObjNum 1114 snap 0 seq_num 1114) dirty exists 2024-09-21T18:09:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:09:54.971 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:54.969+0000 7f8c3c0ce640 -1 osd.12 116 handle_osd_map injecting map crc failure 2024-09-21T18:09:54.972 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:54.968+0000 7f440980f640 -1 osd.13 116 handle_osd_map injecting map crc failure 2024-09-21T18:09:54.988 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:54.988 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:09:55.047 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:55.045+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:55.048 INFO:tasks.rados.rados.0.smithi160.stdout:3294: finishing write tid 1 to smithi16030472-468 2024-09-21T18:09:55.064 INFO:tasks.rados.rados.0.smithi160.stdout:3295: finishing write tid 1 to smithi16030472-149 2024-09-21T18:09:55.064 INFO:tasks.rados.rados.0.smithi160.stdout:3295: finishing write tid 2 to smithi16030472-149 2024-09-21T18:09:55.064 INFO:tasks.rados.rados.0.smithi160.stdout:3295: finishing write tid 3 to smithi16030472-149 2024-09-21T18:09:55.111 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:55.109+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:55.124 INFO:tasks.rados.rados.0.smithi160.stdout:3292: got (2) No such file or directory 2024-09-21T18:09:55.126 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:55.120+0000 7f4976023640 -1 osd.5 116 handle_osd_map injecting map crc failure 2024-09-21T18:09:55.130 INFO:tasks.rados.rados.0.smithi160.stdout:3293: expect (ObjNum 1277 snap 0 seq_num 1277) 2024-09-21T18:09:55.130 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:55.173 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.174 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:09:55.212 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:55.208+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:55.262 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 410430 2024-09-21T18:09:55.312 INFO:tasks.rados.rados.0.smithi160.stdout:3299: got (2) No such file or directory 2024-09-21T18:09:55.312 INFO:tasks.rados.rados.0.smithi160.stdout:3301: expect (ObjNum 1279 snap 0 seq_num 1279) 2024-09-21T18:09:55.312 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:55.312 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:55.309+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:55.349 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:55.359 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.359 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:09:55.409 INFO:tasks.rados.rados.0.smithi160.stdout:3294: finishing write tid 2 to smithi16030472-468 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: finishing write tid 3 to smithi16030472-468 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: finishing write tid 4 to smithi16030472-468 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: finishing write tid 5 to smithi16030472-468 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: oid 468 updating version 0 to 1019 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: oid 468 version 1019 is already newer than 1018 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: oid 468 updating version 1019 to 1020 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: oid 468 updating version 1020 to 1021 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 468 v 1021 (ObjNum 1412 snap 0 seq_num 1412) dirty exists 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3294: left oid 468 (ObjNum 1412 snap 0 seq_num 1412) 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3290: finishing copy_from racing read to smithi16030472-76 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3288: finishing write tid 3 to smithi16030472-456 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3286: done (14 left) 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3292: done (13 left) 2024-09-21T18:09:55.410 INFO:tasks.rados.rados.0.smithi160.stdout:3293: done (12 left) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3294: done (11 left) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3299: done (10 left) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3301: done (9 left) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3303: done (8 left) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3304: cache_flush oid 391 snap -1 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: finishing write tid 4 to smithi16030472-456 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: finishing write tid 5 to smithi16030472-456 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: oid 456 updating version 0 to 938 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: oid 456 updating version 938 to 940 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: oid 456 version 940 is already newer than 939 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: oid 456 updating version 940 to 941 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 456 v 941 (ObjNum 1411 snap 0 seq_num 1411) dirty exists 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3288: left oid 456 (ObjNum 1411 snap 0 seq_num 1411) 2024-09-21T18:09:55.411 INFO:tasks.rados.rados.0.smithi160.stdout:3297: got (0) Success 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 35 v 0 (ObjNum 1025 snap 0 seq_num 1025) dirty exists 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3302: finishing copy_from racing read to smithi16030472-181 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3302: finishing copy_from to smithi16030472-181 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 181 v 605 (ObjNum 813 snap 0 seq_num 813) dirty exists 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3298: got (0) Success 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 0 (ObjNum 129 snap 0 seq_num 3132629064) clean dne 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3295: finishing write tid 4 to smithi16030472-149 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3288: done (8 left) 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3296: done (7 left) 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3297: done (6 left) 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3298: done (5 left) 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3302: done (4 left) 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3305: delete oid 91 current snap is 0 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3295: finishing write tid 5 to smithi16030472-149 2024-09-21T18:09:55.412 INFO:tasks.rados.rados.0.smithi160.stdout:3295: oid 149 updating version 0 to 979 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:3295: oid 149 updating version 979 to 981 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:3295: oid 149 version 981 is already newer than 978 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:3295: oid 149 version 981 is already newer than 980 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 149 v 981 (ObjNum 1413 snap 0 seq_num 1413) dirty exists 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:3295: left oid 149 (ObjNum 1413 snap 0 seq_num 1413) 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:3300: got (0) Success 2024-09-21T18:09:55.413 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 143 v 0 (ObjNum 1300 snap 0 seq_num 1300) dirty exists 2024-09-21T18:09:55.414 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:55.410+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:55.510 INFO:tasks.rados.rados.0.smithi160.stdout:3290: finishing copy_from to smithi16030472-76 2024-09-21T18:09:55.510 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 822 (ObjNum 1208 snap 0 seq_num 1208) dirty exists 2024-09-21T18:09:55.515 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:55.508+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:55.545 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.545 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:09:55.614 INFO:tasks.rados.rados.0.smithi160.stdout:3290: done (4 left) 2024-09-21T18:09:55.614 INFO:tasks.rados.rados.0.smithi160.stdout:3295: done (3 left) 2024-09-21T18:09:55.614 INFO:tasks.rados.rados.0.smithi160.stdout:3300: done (2 left) 2024-09-21T18:09:55.614 INFO:tasks.rados.rados.0.smithi160.stdout:3305: done (1 left) 2024-09-21T18:09:55.614 INFO:tasks.rados.rados.0.smithi160.stdout:3306: write (excl) oid 393 current snap is 0 2024-09-21T18:09:55.615 INFO:tasks.rados.rados.0.smithi160.stdout:3306: seq_num 1416 ranges {598712=594242,1677679=532825,2985546=430582} 2024-09-21T18:09:55.616 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:55.612+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:55.621 INFO:tasks.rados.rados.0.smithi160.stdout:3306: writing smithi16030472-393 from 598712 to 1192954 tid 1 2024-09-21T18:09:55.627 INFO:tasks.rados.rados.0.smithi160.stdout:3306: writing smithi16030472-393 from 1677679 to 2210504 tid 2 2024-09-21T18:09:55.634 INFO:tasks.rados.rados.0.smithi160.stdout:3306: writing smithi16030472-393 from 2985546 to 3416128 tid 3 2024-09-21T18:09:55.634 INFO:tasks.rados.rados.0.smithi160.stdout:3307: delete oid 47 current snap is 0 2024-09-21T18:09:55.634 INFO:tasks.rados.rados.0.smithi160.stdout:3304: got (0) Success 2024-09-21T18:09:55.634 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 391 v 0 (ObjNum 65 snap 0 seq_num 3132103712) clean dne 2024-09-21T18:09:55.682 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.682 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:09:55.717 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:55.714+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:55.736 INFO:tasks.rados.rados.0.smithi160.stdout:3306: finishing write tid 1 to smithi16030472-393 2024-09-21T18:09:55.737 INFO:tasks.rados.rados.0.smithi160.stdout:3306: finishing write tid 2 to smithi16030472-393 2024-09-21T18:09:55.818 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.818 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.818 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:55.816+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:55.828 INFO:tasks.rados.rados.0.smithi160.stdout:3304: done (2 left) 2024-09-21T18:09:55.828 INFO:tasks.rados.rados.0.smithi160.stdout:3307: done (1 left) 2024-09-21T18:09:55.828 INFO:tasks.rados.rados.0.smithi160.stdout:3308: write oid 425 current snap is 0 2024-09-21T18:09:55.828 INFO:tasks.rados.rados.0.smithi160.stdout:3308: seq_num 1418 ranges {632326=744740,1794643=536494,3040392=453126} 2024-09-21T18:09:55.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:09:55.843 INFO:tasks.rados.rados.0.smithi160.stdout:3308: writing smithi16030472-425 from 632326 to 1377066 tid 1 2024-09-21T18:09:55.851 INFO:tasks.rados.rados.0.smithi160.stdout:3308: writing smithi16030472-425 from 1794643 to 2331137 tid 2 2024-09-21T18:09:55.860 INFO:tasks.rados.rados.0.smithi160.stdout:3308: writing smithi16030472-425 from 3040392 to 3493518 tid 3 2024-09-21T18:09:55.861 INFO:tasks.rados.rados.0.smithi160.stdout:3309: write (excl) oid 397 current snap is 0 2024-09-21T18:09:55.861 INFO:tasks.rados.rados.0.smithi160.stdout:3309: seq_num 1419 ranges {649133=419989,1853125=680152,2951462=664398} 2024-09-21T18:09:55.872 INFO:tasks.rados.rados.0.smithi160.stdout:3309: writing smithi16030472-397 from 649133 to 1069122 tid 1 2024-09-21T18:09:55.886 INFO:tasks.rados.rados.0.smithi160.stdout:3309: writing smithi16030472-397 from 1853125 to 2533277 tid 2 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3309: writing smithi16030472-397 from 2951462 to 3615860 tid 3 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3310: read oid 482 snap -1 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3310: expect deleted 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3311: read oid 164 snap -1 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3311: expect deleted 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3312: write oid 31 current snap is 0 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3306: finishing write tid 3 to smithi16030472-393 2024-09-21T18:09:55.895 INFO:tasks.rados.rados.0.smithi160.stdout:3312: seq_num 1420 ranges {665940=495238,1795254=540163,2978885=475670} 2024-09-21T18:09:55.903 INFO:tasks.rados.rados.0.smithi160.stdout:3312: writing smithi16030472-31 from 665940 to 1161178 tid 1 2024-09-21T18:09:55.911 INFO:tasks.rados.rados.0.smithi160.stdout:3312: writing smithi16030472-31 from 1795254 to 2335417 tid 2 2024-09-21T18:09:55.919 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:55.912+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:55.919 INFO:tasks.rados.rados.0.smithi160.stdout:3312: writing smithi16030472-31 from 2978885 to 3454555 tid 3 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3313: read oid 289 snap -1 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3313: expect deleted 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3314: cache_try_flush oid 421 snap -1 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: finishing write tid 4 to smithi16030472-393 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: finishing write tid 5 to smithi16030472-393 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: oid 393 updating version 0 to 987 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: oid 393 version 987 is already newer than 986 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: oid 393 version 987 is already newer than 984 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: oid 393 version 987 is already newer than 985 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 987 (ObjNum 1416 snap 0 seq_num 1416) dirty exists 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: left oid 393 (ObjNum 1416 snap 0 seq_num 1416) 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3306: done (7 left) 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3315: write (excl) oid 21 current snap is 0 2024-09-21T18:09:55.920 INFO:tasks.rados.rados.0.smithi160.stdout:3315: seq_num 1421 ranges {682747=570487,1853736=683821,3006308=403295} 2024-09-21T18:09:55.929 INFO:tasks.rados.rados.0.smithi160.stdout:3315: writing smithi16030472-21 from 682747 to 1253234 tid 1 2024-09-21T18:09:55.938 INFO:tasks.rados.rados.0.smithi160.stdout:3315: writing smithi16030472-21 from 1853736 to 2537557 tid 2 2024-09-21T18:09:55.944 INFO:tasks.rados.rados.0.smithi160.stdout:3315: writing smithi16030472-21 from 3006308 to 3409603 tid 3 2024-09-21T18:09:55.944 INFO:tasks.rados.rados.0.smithi160.stdout:3316: cache_evict oid 29 snap -1 2024-09-21T18:09:55.944 INFO:tasks.rados.rados.0.smithi160.stdout:3317: read oid 303 snap -1 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3317: expect (ObjNum 824 snap 0 seq_num 824) 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3318: copy_from oid 385 from oid 435 current snap is 0 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3319: copy_from oid 284 from oid 207 current snap is 0 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3320: read oid 392 snap -1 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3320: expect deleted 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3321: cache_try_flush oid 389 snap -1 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3322: cache_flush oid 243 snap -1 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout:3323: cache_try_flush oid 227 snap -1 2024-09-21T18:09:55.945 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:55.955 INFO:tasks.rados.rados.0.smithi160.stdout:3311: done (15 left) 2024-09-21T18:09:55.955 INFO:tasks.rados.rados.0.smithi160.stdout:3324: copy_from oid 1 from oid 222 current snap is 0 2024-09-21T18:09:55.955 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:55.961 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:55.961 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:55.970 INFO:tasks.rados.rados.0.smithi160.stdout:3315: finishing write tid 1 to smithi16030472-21 2024-09-21T18:09:55.977 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:09:55.980 INFO:tasks.rados.rados.0.smithi160.stdout:3309: finishing write tid 1 to smithi16030472-397 2024-09-21T18:09:55.990 INFO:tasks.rados.rados.0.smithi160.stdout:3313: done (15 left) 2024-09-21T18:09:55.990 INFO:tasks.rados.rados.0.smithi160.stdout:3325: cache_evict oid 125 snap -1 2024-09-21T18:09:55.990 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:55.990 INFO:tasks.rados.rados.0.smithi160.stdout:3309: finishing write tid 2 to smithi16030472-397 2024-09-21T18:09:55.990 INFO:tasks.rados.rados.0.smithi160.stdout:3309: finishing write tid 3 to smithi16030472-397 2024-09-21T18:09:55.992 INFO:tasks.rados.rados.0.smithi160.stdout:3320: done (15 left) 2024-09-21T18:09:55.992 INFO:tasks.rados.rados.0.smithi160.stdout:3326: write (excl) oid 46 current snap is 0 2024-09-21T18:09:55.992 INFO:tasks.rados.rados.0.smithi160.stdout:3326: seq_num 1422 ranges {699554=762089,2028571=543832,3150084=614567} 2024-09-21T18:09:56.004 INFO:tasks.rados.rados.0.smithi160.stdout:3326: writing smithi16030472-46 from 699554 to 1461643 tid 1 2024-09-21T18:09:56.013 INFO:tasks.rados.rados.0.smithi160.stdout:3326: writing smithi16030472-46 from 2028571 to 2572403 tid 2 2024-09-21T18:09:56.019 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:56.012+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:09:56.022 INFO:tasks.rados.rados.0.smithi160.stdout:3326: writing smithi16030472-46 from 3150084 to 3764651 tid 3 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout:3322: got (0) Success 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 243 v 0 (ObjNum 1227 snap 0 seq_num 1227) dirty exists 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout:3322: done (15 left) 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout:3327: copy_from oid 91 from oid 65 current snap is 0 2024-09-21T18:09:56.023 INFO:tasks.rados.rados.0.smithi160.stdout:3317: expect (ObjNum 824 snap 0 seq_num 824) 2024-09-21T18:09:56.099 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2979418 2024-09-21T18:09:56.111 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.111 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.119 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:56.112+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:56.126 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:09:56.140 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:56.220 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:56.216+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:56.245 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.245 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.245 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2723482 2024-09-21T18:09:56.259 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:09:56.286 INFO:tasks.rados.rados.0.smithi160.stdout:3309: finishing write tid 4 to smithi16030472-397 2024-09-21T18:09:56.286 INFO:tasks.rados.rados.0.smithi160.stdout:3309: finishing write tid 5 to smithi16030472-397 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3309: oid 397 updating version 0 to 1023 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3309: oid 397 updating version 1023 to 1026 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3309: oid 397 version 1026 is already newer than 1025 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3309: oid 397 version 1026 is already newer than 1024 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 397 v 1026 (ObjNum 1419 snap 0 seq_num 1419) dirty exists 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3309: left oid 397 (ObjNum 1419 snap 0 seq_num 1419) 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3312: finishing write tid 1 to smithi16030472-31 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3312: finishing write tid 2 to smithi16030472-31 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: finishing write tid 2 to smithi16030472-21 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: finishing write tid 3 to smithi16030472-21 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: finishing write tid 4 to smithi16030472-21 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: finishing write tid 5 to smithi16030472-21 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: oid 21 updating version 0 to 943 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: oid 21 updating version 943 to 945 2024-09-21T18:09:56.287 INFO:tasks.rados.rados.0.smithi160.stdout:3315: oid 21 version 945 is already newer than 944 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3315: oid 21 updating version 945 to 946 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 946 (ObjNum 1421 snap 0 seq_num 1421) dirty exists 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3315: left oid 21 (ObjNum 1421 snap 0 seq_num 1421) 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3309: done (15 left) 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3310: done (14 left) 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3315: done (13 left) 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3317: done (12 left) 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3328: write oid 303 current snap is 0 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3314: got (0) Success 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 421 v 0 (ObjNum 1119 snap 0 seq_num 1119) dirty exists 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3323: got (0) Success 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 227 v 0 (ObjNum 1152 snap 0 seq_num 1152) dirty exists 2024-09-21T18:09:56.288 INFO:tasks.rados.rados.0.smithi160.stdout:3328: seq_num 1423 ranges {716361=437338,1687053=687490,2777507=425839,3877756=38605} 2024-09-21T18:09:56.296 INFO:tasks.rados.rados.0.smithi160.stdout:3328: writing smithi16030472-303 from 716361 to 1153699 tid 1 2024-09-21T18:09:56.306 INFO:tasks.rados.rados.0.smithi160.stdout:3328: writing smithi16030472-303 from 1687053 to 2374543 tid 2 2024-09-21T18:09:56.313 INFO:tasks.rados.rados.0.smithi160.stdout:3328: writing smithi16030472-303 from 2777507 to 3203346 tid 3 2024-09-21T18:09:56.320 INFO:tasks.rados.rados.0.smithi160.stdout:3328: writing smithi16030472-303 from 3877756 to 3916361 tid 4 2024-09-21T18:09:56.320 INFO:tasks.rados.rados.0.smithi160.stdout:3314: done (12 left) 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3323: done (11 left) 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3329: read oid 267 snap -1 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3329: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3330: cache_flush oid 359 snap -1 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: finishing write tid 3 to smithi16030472-31 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3331: delete oid 32 current snap is 0 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: finishing write tid 4 to smithi16030472-31 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: finishing write tid 5 to smithi16030472-31 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: oid 31 updating version 0 to 989 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: oid 31 version 989 is already newer than 988 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: oid 31 updating version 989 to 991 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: oid 31 version 991 is already newer than 990 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 991 (ObjNum 1420 snap 0 seq_num 1420) dirty exists 2024-09-21T18:09:56.321 INFO:tasks.rados.rados.0.smithi160.stdout:3312: left oid 31 (ObjNum 1420 snap 0 seq_num 1420) 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: finishing write tid 1 to smithi16030472-46 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: finishing write tid 2 to smithi16030472-46 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: finishing write tid 3 to smithi16030472-46 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: finishing write tid 4 to smithi16030472-46 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: finishing write tid 5 to smithi16030472-46 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: oid 46 updating version 0 to 993 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: oid 46 updating version 993 to 994 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: oid 46 version 994 is already newer than 992 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: oid 46 updating version 994 to 995 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 46 v 995 (ObjNum 1422 snap 0 seq_num 1422) dirty exists 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3326: left oid 46 (ObjNum 1422 snap 0 seq_num 1422) 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3308: finishing write tid 1 to smithi16030472-425 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3308: finishing write tid 2 to smithi16030472-425 2024-09-21T18:09:56.322 INFO:tasks.rados.rados.0.smithi160.stdout:3308: finishing write tid 3 to smithi16030472-425 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: finishing write tid 4 to smithi16030472-425 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: finishing write tid 5 to smithi16030472-425 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: oid 425 updating version 0 to 983 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: oid 425 updating version 983 to 984 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: oid 425 updating version 984 to 986 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: oid 425 version 986 is already newer than 985 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 425 v 986 (ObjNum 1418 snap 0 seq_num 1418) dirty exists 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3308: left oid 425 (ObjNum 1418 snap 0 seq_num 1418) 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3316: got (0) Success 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:3321: got (0) Success 2024-09-21T18:09:56.323 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 389 v 0 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:09:56.347 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:56.344+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:56.365 INFO:tasks.rados.rados.0.smithi160.stdout:3325: got (16) Device or resource busy 2024-09-21T18:09:56.395 INFO:tasks.rados.rados.0.smithi160.stdout:3318: finishing copy_from racing read to smithi16030472-385 2024-09-21T18:09:56.397 INFO:tasks.rados.rados.0.smithi160.stdout:3318: finishing copy_from to smithi16030472-385 2024-09-21T18:09:56.397 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 385 v 471 (ObjNum 687 snap 0 seq_num 687) dirty exists 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3308: done (13 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3312: done (12 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3316: done (11 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3318: done (10 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3321: done (9 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3325: done (8 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3326: done (7 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3331: done (6 left) 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3332: cache_try_flush oid 164 snap -1 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3333: cache_evict oid 252 snap -1 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3334: cache_try_flush oid 47 snap -1 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3335: write oid 72 current snap is 0 2024-09-21T18:09:56.415 INFO:tasks.rados.rados.0.smithi160.stdout:3335: seq_num 1425 ranges {749975=587836,2087664=691159,3516000=433975} 2024-09-21T18:09:56.425 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:09:56.421+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:09:56.428 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.428 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.431 INFO:tasks.rados.rados.0.smithi160.stdout:3335: writing smithi16030472-72 from 749975 to 1337811 tid 1 2024-09-21T18:09:56.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:09:56.448 INFO:tasks.rados.rados.0.smithi160.stdout:3335: writing smithi16030472-72 from 2087664 to 2778823 tid 2 2024-09-21T18:09:56.461 INFO:tasks.rados.rados.0.smithi160.stdout:3335: writing smithi16030472-72 from 3516000 to 3949975 tid 3 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3336: copy_from oid 269 from oid 392 current snap is 0 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3333: got (16) Device or resource busy 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3332: got (0) Success 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 0 (ObjNum 129 snap 0 seq_num 3145242888) clean dne 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3334: got (0) Success 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 47 v 0 (ObjNum 129 snap 0 seq_num 3128293640) clean dne 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3324: finishing copy_from racing read to smithi16030472-1 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3319: finishing copy_from racing read to smithi16030472-284 2024-09-21T18:09:56.462 INFO:tasks.rados.rados.0.smithi160.stdout:3329: expect (ObjNum 266 snap 0 seq_num 266) 2024-09-21T18:09:56.467 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1707451 2024-09-21T18:09:56.491 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:56.531 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:56.528+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 1 to smithi16030472-303 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 2 to smithi16030472-303 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:3319: finishing copy_from to smithi16030472-284 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 284 v 949 (ObjNum 1362 snap 0 seq_num 1362) dirty exists 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:3324: finishing copy_from to smithi16030472-1 2024-09-21T18:09:56.559 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 1 v 703 (ObjNum 988 snap 0 seq_num 988) dirty exists 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 3 to smithi16030472-303 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 4 to smithi16030472-303 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3319: done (10 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3324: done (9 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3329: done (8 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3332: done (7 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3333: done (6 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3334: done (5 left) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3337: cache_try_flush oid 281 snap -1 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3338: read oid 297 snap -1 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3338: expect (ObjNum 1230 snap 0 seq_num 1230) 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3339: cache_try_flush oid 55 snap -1 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3340: copy_from oid 134 from oid 17 current snap is 0 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3341: read oid 247 snap -1 2024-09-21T18:09:56.560 INFO:tasks.rados.rados.0.smithi160.stdout:3341: expect (ObjNum 1393 snap 0 seq_num 1393) 2024-09-21T18:09:56.561 INFO:tasks.rados.rados.0.smithi160.stdout:3342: write oid 490 current snap is 0 2024-09-21T18:09:56.561 INFO:tasks.rados.rados.0.smithi160.stdout:3342: seq_num 1426 ranges {766782=663085,2146146=551170,3143423=776008} 2024-09-21T18:09:56.577 INFO:tasks.rados.rados.0.smithi160.stdout:3342: writing smithi16030472-490 from 766782 to 1429867 tid 1 2024-09-21T18:09:56.592 INFO:tasks.rados.rados.0.smithi160.stdout:3342: writing smithi16030472-490 from 2146146 to 2697316 tid 2 2024-09-21T18:09:56.606 INFO:tasks.rados.rados.0.smithi160.stdout:3342: writing smithi16030472-490 from 3143423 to 3919431 tid 3 2024-09-21T18:09:56.607 INFO:tasks.rados.rados.0.smithi160.stdout:3343: read oid 114 snap -1 2024-09-21T18:09:56.607 INFO:tasks.rados.rados.0.smithi160.stdout:3343: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:09:56.607 INFO:tasks.rados.rados.0.smithi160.stdout:3344: write oid 187 current snap is 0 2024-09-21T18:09:56.607 INFO:tasks.rados.rados.0.smithi160.stdout:3344: seq_num 1427 ranges {783589=738334,2204628=694828,3570846=412743} 2024-09-21T18:09:56.615 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.615 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.621 INFO:tasks.rados.rados.0.smithi160.stdout:3344: writing smithi16030472-187 from 783589 to 1521923 tid 1 2024-09-21T18:09:56.627 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:56.622+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:56.630 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:09:56.632 INFO:tasks.rados.rados.0.smithi160.stdout:3344: writing smithi16030472-187 from 2204628 to 2899456 tid 2 2024-09-21T18:09:56.640 INFO:tasks.rados.rados.0.smithi160.stdout:3344: writing smithi16030472-187 from 3570846 to 3983589 tid 3 2024-09-21T18:09:56.640 INFO:tasks.rados.rados.0.smithi160.stdout:3345: write oid 130 current snap is 0 2024-09-21T18:09:56.640 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 5 to smithi16030472-303 2024-09-21T18:09:56.640 INFO:tasks.rados.rados.0.smithi160.stdout:3345: seq_num 1428 ranges {400396=413583,1463110=438486} 2024-09-21T18:09:56.645 INFO:tasks.rados.rados.0.smithi160.stdout:3345: writing smithi16030472-130 from 400396 to 813979 tid 1 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3345: writing smithi16030472-130 from 1463110 to 1901596 tid 2 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3346: read oid 429 snap -1 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3346: expect (ObjNum 428 snap 0 seq_num 428) 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3347: write oid 204 current snap is 0 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3328: finishing write tid 6 to smithi16030472-303 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3328: oid 303 updating version 0 to 1000 2024-09-21T18:09:56.653 INFO:tasks.rados.rados.0.smithi160.stdout:3328: oid 303 updating version 1000 to 1001 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:3328: oid 303 version 1001 is already newer than 997 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:3328: oid 303 updating version 1001 to 1002 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:3328: oid 303 version 1002 is already newer than 996 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 303 v 1002 (ObjNum 1423 snap 0 seq_num 1423) dirty exists 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:3328: left oid 303 (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:09:56.654 INFO:tasks.rados.rados.0.smithi160.stdout:3347: seq_num 1429 ranges {417203=488832,1405239=611964} 2024-09-21T18:09:56.658 INFO:tasks.rados.rados.0.smithi160.stdout:3347: writing smithi16030472-204 from 417203 to 906035 tid 1 2024-09-21T18:09:56.666 INFO:tasks.rados.rados.0.smithi160.stdout:3347: writing smithi16030472-204 from 1405239 to 2017203 tid 2 2024-09-21T18:09:56.666 INFO:tasks.rados.rados.0.smithi160.stdout:3328: done (15 left) 2024-09-21T18:09:56.666 INFO:tasks.rados.rados.0.smithi160.stdout:3348: cache_flush oid 34 snap -1 2024-09-21T18:09:56.666 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:56.666 INFO:tasks.rados.rados.0.smithi160.stdout:3338: expect (ObjNum 1230 snap 0 seq_num 1230) 2024-09-21T18:09:56.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:09:56.728 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:56.725+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:56.770 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 490545 2024-09-21T18:09:56.804 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.804 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.817 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:56.818 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:09:56.829 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:56.825+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:56.868 INFO:tasks.rados.rados.0.smithi160.stdout:3341: expect (ObjNum 1393 snap 0 seq_num 1393) 2024-09-21T18:09:56.868 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:56.890 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:56.890 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:56.909 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:56.910 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:56.916 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2035886 2024-09-21T18:09:56.929 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:56.930 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:09:56.926+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:09:56.930 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:56.930 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:56.940 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:56.940 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:56.951 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:56.951 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:56.954 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:3336: finishing copy_from to smithi16030472-269 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:3336: got expected ENOENT (src dne) 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:3327: finishing copy_from to smithi16030472-91 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 880 (ObjNum 1277 snap 0 seq_num 1277) dirty exists 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:3327: finishing copy_from racing read to smithi16030472-91 2024-09-21T18:09:56.966 INFO:tasks.rados.rados.0.smithi160.stdout:3343: expect (ObjNum 841 snap 0 seq_num 841) 2024-09-21T18:09:56.971 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:56.971 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:56.991 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:56.991 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:56.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:09:57.008 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:57.013 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:57.030 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:57.025+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:57.033 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:57.033 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:57.056 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:57.056 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:57.073 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.073 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:57.073 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:57.073 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.079 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:57.079 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3344: finishing write tid 1 to smithi16030472-187 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: finishing write tid 1 to smithi16030472-72 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: finishing write tid 2 to smithi16030472-72 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3336: finishing copy_from racing read to smithi16030472-269 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3344: finishing write tid 2 to smithi16030472-187 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: finishing write tid 3 to smithi16030472-72 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: finishing write tid 4 to smithi16030472-72 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: finishing write tid 5 to smithi16030472-72 2024-09-21T18:09:57.081 INFO:tasks.rados.rados.0.smithi160.stdout:3335: oid 72 updating version 0 to 875 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3335: oid 72 updating version 875 to 876 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3335: oid 72 version 876 is already newer than 874 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3335: oid 72 updating version 876 to 877 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 72 v 877 (ObjNum 1425 snap 0 seq_num 1425) dirty exists 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3335: left oid 72 (ObjNum 1425 snap 0 seq_num 1425) 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3339: got (0) Success 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 55 v 0 (ObjNum 1271 snap 0 seq_num 1271) dirty exists 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3344: finishing write tid 3 to smithi16030472-187 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3330: got (0) Success 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 0 (ObjNum 1364 snap 0 seq_num 1364) dirty exists 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3344: finishing write tid 4 to smithi16030472-187 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3337: got (0) Success 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 281 v 0 (ObjNum 974 snap 0 seq_num 974) dirty exists 2024-09-21T18:09:57.082 INFO:tasks.rados.rados.0.smithi160.stdout:3340: finishing copy_from racing read to smithi16030472-134 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: finishing write tid 5 to smithi16030472-187 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: oid 187 updating version 0 to 1031 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: oid 187 version 1031 is already newer than 1029 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: oid 187 version 1031 is already newer than 1030 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: oid 187 version 1031 is already newer than 1028 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 1031 (ObjNum 1427 snap 0 seq_num 1427) dirty exists 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3344: left oid 187 (ObjNum 1427 snap 0 seq_num 1427) 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3347: finishing write tid 1 to smithi16030472-204 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3347: finishing write tid 2 to smithi16030472-204 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3347: finishing write tid 3 to smithi16030472-204 2024-09-21T18:09:57.083 INFO:tasks.rados.rados.0.smithi160.stdout:3346: expect (ObjNum 428 snap 0 seq_num 428) 2024-09-21T18:09:57.088 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:09:57.095 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2685016 2024-09-21T18:09:57.101 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:57.102 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:57.102 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:57.122 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:57.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:57.132 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:57.126+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:57.135 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 374968 2024-09-21T18:09:57.143 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:57.143 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:57.164 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:57.164 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:57.168 INFO:tasks.rados.rados.0.smithi160.stdout:3342: finishing write tid 1 to smithi16030472-490 2024-09-21T18:09:57.168 INFO:tasks.rados.rados.0.smithi160.stdout:3342: finishing write tid 2 to smithi16030472-490 2024-09-21T18:09:57.168 INFO:tasks.rados.rados.0.smithi160.stdout:3340: finishing copy_from to smithi16030472-134 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 134 v 329 (ObjNum 532 snap 0 seq_num 532) dirty exists 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3342: finishing write tid 3 to smithi16030472-490 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3348: got (0) Success 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 0 (ObjNum 1114 snap 0 seq_num 1114) dirty exists 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3347: finishing write tid 4 to smithi16030472-204 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3347: oid 204 updating version 0 to 1033 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3347: oid 204 version 1033 is already newer than 1032 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3347: oid 204 updating version 1033 to 1034 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 204 v 1034 (ObjNum 1429 snap 0 seq_num 1429) dirty exists 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3347: left oid 204 (ObjNum 1429 snap 0 seq_num 1429) 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3327: done (15 left) 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3330: done (14 left) 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3335: done (13 left) 2024-09-21T18:09:57.169 INFO:tasks.rados.rados.0.smithi160.stdout:3336: done (12 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3337: done (11 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3338: done (10 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3339: done (9 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3340: done (8 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3341: done (7 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3343: done (6 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3344: done (5 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3346: done (4 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3347: done (3 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3348: done (2 left) 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3349: cache_evict oid 263 snap -1 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3350: write oid 103 current snap is 0 2024-09-21T18:09:57.170 INFO:tasks.rados.rados.0.smithi160.stdout:3350: seq_num 1430 ranges {434010=680434,1580074=442155} 2024-09-21T18:09:57.186 INFO:tasks.rados.rados.0.smithi160.stdout:3350: writing smithi16030472-103 from 434010 to 1114444 tid 1 2024-09-21T18:09:57.186 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:57.186 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:57.191 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3350: writing smithi16030472-103 from 1580074 to 2022229 tid 2 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3351: delete oid 409 current snap is 0 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3349: got (16) Device or resource busy 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3342: finishing write tid 4 to smithi16030472-490 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3342: finishing write tid 5 to smithi16030472-490 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3342: oid 490 updating version 0 to 1009 2024-09-21T18:09:57.201 INFO:tasks.rados.rados.0.smithi160.stdout:3342: oid 490 version 1009 is already newer than 1005 2024-09-21T18:09:57.202 INFO:tasks.rados.rados.0.smithi160.stdout:3342: oid 490 version 1009 is already newer than 1008 2024-09-21T18:09:57.202 INFO:tasks.rados.rados.0.smithi160.stdout:3342: oid 490 version 1009 is already newer than 1004 2024-09-21T18:09:57.202 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 490 v 1009 (ObjNum 1426 snap 0 seq_num 1426) dirty exists 2024-09-21T18:09:57.202 INFO:tasks.rados.rados.0.smithi160.stdout:3342: left oid 490 (ObjNum 1426 snap 0 seq_num 1426) 2024-09-21T18:09:57.213 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.214 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:57.214 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:57.214 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:57.214 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.228 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:09:57.235 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:57.228+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:57.295 INFO:tasks.rados.rados.0.smithi160.stdout:3345: finishing write tid 1 to smithi16030472-130 2024-09-21T18:09:57.296 INFO:tasks.rados.rados.0.smithi160.stdout:3345: finishing write tid 2 to smithi16030472-130 2024-09-21T18:09:57.336 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:57.330+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:57.357 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 146.32529085900001, 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.358 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:09:57.359 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:09:57.360 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.361 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:57.362 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 146.321680308, 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:09:57.363 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.364 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.365 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.366 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 146.31887376899999, 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:57.367 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.368 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.369 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.370 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 146.315125569, 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.371 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.372 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.373 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.374 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:57.375 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.69427476499999, 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.376 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:09:57.377 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:09:57.378 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.379 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.59342069300001, 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.380 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.381 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.382 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.383 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.59266441299999, 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:57.384 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.385 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.386 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.387 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.59227285599999, 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.388 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:09:57.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:57.392 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.59197173199999, 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:09:57.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:09:57.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:09:57.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.591247882, 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:09:57.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:09:57.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.490351043, 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.401 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:09:57.405 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.49025675999999, 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:09:57.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.407 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:09:57.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.409 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.48934356500001, 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.488773877, 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.414 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.418 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.48804639299999, 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.422 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 145.421864484, 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:57.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:09:57.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:09:57.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 144.16581867900001, 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:09:57.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:09:57.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 144.16553588100001, 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.431 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 144.16545689599999, 2024-09-21T18:09:57.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.439 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 144.16283856499999, 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:09:57.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:09:57.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.443 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.444 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:09:57.444 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:09:57.444 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.445 INFO:tasks.rados.rados.0.smithi160.stdout:3350: finishing write tid 1 to smithi16030472-103 2024-09-21T18:09:57.445 INFO:tasks.rados.rados.0.smithi160.stdout:3350: finishing write tid 2 to smithi16030472-103 2024-09-21T18:09:57.446 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:57.440+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:57.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:09:57.495 INFO:tasks.rados.rados.0.smithi160.stdout:3345: finishing write tid 3 to smithi16030472-130 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:3345: finishing write tid 4 to smithi16030472-130 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:3345: oid 130 updating version 0 to 1010 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:3345: oid 130 updating version 1010 to 1011 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:3345: oid 130 updating version 1011 to 1012 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 130 v 1012 (ObjNum 1428 snap 0 seq_num 1428) dirty exists 2024-09-21T18:09:57.496 INFO:tasks.rados.rados.0.smithi160.stdout:3345: left oid 130 (ObjNum 1428 snap 0 seq_num 1428) 2024-09-21T18:09:57.499 INFO:tasks.rados.rados.0.smithi160.stdout:3350: finishing write tid 3 to smithi16030472-103 2024-09-21T18:09:57.500 INFO:tasks.rados.rados.0.smithi160.stdout:3350: finishing write tid 4 to smithi16030472-103 2024-09-21T18:09:57.500 INFO:tasks.rados.rados.0.smithi160.stdout:3350: oid 103 updating version 0 to 948 2024-09-21T18:09:57.500 INFO:tasks.rados.rados.0.smithi160.stdout:3350: oid 103 updating version 948 to 949 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3350: oid 103 version 949 is already newer than 947 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 103 v 949 (ObjNum 1430 snap 0 seq_num 1430) dirty exists 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3350: left oid 103 (ObjNum 1430 snap 0 seq_num 1430) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3342: done (4 left) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3345: done (3 left) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3349: done (2 left) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3350: done (1 left) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3351: done (0 left) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3352: cache_try_flush oid 99 snap -1 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3353: read oid 222 snap -1 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3353: expect (ObjNum 988 snap 0 seq_num 988) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3354: cache_evict oid 247 snap -1 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3355: copy_from oid 406 from oid 287 current snap is 0 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3356: read oid 365 snap -1 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3356: expect (ObjNum 975 snap 0 seq_num 975) 2024-09-21T18:09:57.501 INFO:tasks.rados.rados.0.smithi160.stdout:3357: read oid 169 snap -1 2024-09-21T18:09:57.502 INFO:tasks.rados.rados.0.smithi160.stdout:3357: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:09:57.502 INFO:tasks.rados.rados.0.smithi160.stdout:3358: delete oid 145 current snap is 0 2024-09-21T18:09:57.502 INFO:tasks.rados.rados.0.smithi160.stdout:3354: got (16) Device or resource busy 2024-09-21T18:09:57.546 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:57.540+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:57.579 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.579 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:57.580 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:09:57.580 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.594 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:09:57.611 INFO:tasks.rados.rados.0.smithi160.stdout:3356: expect (ObjNum 975 snap 0 seq_num 975) 2024-09-21T18:09:57.611 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:57.647 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:57.645+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:57.715 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.715 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:09:57.715 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:09:57.715 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:09:57.715 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.730 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:09:57.760 INFO:tasks.rados.rados.0.smithi160.stdout:3357: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:09:57.775 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:57.772+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:57.811 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1975957 2024-09-21T18:09:57.836 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 936650 2024-09-21T18:09:57.849 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:57.842+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:57.857 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:09:57.857 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:09:57.857 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:09:57.857 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:09:57.857 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:09:57.867 INFO:tasks.rados.rados.0.smithi160.stdout:3353: expect (ObjNum 988 snap 0 seq_num 988) 2024-09-21T18:09:57.867 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:57.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:09:57.923 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2052760 2024-09-21T18:09:57.928 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:57.949 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:57.944+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:57.965 INFO:tasks.rados.rados.0.smithi160.stdout:3355: finishing copy_from to smithi16030472-406 2024-09-21T18:09:57.965 INFO:tasks.rados.rados.0.smithi160.stdout:3355: got expected ENOENT (src dne) 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3355: finishing copy_from racing read to smithi16030472-406 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3352: got (0) Success 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 0 (ObjNum -1153800000 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3352: done (6 left) 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3353: done (5 left) 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3354: done (4 left) 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3355: done (3 left) 2024-09-21T18:09:57.966 INFO:tasks.rados.rados.0.smithi160.stdout:3356: done (2 left) 2024-09-21T18:09:57.967 INFO:tasks.rados.rados.0.smithi160.stdout:3357: done (1 left) 2024-09-21T18:09:57.967 INFO:tasks.rados.rados.0.smithi160.stdout:3358: done (0 left) 2024-09-21T18:09:57.967 INFO:tasks.rados.rados.0.smithi160.stdout:3359: delete oid 124 current snap is 0 2024-09-21T18:09:57.975 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:09:58.006 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.006 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:58.006 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:58.006 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.021 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:09:58.050 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:58.044+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:58.115 INFO:tasks.rados.rados.0.smithi160.stdout:3359: done (0 left) 2024-09-21T18:09:58.115 INFO:tasks.rados.rados.0.smithi160.stdout:3360: read oid 126 snap -1 2024-09-21T18:09:58.115 INFO:tasks.rados.rados.0.smithi160.stdout:3360: expect deleted 2024-09-21T18:09:58.115 INFO:tasks.rados.rados.0.smithi160.stdout:3361: delete oid 103 current snap is 0 2024-09-21T18:09:58.143 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.143 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:58.143 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:58.143 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:58.143 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.151 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:58.149+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:58.157 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:09:58.220 INFO:tasks.rados.rados.0.smithi160.stdout:3360: done (1 left) 2024-09-21T18:09:58.220 INFO:tasks.rados.rados.0.smithi160.stdout:3361: done (0 left) 2024-09-21T18:09:58.220 INFO:tasks.rados.rados.0.smithi160.stdout:3362: cache_evict oid 337 snap -1 2024-09-21T18:09:58.220 INFO:tasks.rados.rados.0.smithi160.stdout:3363: write (excl) oid 381 current snap is 0 2024-09-21T18:09:58.220 INFO:tasks.rados.rados.0.smithi160.stdout:3363: seq_num 1435 ranges {518045=656679,1756131=361914} 2024-09-21T18:09:58.239 INFO:tasks.rados.rados.0.smithi160.stdout:3363: writing smithi16030472-381 from 518045 to 1174724 tid 1 2024-09-21T18:09:58.251 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:09:58.246+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3363: writing smithi16030472-381 from 1756131 to 2118045 tid 2 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3364: copy_from oid 449 from oid 272 current snap is 0 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3362: got (16) Device or resource busy 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3362: done (2 left) 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3365: read oid 486 snap -1 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3365: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3366: read oid 391 snap -1 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3366: expect deleted 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3367: copy_from oid 266 from oid 94 current snap is 0 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3368: write (excl) oid 313 current snap is 0 2024-09-21T18:09:58.255 INFO:tasks.rados.rados.0.smithi160.stdout:3368: seq_num 1436 ranges {534852=731928,1814613=320239} 2024-09-21T18:09:58.275 INFO:tasks.rados.rados.0.smithi160.stdout:3368: writing smithi16030472-313 from 534852 to 1266780 tid 1 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:09:58.279 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 159.945527199, 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:09:58.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:09:58.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.105675203, 2024-09-21T18:09:58.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:09:58.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.00195475500001, 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:09:58.287 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:09:58.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:09:58.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.00118001199999, 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.291 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:09:58.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.00053199300001, 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.295 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.986660298, 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.299 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.98371928200001, 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.98070243399999, 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:09:58.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.97849085199999, 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:09:58.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.87315376000001, 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:58.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:09:58.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:09:58.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.318 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.181731386, 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:09:58.323 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:58.323 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 153.178701386, 2024-09-21T18:09:58.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:09:58.325 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.325 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:09:58.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:09:58.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:09:58.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 148.21213742399999, 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.329 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:09:58.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:09:58.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 146.545301989, 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.333 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:09:58.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.24181090799999, 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:09:58.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:09:58.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.239513875, 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:58.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:09:58.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.344 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.23746342699999, 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:09:58.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:09:58.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.348 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.23530806599999, 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:58.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:09:58.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:09:58.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:09:58.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.23454530800001, 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:58.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:09:58.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:09:58.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:09:58.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 139.13264026300001, 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:58.357 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:58.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:09:58.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3368: writing smithi16030472-313 from 1814613 to 2134852 tid 2 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3369: cache_flush oid 244 snap -1 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3370: cache_flush oid 236 snap -1 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3371: cache_try_flush oid 361 snap -1 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3372: copy_from oid 374 from oid 34 current snap is 0 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3373: read oid 179 snap -1 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3373: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:58.364 INFO:tasks.rados.rados.0.smithi160.stdout:3374: delete oid 252 current snap is 0 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:3368: finishing write tid 1 to smithi16030472-313 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:3368: finishing write tid 2 to smithi16030472-313 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:3368: finishing write tid 3 to smithi16030472-313 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:3369: got (0) Success 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 244 v 0 (ObjNum 243 snap 0 seq_num 243) dirty exists 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:3370: got (0) Success 2024-09-21T18:09:58.365 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 236 v 0 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:09:58.365 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:58.360+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:58.377 INFO:tasks.rados.rados.0.smithi160.stdout:3368: finishing write tid 4 to smithi16030472-313 2024-09-21T18:09:58.377 INFO:tasks.rados.rados.0.smithi160.stdout:3368: oid 313 updating version 0 to 1038 2024-09-21T18:09:58.378 INFO:tasks.rados.rados.0.smithi160.stdout:3368: oid 313 version 1038 is already newer than 1036 2024-09-21T18:09:58.378 INFO:tasks.rados.rados.0.smithi160.stdout:3368: oid 313 version 1038 is already newer than 1037 2024-09-21T18:09:58.378 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 313 v 1038 (ObjNum 1436 snap 0 seq_num 1436) dirty exists 2024-09-21T18:09:58.378 INFO:tasks.rados.rados.0.smithi160.stdout:3368: left oid 313 (ObjNum 1436 snap 0 seq_num 1436) 2024-09-21T18:09:58.405 INFO:tasks.rados.rados.0.smithi160.stdout:3366: done (11 left) 2024-09-21T18:09:58.405 INFO:tasks.rados.rados.0.smithi160.stdout:3368: done (10 left) 2024-09-21T18:09:58.405 INFO:tasks.rados.rados.0.smithi160.stdout:3369: done (9 left) 2024-09-21T18:09:58.406 INFO:tasks.rados.rados.0.smithi160.stdout:3370: done (8 left) 2024-09-21T18:09:58.406 INFO:tasks.rados.rados.0.smithi160.stdout:3374: done (7 left) 2024-09-21T18:09:58.406 INFO:tasks.rados.rados.0.smithi160.stdout:3375: write oid 293 current snap is 0 2024-09-21T18:09:58.406 INFO:tasks.rados.rados.0.smithi160.stdout:3375: seq_num 1438 ranges {568466=598779,1931577=236889} 2024-09-21T18:09:58.420 INFO:tasks.rados.rados.0.smithi160.stdout:3375: writing smithi16030472-293 from 568466 to 1167245 tid 1 2024-09-21T18:09:58.431 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:58.426+0000 7fca4e93e640 -1 osd.2 117 handle_osd_map injecting map crc failure 2024-09-21T18:09:58.432 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:58.426+0000 7fca4e93e640 -1 osd.2 117 handle_osd_map injecting map crc failure 2024-09-21T18:09:58.433 INFO:tasks.rados.rados.0.smithi160.stdout:3375: writing smithi16030472-293 from 1931577 to 2168466 tid 2 2024-09-21T18:09:58.433 INFO:tasks.rados.rados.0.smithi160.stdout:3376: cache_evict oid 413 snap -1 2024-09-21T18:09:58.433 INFO:tasks.rados.rados.0.smithi160.stdout:3377: cache_flush oid 430 snap -1 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3378: read oid 10 snap -1 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3378: expect (ObjNum 412 snap 0 seq_num 412) 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3379: read oid 44 snap -1 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3379: expect deleted 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3380: cache_flush oid 490 snap -1 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3381: write oid 123 current snap is 0 2024-09-21T18:09:58.434 INFO:tasks.rados.rados.0.smithi160.stdout:3381: seq_num 1439 ranges {585273=674028,1990059=195214} 2024-09-21T18:09:58.449 INFO:tasks.rados.rados.0.smithi160.stdout:3381: writing smithi16030472-123 from 585273 to 1259301 tid 1 2024-09-21T18:09:58.460 INFO:tasks.rados.rados.0.smithi160.stdout:3381: writing smithi16030472-123 from 1990059 to 2185273 tid 2 2024-09-21T18:09:58.460 INFO:tasks.rados.rados.0.smithi160.stdout:3382: read oid 193 snap -1 2024-09-21T18:09:58.460 INFO:tasks.rados.rados.0.smithi160.stdout:3382: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:09:58.460 INFO:tasks.rados.rados.0.smithi160.stdout:3383: write (excl) oid 309 current snap is 0 2024-09-21T18:09:58.461 INFO:tasks.rados.rados.0.smithi160.stdout:3373: expect (ObjNum 178 snap 0 seq_num 178) 2024-09-21T18:09:58.463 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:09:58.458+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:09:58.534 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2456679 2024-09-21T18:09:58.535 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.535 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:58.535 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:58.535 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.536 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:58.528+0000 7f6a3cca9640 -1 osd.3 117 handle_osd_map injecting map crc failure 2024-09-21T18:09:58.536 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:09:58.528+0000 7fb25ec0a640 -1 osd.11 117 handle_osd_map injecting map crc failure 2024-09-21T18:09:58.547 INFO:tasks.rados.rados.0.smithi160.stdout:3363: finishing write tid 1 to smithi16030472-381 2024-09-21T18:09:58.547 INFO:tasks.rados.rados.0.smithi160.stdout:3363: finishing write tid 2 to smithi16030472-381 2024-09-21T18:09:58.547 INFO:tasks.rados.rados.0.smithi160.stdout:3363: finishing write tid 3 to smithi16030472-381 2024-09-21T18:09:58.547 INFO:tasks.rados.rados.0.smithi160.stdout:3383: seq_num 1440 ranges {602080=749277,2048541=153539} 2024-09-21T18:09:58.549 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:09:58.558 INFO:tasks.rados.rados.0.smithi160.stdout:3383: writing smithi16030472-309 from 602080 to 1351357 tid 1 2024-09-21T18:09:58.564 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:09:58.560+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:09:58.566 INFO:tasks.rados.rados.0.smithi160.stdout:3383: writing smithi16030472-309 from 2048541 to 2202080 tid 2 2024-09-21T18:09:58.566 INFO:tasks.rados.rados.0.smithi160.stdout:3373: done (15 left) 2024-09-21T18:09:58.567 INFO:tasks.rados.rados.0.smithi160.stdout:3379: done (14 left) 2024-09-21T18:09:58.567 INFO:tasks.rados.rados.0.smithi160.stdout:3384: write (excl) oid 330 current snap is 0 2024-09-21T18:09:58.567 INFO:tasks.rados.rados.0.smithi160.stdout:3384: seq_num 1441 ranges {618887=424526,1590670=604158} 2024-09-21T18:09:58.576 INFO:tasks.rados.rados.0.smithi160.stdout:3384: writing smithi16030472-330 from 618887 to 1043413 tid 1 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3384: writing smithi16030472-330 from 1590670 to 2194828 tid 2 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3385: cache_evict oid 404 snap -1 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3376: got (16) Device or resource busy 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3376: done (15 left) 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3386: read oid 288 snap -1 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout:3386: expect (ObjNum 872 snap 0 seq_num 872) 2024-09-21T18:09:58.588 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:58.665 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:58.660+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:58.667 INFO:tasks.rados.rados.0.smithi160.stdout:3363: finishing write tid 4 to smithi16030472-381 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3363: oid 381 updating version 0 to 1015 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3363: oid 381 version 1015 is already newer than 1013 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3363: oid 381 version 1015 is already newer than 1014 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 1015 (ObjNum 1435 snap 0 seq_num 1435) dirty exists 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3363: left oid 381 (ObjNum 1435 snap 0 seq_num 1435) 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3363: done (15 left) 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3387: write oid 244 current snap is 0 2024-09-21T18:09:58.668 INFO:tasks.rados.rados.0.smithi160.stdout:3387: seq_num 1442 ranges {635694=499775,1649152=464169} 2024-09-21T18:09:58.679 INFO:tasks.rados.rados.0.smithi160.stdout:3387: writing smithi16030472-244 from 635694 to 1135469 tid 1 2024-09-21T18:09:58.689 INFO:tasks.rados.rados.0.smithi160.stdout:3387: writing smithi16030472-244 from 1649152 to 2113321 tid 2 2024-09-21T18:09:58.689 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:09:58.690 INFO:tasks.rados.rados.0.smithi160.stdout:3381: finishing write tid 1 to smithi16030472-123 2024-09-21T18:09:58.690 INFO:tasks.rados.rados.0.smithi160.stdout:3381: finishing write tid 2 to smithi16030472-123 2024-09-21T18:09:58.690 INFO:tasks.rados.rados.0.smithi160.stdout:3381: finishing write tid 3 to smithi16030472-123 2024-09-21T18:09:58.690 INFO:tasks.rados.rados.0.smithi160.stdout:3364: finishing copy_from to smithi16030472-449 2024-09-21T18:09:58.691 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 449 v 337 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:09:58.691 INFO:tasks.rados.rados.0.smithi160.stdout:3365: expect (ObjNum 485 snap 0 seq_num 485) 2024-09-21T18:09:58.697 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 766970 2024-09-21T18:09:58.713 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.713 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:58.713 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:58.714 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:58.714 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.720 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:58.728 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:09:58.766 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:58.760+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:58.777 INFO:tasks.rados.rados.0.smithi160.stdout:3364: finishing copy_from racing read to smithi16030472-449 2024-09-21T18:09:58.777 INFO:tasks.rados.rados.0.smithi160.stdout:3377: got (0) Success 2024-09-21T18:09:58.777 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 430 v 0 (ObjNum 989 snap 0 seq_num 989) dirty exists 2024-09-21T18:09:58.777 INFO:tasks.rados.rados.0.smithi160.stdout:3378: expect (ObjNum 412 snap 0 seq_num 412) 2024-09-21T18:09:58.777 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:58.818 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2587827 2024-09-21T18:09:58.828 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:58.853 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.853 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:58.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:58.853 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:09:58.853 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:58.860 INFO:tasks.rados.rados.0.smithi160.stdout:3382: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:09:58.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:09:58.868 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:09:58.865+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:09:58.965 INFO:tasks.rados.rados.0.smithi160.stdout:3371: got (0) Success 2024-09-21T18:09:58.965 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 361 v 0 (ObjNum -1156610528 snap 21993 seq_num 304) clean dne 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3383: finishing write tid 1 to smithi16030472-309 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3381: finishing write tid 4 to smithi16030472-123 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3381: oid 123 updating version 0 to 1041 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3381: oid 123 updating version 1041 to 1042 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3381: oid 123 version 1042 is already newer than 1040 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 1042 (ObjNum 1439 snap 0 seq_num 1439) dirty exists 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3381: left oid 123 (ObjNum 1439 snap 0 seq_num 1439) 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3367: finishing copy_from racing read to smithi16030472-266 2024-09-21T18:09:58.966 INFO:tasks.rados.rados.0.smithi160.stdout:3383: finishing write tid 2 to smithi16030472-309 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3385: got (16) Device or resource busy 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3364: done (15 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3365: done (14 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3371: done (13 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3377: done (12 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3378: done (11 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3381: done (10 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3382: done (9 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3385: done (8 left) 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3388: cache_try_flush oid 443 snap -1 2024-09-21T18:09:58.967 INFO:tasks.rados.rados.0.smithi160.stdout:3380: got (0) Success 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 490 v 0 (ObjNum 1426 snap 0 seq_num 1426) dirty exists 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:3372: finishing copy_from racing read to smithi16030472-374 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:3372: finishing copy_from to smithi16030472-374 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 374 v 758 (ObjNum 1114 snap 0 seq_num 1114) dirty exists 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:3386: expect (ObjNum 872 snap 0 seq_num 872) 2024-09-21T18:09:58.968 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:58.969 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:58.964+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:58.987 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:58.987 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:58.987 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:58.987 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:59.002 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:09:59.007 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:59.050 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:59.071 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:59.065+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: finishing write tid 1 to smithi16030472-293 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3384: finishing write tid 1 to smithi16030472-330 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: finishing write tid 2 to smithi16030472-293 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: finishing write tid 3 to smithi16030472-293 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: finishing write tid 4 to smithi16030472-293 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: oid 293 updating version 0 to 953 2024-09-21T18:09:59.094 INFO:tasks.rados.rados.0.smithi160.stdout:3375: oid 293 updating version 953 to 954 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3375: oid 293 version 954 is already newer than 952 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 954 (ObjNum 1438 snap 0 seq_num 1438) dirty exists 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3375: left oid 293 (ObjNum 1438 snap 0 seq_num 1438) 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3367: finishing copy_from to smithi16030472-266 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 576 (ObjNum 845 snap 0 seq_num 845) dirty exists 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: finishing write tid 3 to smithi16030472-309 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: finishing write tid 4 to smithi16030472-309 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: oid 309 updating version 0 to 1016 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: oid 309 updating version 1016 to 1017 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: oid 309 updating version 1017 to 1019 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 309 v 1019 (ObjNum 1440 snap 0 seq_num 1440) dirty exists 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3383: left oid 309 (ObjNum 1440 snap 0 seq_num 1440) 2024-09-21T18:09:59.095 INFO:tasks.rados.rados.0.smithi160.stdout:3388: got (2) No such file or directory 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3387: finishing write tid 1 to smithi16030472-244 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3387: finishing write tid 2 to smithi16030472-244 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: finishing write tid 2 to smithi16030472-330 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: finishing write tid 3 to smithi16030472-330 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: finishing write tid 4 to smithi16030472-330 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: oid 330 updating version 0 to 1044 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: oid 330 updating version 1044 to 1045 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: oid 330 version 1045 is already newer than 1043 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 330 v 1045 (ObjNum 1441 snap 0 seq_num 1441) dirty exists 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3384: left oid 330 (ObjNum 1441 snap 0 seq_num 1441) 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3367: done (8 left) 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3372: done (7 left) 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3375: done (6 left) 2024-09-21T18:09:59.096 INFO:tasks.rados.rados.0.smithi160.stdout:3380: done (5 left) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3383: done (4 left) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3384: done (3 left) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3386: done (2 left) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3388: done (1 left) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3389: read oid 438 snap -1 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3389: expect (ObjNum 437 snap 0 seq_num 437) 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3390: cache_evict oid 32 snap -1 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3391: cache_evict oid 249 snap -1 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3392: write oid 187 current snap is 0 2024-09-21T18:09:59.097 INFO:tasks.rados.rados.0.smithi160.stdout:3392: seq_num 1443 ranges {652501=575024,1707634=544867} 2024-09-21T18:09:59.113 INFO:tasks.rados.rados.0.smithi160.stdout:3392: writing smithi16030472-187 from 652501 to 1227525 tid 1 2024-09-21T18:09:59.123 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:59.123 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:59.123 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:59.124 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:59.124 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:59.130 INFO:tasks.rados.rados.0.smithi160.stdout:3392: writing smithi16030472-187 from 1707634 to 2252501 tid 2 2024-09-21T18:09:59.130 INFO:tasks.rados.rados.0.smithi160.stdout:3393: cache_evict oid 365 snap -1 2024-09-21T18:09:59.130 INFO:tasks.rados.rados.0.smithi160.stdout:3390: got (16) Device or resource busy 2024-09-21T18:09:59.130 INFO:tasks.rados.rados.0.smithi160.stdout:3390: done (5 left) 2024-09-21T18:09:59.131 INFO:tasks.rados.rados.0.smithi160.stdout:3394: cache_try_flush oid 407 snap -1 2024-09-21T18:09:59.131 INFO:tasks.rados.rados.0.smithi160.stdout:3395: delete oid 90 current snap is 0 2024-09-21T18:09:59.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:09:59.172 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:09:59.168+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:09:59.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:09:59.194 INFO:tasks.rados.rados.0.smithi160.stdout:3387: finishing write tid 3 to smithi16030472-244 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:3387: finishing write tid 4 to smithi16030472-244 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:3387: oid 244 updating version 0 to 988 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:3387: oid 244 updating version 988 to 990 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:3387: oid 244 version 990 is already newer than 989 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 244 v 990 (ObjNum 1442 snap 0 seq_num 1442) dirty exists 2024-09-21T18:09:59.195 INFO:tasks.rados.rados.0.smithi160.stdout:3387: left oid 244 (ObjNum 1442 snap 0 seq_num 1442) 2024-09-21T18:09:59.197 INFO:tasks.rados.rados.0.smithi160.stdout:3391: got (0) Success 2024-09-21T18:09:59.197 INFO:tasks.rados.rados.0.smithi160.stdout:3392: finishing write tid 1 to smithi16030472-187 2024-09-21T18:09:59.197 INFO:tasks.rados.rados.0.smithi160.stdout:3392: finishing write tid 2 to smithi16030472-187 2024-09-21T18:09:59.198 INFO:tasks.rados.rados.0.smithi160.stdout:3392: finishing write tid 3 to smithi16030472-187 2024-09-21T18:09:59.198 INFO:tasks.rados.rados.0.smithi160.stdout:3389: expect (ObjNum 437 snap 0 seq_num 437) 2024-09-21T18:09:59.246 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2954424 2024-09-21T18:09:59.273 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:59.264+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:59.306 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:59.306 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:09:59.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:09:59.306 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 147.707599106, 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:09:59.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:09:59.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:09:59.310 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.277629159, 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:09:59.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.180108784, 2024-09-21T18:09:59.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.17942453400001, 2024-09-21T18:09:59.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:09:59.323 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.17859117899999, 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:09:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:09:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:09:59.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.175872549, 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:09:59.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.080064143, 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 125.077759602, 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:09:59.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:09:59.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 106.100130982, 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.340 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:09:59.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:09:59.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:09:59.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.9989753, 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.344 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:09:59.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:09:59.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:09:59.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.998179638, 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.348 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:09:59.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:09:59.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3802 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'373, mlcod=72'364)", 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.99704381799999, 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25507419100000001, 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306356+0000", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306358+0000", 2024-09-21T18:09:59.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.277e-06 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306364+0000", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.229e-06 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306375+0000", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0797e-05 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306382+0000", 2024-09-21T18:09:59.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8399999999999997e-06 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.551985+0000", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24560364900000001 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.552047+0000", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1398999999999999e-05 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561373+0000", 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0093267609999999994 2024-09-21T18:09:59.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561431+0000", 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7238999999999999e-05 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.97632622499999, 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:09:59.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:09:59.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.966973541, 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.360 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:09:59.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:09:59.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.961598893, 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.364 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:09:59.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:09:59.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:09:59.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 105.95905061800001, 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.368 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:09:59.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:09:59.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:09:59.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 99.731350081000002, 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.372 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:09:59.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:09:59.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:59.376 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 99.730683787999993, 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 99.730591676000003, 2024-09-21T18:09:59.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.384 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:6894 4.2 e96/33 4:60b99da3:::smithi16030472-227:head v 96'728, mlcod=96'726)", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 43.459951232999998, 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.303226251, 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:09:59.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843452+0000", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0429999999999999e-06 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844181+0000", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072923900000000002 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844209+0000", 2024-09-21T18:09:59.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7313000000000001e-05 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944352+0000", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100143339 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944508+0000", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015568600000000001 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944543+0000", 2024-09-21T18:09:59.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5649999999999999e-05 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146570+0000", 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20202621900000001 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146675+0000", 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010576199999999999 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:09:59.388 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:09:59.389 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:59.390 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:09:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: finishing write tid 4 to smithi16030472-187 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: oid 187 updating version 0 to 1046 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: oid 187 updating version 1046 to 1048 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: oid 187 version 1048 is already newer than 1047 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 1048 (ObjNum 1443 snap 0 seq_num 1443) dirty exists 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: left oid 187 (ObjNum 1443 snap 0 seq_num 1443) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3393: got (16) Device or resource busy 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3394: got (0) Success 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 407 v 0 (ObjNum 814 snap 0 seq_num 814) dirty exists 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3387: done (6 left) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3389: done (5 left) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3391: done (4 left) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3392: done (3 left) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3393: done (2 left) 2024-09-21T18:09:59.391 INFO:tasks.rados.rados.0.smithi160.stdout:3394: done (1 left) 2024-09-21T18:09:59.392 INFO:tasks.rados.rados.0.smithi160.stdout:3395: done (0 left) 2024-09-21T18:09:59.392 INFO:tasks.rados.rados.0.smithi160.stdout:3396: delete oid 29 current snap is 0 2024-09-21T18:09:59.392 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:09:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:09:59.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:09:59.393 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:09:59.385+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:09:59.394 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:09:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:09:59.417 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:09:59.417 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:09:59.431 INFO:tasks.rados.rados.0.smithi160.stdout:3396: done (0 left) 2024-09-21T18:09:59.431 INFO:tasks.rados.rados.0.smithi160.stdout:3397: copy_from oid 477 from oid 235 current snap is 0 2024-09-21T18:09:59.432 INFO:tasks.rados.rados.0.smithi160.stdout:3398: read oid 5 snap -1 2024-09-21T18:09:59.432 INFO:tasks.rados.rados.0.smithi160.stdout:3398: expect (ObjNum 1307 snap 0 seq_num 1307) 2024-09-21T18:09:59.432 INFO:tasks.rados.rados.0.smithi160.stdout:3399: read oid 377 snap -1 2024-09-21T18:09:59.432 INFO:tasks.rados.rados.0.smithi160.stdout:3399: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:09:59.432 INFO:tasks.rados.rados.0.smithi160.stdout:3400: delete oid 230 current snap is 0 2024-09-21T18:09:59.438 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:09:59.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:09:59.461 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:09:59.461 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:09:59.482 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:09:59.483 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:09:59.483 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:09:59.490 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:09:59.488+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:09:59.502 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:09:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:09:59.520 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:59.521 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:59.521 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:09:59.521 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:59.524 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:09:59.524 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:09:59.535 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:09:59.541 INFO:tasks.rados.rados.0.smithi160.stdout:3400: done (3 left) 2024-09-21T18:09:59.541 INFO:tasks.rados.rados.0.smithi160.stdout:3401: read oid 253 snap -1 2024-09-21T18:09:59.542 INFO:tasks.rados.rados.0.smithi160.stdout:3401: expect deleted 2024-09-21T18:09:59.542 INFO:tasks.rados.rados.0.smithi160.stdout:3402: read oid 397 snap -1 2024-09-21T18:09:59.542 INFO:tasks.rados.rados.0.smithi160.stdout:3402: expect (ObjNum 1419 snap 0 seq_num 1419) 2024-09-21T18:09:59.542 INFO:tasks.rados.rados.0.smithi160.stdout:3403: delete oid 29 current snap is 0 2024-09-21T18:09:59.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:09:59.546 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:09:59.591 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:09:59.588+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:09:59.638 INFO:tasks.rados.rados.0.smithi160.stdout:3403: done (5 left) 2024-09-21T18:09:59.638 INFO:tasks.rados.rados.0.smithi160.stdout:3404: write oid 360 current snap is 0 2024-09-21T18:09:59.638 INFO:tasks.rados.rados.0.smithi160.stdout:3404: seq_num 1448 ranges {736536=667622,2000044=336492} 2024-09-21T18:09:59.656 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:09:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:09:59.662 INFO:tasks.rados.rados.0.smithi160.stdout:3404: writing smithi16030472-360 from 736536 to 1404158 tid 1 2024-09-21T18:09:59.672 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:09:59.678 INFO:tasks.rados.rados.0.smithi160.stdout:3404: writing smithi16030472-360 from 2000044 to 2336536 tid 2 2024-09-21T18:09:59.678 INFO:tasks.rados.rados.0.smithi160.stdout:3405: cache_try_flush oid 269 snap -1 2024-09-21T18:09:59.678 INFO:tasks.rados.rados.0.smithi160.stdout:3398: expect (ObjNum 1307 snap 0 seq_num 1307) 2024-09-21T18:09:59.687 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:09:59.687 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:09:59.689 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2103217 2024-09-21T18:09:59.692 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:09:59.689+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:09:59.720 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:09:59.720 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:09:59.738 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:59.746 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:09:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:09:59.775 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:09:59.775 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:09:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:09:59.781 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:09:59.781 INFO:tasks.rados.rados.0.smithi160.stdout:3397: finishing copy_from racing read to smithi16030472-477 2024-09-21T18:09:59.782 INFO:tasks.rados.rados.0.smithi160.stdout:3399: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:09:59.793 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:09:59.788+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:09:59.799 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:59.800 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:59.814 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:09:59.816 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:59.850 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:09:59.884 INFO:tasks.rados.rados.0.smithi160.stdout:3402: expect (ObjNum 1419 snap 0 seq_num 1419) 2024-09-21T18:09:59.884 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 154927 2024-09-21T18:09:59.893 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:09:59.889+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:09:59.932 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:09:59.932 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:09:59.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:09:59.969 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2993141 2024-09-21T18:09:59.994 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:09:59.988+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:09:59.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3397: finishing copy_from to smithi16030472-477 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 477 v 631 (ObjNum 849 snap 0 seq_num 849) dirty exists 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3397: done (6 left) 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3398: done (5 left) 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3399: done (4 left) 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3402: done (3 left) 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3406: write oid 79 current snap is 0 2024-09-21T18:10:00.039 INFO:tasks.rados.rados.0.smithi160.stdout:3406: seq_num 1449 ranges {753343=742871,1942173=411170} 2024-09-21T18:10:00.051 INFO:tasks.rados.rados.0.smithi160.stdout:3406: writing smithi16030472-79 from 753343 to 1496214 tid 1 2024-09-21T18:10:00.063 INFO:tasks.rados.rados.0.smithi160.stdout:3406: writing smithi16030472-79 from 1942173 to 2353343 tid 2 2024-09-21T18:10:00.063 INFO:tasks.rados.rados.0.smithi160.stdout:3407: read oid 18 snap -1 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:3407: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:3408: delete oid 271 current snap is 0 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:3405: got (0) Success 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 269 v 0 (ObjNum 33 snap 0 seq_num 3111029040) clean dne 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:3404: finishing write tid 1 to smithi16030472-360 2024-09-21T18:10:00.064 INFO:tasks.rados.rados.0.smithi160.stdout:3404: finishing write tid 2 to smithi16030472-360 2024-09-21T18:10:00.074 INFO:tasks.rados.rados.0.smithi160.stdout:3404: finishing write tid 3 to smithi16030472-360 2024-09-21T18:10:00.074 INFO:tasks.rados.rados.0.smithi160.stdout:3404: finishing write tid 4 to smithi16030472-360 2024-09-21T18:10:00.074 INFO:tasks.rados.rados.0.smithi160.stdout:3404: oid 360 updating version 0 to 992 2024-09-21T18:10:00.074 INFO:tasks.rados.rados.0.smithi160.stdout:3404: oid 360 version 992 is already newer than 991 2024-09-21T18:10:00.074 INFO:tasks.rados.rados.0.smithi160.stdout:3404: oid 360 updating version 992 to 993 2024-09-21T18:10:00.075 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 360 v 993 (ObjNum 1448 snap 0 seq_num 1448) dirty exists 2024-09-21T18:10:00.075 INFO:tasks.rados.rados.0.smithi160.stdout:3404: left oid 360 (ObjNum 1448 snap 0 seq_num 1448) 2024-09-21T18:10:00.082 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.082 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.095 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:00.089+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:00.096 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:00.163 INFO:tasks.rados.rados.0.smithi160.stdout:3406: finishing write tid 1 to smithi16030472-79 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3401: done (5 left) 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3404: done (4 left) 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3405: done (3 left) 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3408: done (2 left) 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3409: copy_from oid 40 from oid 452 current snap is 0 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3410: write (excl) oid 170 current snap is 0 2024-09-21T18:10:00.169 INFO:tasks.rados.rados.0.smithi160.stdout:3410: seq_num 1451 ranges {786957=493369,2059137=327820} 2024-09-21T18:10:00.184 INFO:tasks.rados.rados.0.smithi160.stdout:3410: writing smithi16030472-170 from 786957 to 1280326 tid 1 2024-09-21T18:10:00.195 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:00.193+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:00.201 INFO:tasks.rados.rados.0.smithi160.stdout:3410: writing smithi16030472-170 from 2059137 to 2386957 tid 2 2024-09-21T18:10:00.201 INFO:tasks.rados.rados.0.smithi160.stdout:3411: cache_evict oid 424 snap -1 2024-09-21T18:10:00.201 INFO:tasks.rados.rados.0.smithi160.stdout:3412: delete oid 400 current snap is 0 2024-09-21T18:10:00.201 INFO:tasks.rados.rados.0.smithi160.stdout:3409: finishing copy_from racing read to smithi16030472-40 2024-09-21T18:10:00.213 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.213 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.227 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:00.267 INFO:tasks.rados.rados.0.smithi160.stdout:3407: expect (ObjNum 235 snap 0 seq_num 235) 2024-09-21T18:10:00.271 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1915090 2024-09-21T18:10:00.297 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:00.289+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:00.338 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1418273 2024-09-21T18:10:00.345 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.345 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.357 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:00.365 INFO:tasks.rados.rados.0.smithi160.stdout:3411: got (0) Success 2024-09-21T18:10:00.366 INFO:tasks.rados.rados.0.smithi160.stdout:3406: finishing write tid 2 to smithi16030472-79 2024-09-21T18:10:00.395 INFO:tasks.rados.rados.0.smithi160.stdout:3410: finishing write tid 1 to smithi16030472-170 2024-09-21T18:10:00.395 INFO:tasks.rados.rados.0.smithi160.stdout:3410: finishing write tid 2 to smithi16030472-170 2024-09-21T18:10:00.398 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:00.392+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:00.401 INFO:tasks.rados.rados.0.smithi160.stdout:3406: finishing write tid 3 to smithi16030472-79 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:3406: finishing write tid 4 to smithi16030472-79 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:3406: oid 79 updating version 0 to 956 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:3406: oid 79 version 956 is already newer than 955 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:3406: oid 79 updating version 956 to 957 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 79 v 957 (ObjNum 1449 snap 0 seq_num 1449) dirty exists 2024-09-21T18:10:00.402 INFO:tasks.rados.rados.0.smithi160.stdout:3406: left oid 79 (ObjNum 1449 snap 0 seq_num 1449) 2024-09-21T18:10:00.403 INFO:tasks.rados.rados.0.smithi160.stdout:3409: finishing copy_from to smithi16030472-40 2024-09-21T18:10:00.403 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 40 v 811 (ObjNum 1123 snap 0 seq_num 1123) dirty exists 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3406: done (5 left) 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3407: done (4 left) 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3409: done (3 left) 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3411: done (2 left) 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3412: done (1 left) 2024-09-21T18:10:00.405 INFO:tasks.rados.rados.0.smithi160.stdout:3413: write (excl) oid 118 current snap is 0 2024-09-21T18:10:00.406 INFO:tasks.rados.rados.0.smithi160.stdout:3413: seq_num 1453 ranges {420571=760220,1776101=626172} 2024-09-21T18:10:00.411 INFO:tasks.rados.rados.0.smithi160.stdout:3413: writing smithi16030472-118 from 420571 to 1180791 tid 1 2024-09-21T18:10:00.418 INFO:tasks.rados.rados.0.smithi160.stdout:3413: writing smithi16030472-118 from 1776101 to 2402273 tid 2 2024-09-21T18:10:00.418 INFO:tasks.rados.rados.0.smithi160.stdout:3414: read oid 139 snap -1 2024-09-21T18:10:00.418 INFO:tasks.rados.rados.0.smithi160.stdout:3414: expect (ObjNum 978 snap 0 seq_num 978) 2024-09-21T18:10:00.418 INFO:tasks.rados.rados.0.smithi160.stdout:3415: delete oid 345 current snap is 0 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3415: done (3 left) 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3416: copy_from oid 481 from oid 227 current snap is 0 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3417: read oid 356 snap -1 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3417: expect (ObjNum 714 snap 0 seq_num 714) 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3418: cache_evict oid 391 snap -1 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3419: copy_from oid 317 from oid 97 current snap is 0 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3420: read oid 475 snap -1 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3420: expect (ObjNum 1388 snap 0 seq_num 1388) 2024-09-21T18:10:00.423 INFO:tasks.rados.rados.0.smithi160.stdout:3421: delete oid 109 current snap is 0 2024-09-21T18:10:00.480 INFO:tasks.rados.rados.0.smithi160.stdout:3418: got (0) Success 2024-09-21T18:10:00.494 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.494 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.498 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:00.493+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:00.507 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:00.600 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:00.596+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:00.676 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.676 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.690 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:00.700 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:00.693+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:00.801 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:00.797+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:00.812 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.813 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.827 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:00.902 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:00.896+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:00.927 INFO:tasks.rados.rados.0.smithi160.stdout:3416: finishing copy_from racing read to smithi16030472-481 2024-09-21T18:10:00.927 INFO:tasks.rados.rados.0.smithi160.stdout:3413: finishing write tid 1 to smithi16030472-118 2024-09-21T18:10:00.927 INFO:tasks.rados.rados.0.smithi160.stdout:3413: finishing write tid 2 to smithi16030472-118 2024-09-21T18:10:00.927 INFO:tasks.rados.rados.0.smithi160.stdout:3413: finishing write tid 3 to smithi16030472-118 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3413: finishing write tid 4 to smithi16030472-118 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3413: oid 118 updating version 0 to 995 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3413: oid 118 updating version 995 to 996 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3413: oid 118 version 996 is already newer than 994 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 118 v 996 (ObjNum 1453 snap 0 seq_num 1453) dirty exists 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3413: left oid 118 (ObjNum 1453 snap 0 seq_num 1453) 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: finishing write tid 3 to smithi16030472-170 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: finishing write tid 4 to smithi16030472-170 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: oid 170 updating version 0 to 956 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: oid 170 updating version 956 to 957 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: oid 170 version 957 is already newer than 955 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 170 v 957 (ObjNum 1451 snap 0 seq_num 1451) dirty exists 2024-09-21T18:10:00.928 INFO:tasks.rados.rados.0.smithi160.stdout:3410: left oid 170 (ObjNum 1451 snap 0 seq_num 1451) 2024-09-21T18:10:00.929 INFO:tasks.rados.rados.0.smithi160.stdout:3417: expect (ObjNum 714 snap 0 seq_num 714) 2024-09-21T18:10:00.948 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:00.948 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:00.961 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:00.988 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:10:00.989 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:00.984+0000 7f0b6c216640 -1 osd.9 118 handle_osd_map injecting map crc failure 2024-09-21T18:10:00.993 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.003 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:01.000+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:01.028 INFO:tasks.rados.rados.0.smithi160.stdout:3416: finishing copy_from to smithi16030472-481 2024-09-21T18:10:01.028 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 837 (ObjNum 1152 snap 0 seq_num 1152) dirty exists 2024-09-21T18:10:01.028 INFO:tasks.rados.rados.0.smithi160.stdout:3414: expect (ObjNum 978 snap 0 seq_num 978) 2024-09-21T18:10:01.028 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.104 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1477262 2024-09-21T18:10:01.104 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:01.101+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:01.109 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.109 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:10:01.132 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.187 INFO:tasks.rados.rados.0.smithi160.stdout:3420: expect (ObjNum 1388 snap 0 seq_num 1388) 2024-09-21T18:10:01.205 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:01.201+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:01.258 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.258 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.271 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3208707 2024-09-21T18:10:01.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:10:01.308 INFO:tasks.rados.rados.0.smithi160.stdout:3419: finishing copy_from to smithi16030472-317 2024-09-21T18:10:01.308 INFO:tasks.rados.rados.0.smithi160.stdout:3419: got expected ENOENT (src dne) 2024-09-21T18:10:01.308 INFO:tasks.rados.rados.0.smithi160.stdout:3419: finishing copy_from racing read to smithi16030472-317 2024-09-21T18:10:01.308 INFO:tasks.rados.rados.0.smithi160.stdout:3410: done (8 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3413: done (7 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3414: done (6 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3416: done (5 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3417: done (4 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3418: done (3 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3419: done (2 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3420: done (1 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3421: done (0 left) 2024-09-21T18:10:01.309 INFO:tasks.rados.rados.0.smithi160.stdout:3422: delete oid 158 current snap is 0 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3422: done (0 left) 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3423: cache_flush oid 364 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3424: read oid 410 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3424: expect deleted 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3425: cache_evict oid 500 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3426: cache_try_flush oid 110 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3427: read oid 56 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3427: expect (ObjNum 939 snap 0 seq_num 939) 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3428: cache_flush oid 262 snap -1 2024-09-21T18:10:01.315 INFO:tasks.rados.rados.0.smithi160.stdout:3429: cache_try_flush oid 11 snap -1 2024-09-21T18:10:01.316 INFO:tasks.rados.rados.0.smithi160.stdout:3430: write (excl) oid 361 current snap is 0 2024-09-21T18:10:01.316 INFO:tasks.rados.rados.0.smithi160.stdout:3430: seq_num 1457 ranges {487799=661216,1893676=594123} 2024-09-21T18:10:01.316 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:01.313+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:01.336 INFO:tasks.rados.rados.0.smithi160.stdout:3430: writing smithi16030472-361 from 487799 to 1149015 tid 1 2024-09-21T18:10:01.358 INFO:tasks.rados.rados.0.smithi160.stdout:3430: writing smithi16030472-361 from 1893676 to 2487799 tid 2 2024-09-21T18:10:01.358 INFO:tasks.rados.rados.0.smithi160.stdout:3431: cache_try_flush oid 404 snap -1 2024-09-21T18:10:01.358 INFO:tasks.rados.rados.0.smithi160.stdout:3432: cache_flush oid 63 snap -1 2024-09-21T18:10:01.359 INFO:tasks.rados.rados.0.smithi160.stdout:3427: expect (ObjNum 939 snap 0 seq_num 939) 2024-09-21T18:10:01.407 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1803446 2024-09-21T18:10:01.407 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:01.400+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:01.429 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:01.425+0000 7f7283507640 -1 osd.14 118 handle_osd_map injecting map crc failure 2024-09-21T18:10:01.448 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.448 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.458 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:10:01.508 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:01.501+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:3429: got (0) Success 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 11 v 0 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:3425: got (16) Device or resource busy 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:3426: got (0) Success 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 110 v 0 (ObjNum 109 snap 0 seq_num 109) dirty exists 2024-09-21T18:10:01.567 INFO:tasks.rados.rados.0.smithi160.stdout:3425: done (9 left) 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3426: done (8 left) 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3427: done (7 left) 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3429: done (6 left) 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3433: cache_try_flush oid 6 snap -1 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: finishing write tid 1 to smithi16030472-361 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3423: got (0) Success 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 0 (ObjNum 1407 snap 0 seq_num 1407) dirty exists 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: finishing write tid 2 to smithi16030472-361 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: finishing write tid 3 to smithi16030472-361 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: finishing write tid 4 to smithi16030472-361 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: oid 361 updating version 0 to 998 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: oid 361 updating version 998 to 999 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: oid 361 version 999 is already newer than 997 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 361 v 999 (ObjNum 1457 snap 0 seq_num 1457) dirty exists 2024-09-21T18:10:01.568 INFO:tasks.rados.rados.0.smithi160.stdout:3430: left oid 361 (ObjNum 1457 snap 0 seq_num 1457) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3428: got (0) Success 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 262 v 0 (ObjNum 1351 snap 0 seq_num 1351) dirty exists 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3431: got (0) Success 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 404 v 0 (ObjNum 1380 snap 0 seq_num 1380) dirty exists 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3423: done (6 left) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3428: done (5 left) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3430: done (4 left) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3431: done (3 left) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3434: copy_from oid 413 from oid 403 current snap is 0 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3435: cache_flush oid 151 snap -1 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3436: copy_from oid 33 from oid 219 current snap is 0 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3437: read oid 200 snap -1 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3437: expect (ObjNum 1277 snap 0 seq_num 1277) 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3438: cache_try_flush oid 316 snap -1 2024-09-21T18:10:01.569 INFO:tasks.rados.rados.0.smithi160.stdout:3439: read oid 64 snap -1 2024-09-21T18:10:01.570 INFO:tasks.rados.rados.0.smithi160.stdout:3439: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:10:01.570 INFO:tasks.rados.rados.0.smithi160.stdout:3440: delete oid 321 current snap is 0 2024-09-21T18:10:01.570 INFO:tasks.rados.rados.0.smithi160.stdout:3438: got (0) Success 2024-09-21T18:10:01.570 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 316 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:10:01.591 INFO:tasks.rados.rados.0.smithi160.stdout:3433: got (0) Success 2024-09-21T18:10:01.591 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 0 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:10:01.591 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.592 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:10:01.608 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:01.601+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:01.668 INFO:tasks.rados.rados.0.smithi160.stdout:3435: got (0) Success 2024-09-21T18:10:01.669 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 151 v 0 (ObjNum 753 snap 0 seq_num 753) dirty exists 2024-09-21T18:10:01.677 INFO:tasks.rados.rados.0.smithi160.stdout:3439: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-21T18:10:01.709 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:01.704+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:01.723 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.734 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.734 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:10:01.764 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:01.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:01.795 INFO:tasks.rados.rados.0.smithi160.stdout:3437: expect (ObjNum 1277 snap 0 seq_num 1277) 2024-09-21T18:10:01.839 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:01.837+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:01.878 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:01.878 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:01.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:10:01.910 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:01.905+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3436: finishing copy_from to smithi16030472-33 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3436: got expected ENOENT (src dne) 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3436: finishing copy_from racing read to smithi16030472-33 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3434: finishing copy_from racing read to smithi16030472-413 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3432: got (0) Success 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 63 v 0 (ObjNum 1312 snap 0 seq_num 1312) dirty exists 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:3434: finishing copy_from to smithi16030472-413 2024-09-21T18:10:01.931 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 915 (ObjNum 1293 snap 0 seq_num 1293) dirty exists 2024-09-21T18:10:01.944 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:01.966 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:01.966 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:01.977 INFO:tasks.rados.rados.0.smithi160.stdout:3424: done (9 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3432: done (8 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3433: done (7 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3434: done (6 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3435: done (5 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3436: done (4 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3437: done (3 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3438: done (2 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3439: done (1 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3440: done (0 left) 2024-09-21T18:10:01.978 INFO:tasks.rados.rados.0.smithi160.stdout:3441: delete oid 420 current snap is 0 2024-09-21T18:10:01.986 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:01.987 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:02.009 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:02.009 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:02.010 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:02.005+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:02.026 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:02.027 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:02.048 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 168.85044838499999, 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:10:02.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:10:02.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.069 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.306510836, 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.070 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:10:02.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:10:02.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.14558737199999, 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.075 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:10:02.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.144912777, 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.080 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:10:02.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.14488623899999, 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.135052497, 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.088 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:02.092 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.122640709, 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:02.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.095 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:10:02.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:10:02.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:10:02.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.11375063899999, 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.099 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:10:02.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:10:02.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.112938622, 2024-09-21T18:10:02.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:10:02.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:10:02.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.11283040999999, 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.112 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:10:02.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:10:02.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:10:02.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.117 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.012559264, 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.118 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:10:02.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:10:02.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.011663005, 2024-09-21T18:10:02.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:10:02.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:10:02.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.011181392, 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.129 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:10:02.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:10:02.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.01072896700001, 2024-09-21T18:10:02.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:10:02.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:10:02.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 151.010678641, 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.138 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:10:02.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:10:02.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 150.88494586100001, 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.142 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:10:02.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 150.88066780400001, 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.148 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:02.149 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.150 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:02.154 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 150.88029756399999, 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:10:02.156 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:10:02.157 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.158 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.159 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.160 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 150.88022783400001, 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:10:02.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.162 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.163 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.165 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 150.28865084399999, 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:02.166 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:02.167 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.168 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.170 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:02.171 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.173 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:02.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:02.173 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:02.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:10:02.175 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:02.175 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3441: done (0 left) 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3442: read oid 274 snap -1 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3442: expect (ObjNum 1387 snap 0 seq_num 1387) 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3443: cache_try_flush oid 342 snap -1 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3444: cache_try_flush oid 347 snap -1 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3445: delete oid 453 current snap is 0 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3445: done (3 left) 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3446: read oid 210 snap -1 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3446: expect (ObjNum 1037 snap 0 seq_num 1037) 2024-09-21T18:10:02.176 INFO:tasks.rados.rados.0.smithi160.stdout:3447: write oid 216 current snap is 0 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3447: seq_num 1461 ranges {555027=678565,1727604=640848} 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3447: writing smithi16030472-216 from 555027 to 1233592 tid 1 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3447: writing smithi16030472-216 from 1727604 to 2368452 tid 2 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3448: write oid 158 current snap is 0 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3448: seq_num 1462 ranges {571834=753814,1786086=784506} 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3448: writing smithi16030472-158 from 571834 to 1325648 tid 1 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3448: writing smithi16030472-158 from 1786086 to 2570592 tid 2 2024-09-21T18:10:02.177 INFO:tasks.rados.rados.0.smithi160.stdout:3449: delete oid 258 current snap is 0 2024-09-21T18:10:02.177 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:02.177 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:02.178 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:02.178 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:02.178 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:02.179 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:02.179 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:02.168+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:02.192 INFO:tasks.rados.rados.0.smithi160.stdout:3442: expect (ObjNum 1387 snap 0 seq_num 1387) 2024-09-21T18:10:02.193 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 401101 2024-09-21T18:10:02.199 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:02.221 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:02.222 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:02.245 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:02.245 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:02.250 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:02.274 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:02.269+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:02.300 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.300 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:02.300 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:02.300 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:10:02.375 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:02.373+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:02.433 INFO:tasks.rados.rados.0.smithi160.stdout:3446: expect (ObjNum 1037 snap 0 seq_num 1037) 2024-09-21T18:10:02.433 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 19482 2024-09-21T18:10:02.443 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:02.443 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:10:02.470 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:02.476 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:02.469+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:02.543 INFO:tasks.rados.rados.0.smithi160.stdout:3448: finishing write tid 1 to smithi16030472-158 2024-09-21T18:10:02.543 INFO:tasks.rados.rados.0.smithi160.stdout:3448: finishing write tid 2 to smithi16030472-158 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: finishing write tid 3 to smithi16030472-158 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: finishing write tid 1 to smithi16030472-216 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: finishing write tid 2 to smithi16030472-216 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: finishing write tid 3 to smithi16030472-216 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: finishing write tid 4 to smithi16030472-158 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: oid 158 updating version 0 to 1033 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: oid 158 updating version 1033 to 1034 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: oid 158 version 1034 is already newer than 1032 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 158 v 1034 (ObjNum 1462 snap 0 seq_num 1462) dirty exists 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3448: left oid 158 (ObjNum 1462 snap 0 seq_num 1462) 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: finishing write tid 4 to smithi16030472-216 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: oid 216 updating version 0 to 960 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: oid 216 version 960 is already newer than 959 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:3447: oid 216 updating version 960 to 961 2024-09-21T18:10:02.544 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 216 v 961 (ObjNum 1461 snap 0 seq_num 1461) dirty exists 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3447: left oid 216 (ObjNum 1461 snap 0 seq_num 1461) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3442: done (6 left) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3446: done (5 left) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3447: done (4 left) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3448: done (3 left) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3449: done (2 left) 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3450: write oid 193 current snap is 0 2024-09-21T18:10:02.545 INFO:tasks.rados.rados.0.smithi160.stdout:3450: seq_num 1464 ranges {605448=504312,1903050=702398} 2024-09-21T18:10:02.561 INFO:tasks.rados.rados.0.smithi160.stdout:3450: writing smithi16030472-193 from 605448 to 1109760 tid 1 2024-09-21T18:10:02.577 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:02.572+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:02.585 INFO:tasks.rados.rados.0.smithi160.stdout:3450: writing smithi16030472-193 from 1903050 to 2605448 tid 2 2024-09-21T18:10:02.585 INFO:tasks.rados.rados.0.smithi160.stdout:3451: write (excl) oid 96 current snap is 0 2024-09-21T18:10:02.586 INFO:tasks.rados.rados.0.smithi160.stdout:3451: seq_num 1465 ranges {622255=579561,1845179=531833} 2024-09-21T18:10:02.601 INFO:tasks.rados.rados.0.smithi160.stdout:3451: writing smithi16030472-96 from 622255 to 1201816 tid 1 2024-09-21T18:10:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:3451: writing smithi16030472-96 from 1845179 to 2377012 tid 2 2024-09-21T18:10:02.614 INFO:tasks.rados.rados.0.smithi160.stdout:3452: cache_flush oid 450 snap -1 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:3450: finishing write tid 1 to smithi16030472-193 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:3444: got (0) Success 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 347 v 0 (ObjNum 1246 snap 0 seq_num 1246) dirty exists 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:3444: done (4 left) 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:3453: delete oid 163 current snap is 0 2024-09-21T18:10:02.615 INFO:tasks.rados.rados.0.smithi160.stdout:3452: got (2) No such file or directory 2024-09-21T18:10:02.635 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.635 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:02.635 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:02.635 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:10:02.635 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:10:02.678 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:02.673+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:02.684 INFO:tasks.rados.rados.0.smithi160.stdout:3443: got (0) Success 2024-09-21T18:10:02.685 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 342 v 0 (ObjNum 1339 snap 0 seq_num 1339) dirty exists 2024-09-21T18:10:02.778 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.778 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:02.778 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:02.778 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.778 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:02.777+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:02.792 INFO:tasks.rados.rados.0.smithi160.stdout:3443: done (4 left) 2024-09-21T18:10:02.792 INFO:tasks.rados.rados.0.smithi160.stdout:3452: done (3 left) 2024-09-21T18:10:02.792 INFO:tasks.rados.rados.0.smithi160.stdout:3453: done (2 left) 2024-09-21T18:10:02.792 INFO:tasks.rados.rados.0.smithi160.stdout:3454: write oid 324 current snap is 0 2024-09-21T18:10:02.792 INFO:tasks.rados.rados.0.smithi160.stdout:3454: seq_num 1467 ranges {655869=730059,1962143=535502} 2024-09-21T18:10:02.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:10:02.807 INFO:tasks.rados.rados.0.smithi160.stdout:3454: writing smithi16030472-324 from 655869 to 1385928 tid 1 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3454: writing smithi16030472-324 from 1962143 to 2497645 tid 2 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3455: read oid 133 snap -1 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3455: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3456: cache_try_flush oid 339 snap -1 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3450: finishing write tid 2 to smithi16030472-193 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3450: finishing write tid 3 to smithi16030472-193 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3457: write oid 208 current snap is 0 2024-09-21T18:10:02.821 INFO:tasks.rados.rados.0.smithi160.stdout:3457: seq_num 1468 ranges {672676=521661,1736978=795513} 2024-09-21T18:10:02.834 INFO:tasks.rados.rados.0.smithi160.stdout:3457: writing smithi16030472-208 from 672676 to 1194337 tid 1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3457: writing smithi16030472-208 from 1736978 to 2532491 tid 2 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3458: read oid 246 snap -1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3458: expect deleted 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3459: cache_flush oid 159 snap -1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3460: cache_evict oid 431 snap -1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3461: cache_evict oid 279 snap -1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3462: cache_evict oid 390 snap -1 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3463: write oid 43 current snap is 0 2024-09-21T18:10:02.851 INFO:tasks.rados.rados.0.smithi160.stdout:3463: seq_num 1469 ranges {689483=596910,2079107=539171} 2024-09-21T18:10:02.863 INFO:tasks.rados.rados.0.smithi160.stdout:3463: writing smithi16030472-43 from 689483 to 1286393 tid 1 2024-09-21T18:10:02.874 INFO:tasks.rados.rados.0.smithi160.stdout:3463: writing smithi16030472-43 from 2079107 to 2618278 tid 2 2024-09-21T18:10:02.874 INFO:tasks.rados.rados.0.smithi160.stdout:3464: delete oid 49 current snap is 0 2024-09-21T18:10:02.879 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:02.877+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:02.896 INFO:tasks.rados.rados.0.smithi160.stdout:3451: finishing write tid 1 to smithi16030472-96 2024-09-21T18:10:02.912 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:02.912 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:02.912 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:02.913 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:02.913 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:3450: finishing write tid 4 to smithi16030472-193 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:3450: oid 193 updating version 0 to 964 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:3450: oid 193 version 964 is already newer than 963 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:3450: oid 193 version 964 is already newer than 962 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 193 v 964 (ObjNum 1464 snap 0 seq_num 1464) dirty exists 2024-09-21T18:10:02.925 INFO:tasks.rados.rados.0.smithi160.stdout:3450: left oid 193 (ObjNum 1464 snap 0 seq_num 1464) 2024-09-21T18:10:02.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:10:02.928 INFO:tasks.rados.rados.0.smithi160.stdout:3460: got (16) Device or resource busy 2024-09-21T18:10:02.931 INFO:tasks.rados.rados.0.smithi160.stdout:3455: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:10:02.980 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:02.973+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:03.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:10:03.076 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:03.081 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:03.077+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 197.20537314699999, 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:03.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:03.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:03.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 196.119980758, 2024-09-21T18:10:03.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:03.111 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:03.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:10:03.129 INFO:tasks.rados.rados.0.smithi160.stdout:3456: got (0) Success 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 339 v 0 (ObjNum 491 snap 0 seq_num 491) dirty exists 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: finishing write tid 2 to smithi16030472-96 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: finishing write tid 3 to smithi16030472-96 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: finishing write tid 4 to smithi16030472-96 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: oid 96 updating version 0 to 1049 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: oid 96 updating version 1049 to 1051 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: oid 96 version 1051 is already newer than 1050 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 96 v 1051 (ObjNum 1465 snap 0 seq_num 1465) dirty exists 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3451: left oid 96 (ObjNum 1465 snap 0 seq_num 1465) 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3461: got (2) No such file or directory 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3457: finishing write tid 1 to smithi16030472-208 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3462: got (2) No such file or directory 2024-09-21T18:10:03.130 INFO:tasks.rados.rados.0.smithi160.stdout:3454: finishing write tid 1 to smithi16030472-324 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3450: done (12 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3451: done (11 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3455: done (10 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3456: done (9 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3458: done (8 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3460: done (7 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3461: done (6 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3462: done (5 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3464: done (4 left) 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3465: write (excl) oid 448 current snap is 0 2024-09-21T18:10:03.131 INFO:tasks.rados.rados.0.smithi160.stdout:3465: seq_num 1471 ranges {723097=747408,2196071=527026} 2024-09-21T18:10:03.143 INFO:tasks.rados.rados.0.smithi160.stdout:3465: writing smithi16030472-448 from 723097 to 1470505 tid 1 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3465: writing smithi16030472-448 from 2196071 to 2723097 tid 2 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3466: cache_try_flush oid 216 snap -1 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3459: got (0) Success 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 159 v 0 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3459: done (5 left) 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3467: read oid 183 snap -1 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3467: expect (ObjNum 1097 snap 0 seq_num 1097) 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3468: copy_from oid 263 from oid 61 current snap is 0 2024-09-21T18:10:03.157 INFO:tasks.rados.rados.0.smithi160.stdout:3469: copy_from oid 42 from oid 303 current snap is 0 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3470: cache_flush oid 31 snap -1 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3471: cache_evict oid 27 snap -1 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3472: copy_from oid 109 from oid 480 current snap is 0 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3473: read oid 360 snap -1 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3473: expect (ObjNum 1448 snap 0 seq_num 1448) 2024-09-21T18:10:03.158 INFO:tasks.rados.rados.0.smithi160.stdout:3474: delete oid 64 current snap is 0 2024-09-21T18:10:03.181 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:03.176+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:03.253 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:03.253 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:03.253 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:03.253 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:03.258 INFO:tasks.rados.rados.0.smithi160.stdout:3457: finishing write tid 2 to smithi16030472-208 2024-09-21T18:10:03.259 INFO:tasks.rados.rados.0.smithi160.stdout:3465: finishing write tid 1 to smithi16030472-448 2024-09-21T18:10:03.266 INFO:tasks.rados.rados.0.smithi160.stdout:3473: expect (ObjNum 1448 snap 0 seq_num 1448) 2024-09-21T18:10:03.267 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:10:03.282 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:03.277+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:03.314 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1832678 2024-09-21T18:10:03.325 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 103246 2024-09-21T18:10:03.360 INFO:tasks.rados.rados.0.smithi160.stdout:3467: expect (ObjNum 1097 snap 0 seq_num 1097) 2024-09-21T18:10:03.383 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:03.381+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:03.425 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:03.437 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:03.437 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:03.437 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:03.437 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:03.437 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:03.451 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3468: finishing copy_from racing read to smithi16030472-263 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: finishing write tid 3 to smithi16030472-208 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: finishing write tid 4 to smithi16030472-208 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: oid 208 updating version 0 to 966 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: oid 208 version 966 is already newer than 965 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: oid 208 updating version 966 to 967 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 208 v 967 (ObjNum 1468 snap 0 seq_num 1468) dirty exists 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3457: left oid 208 (ObjNum 1468 snap 0 seq_num 1468) 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3465: finishing write tid 2 to smithi16030472-448 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3465: finishing write tid 3 to smithi16030472-448 2024-09-21T18:10:03.463 INFO:tasks.rados.rados.0.smithi160.stdout:3465: finishing write tid 4 to smithi16030472-448 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3465: oid 448 updating version 0 to 1004 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3465: oid 448 version 1004 is already newer than 1003 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3465: oid 448 version 1004 is already newer than 1002 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 1004 (ObjNum 1471 snap 0 seq_num 1471) dirty exists 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3465: left oid 448 (ObjNum 1471 snap 0 seq_num 1471) 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3471: got (16) Device or resource busy 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3469: finishing copy_from racing read to smithi16030472-42 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3469: finishing copy_from to smithi16030472-42 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 1002 (ObjNum 1423 snap 0 seq_num 1423) dirty exists 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: finishing write tid 1 to smithi16030472-43 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: finishing write tid 2 to smithi16030472-43 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: finishing write tid 3 to smithi16030472-43 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: finishing write tid 4 to smithi16030472-43 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: oid 43 updating version 0 to 969 2024-09-21T18:10:03.464 INFO:tasks.rados.rados.0.smithi160.stdout:3463: oid 43 updating version 969 to 970 2024-09-21T18:10:03.465 INFO:tasks.rados.rados.0.smithi160.stdout:3463: oid 43 version 970 is already newer than 968 2024-09-21T18:10:03.465 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 43 v 970 (ObjNum 1469 snap 0 seq_num 1469) dirty exists 2024-09-21T18:10:03.465 INFO:tasks.rados.rados.0.smithi160.stdout:3463: left oid 43 (ObjNum 1469 snap 0 seq_num 1469) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3457: done (12 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3463: done (11 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3465: done (10 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3467: done (9 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3469: done (8 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3471: done (7 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3473: done (6 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3474: done (5 left) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3475: copy_from oid 438 from oid 73 current snap is 0 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3476: cache_evict oid 143 snap -1 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3477: read oid 414 snap -1 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3477: expect (ObjNum 1053 snap 0 seq_num 1053) 2024-09-21T18:10:03.466 INFO:tasks.rados.rados.0.smithi160.stdout:3478: copy_from oid 179 from oid 335 current snap is 0 2024-09-21T18:10:03.467 INFO:tasks.rados.rados.0.smithi160.stdout:3479: delete oid 63 current snap is 0 2024-09-21T18:10:03.478 INFO:tasks.rados.rados.0.smithi160.stdout:3472: finishing copy_from racing read to smithi16030472-109 2024-09-21T18:10:03.490 INFO:tasks.rados.rados.0.smithi160.stdout:3454: finishing write tid 2 to smithi16030472-324 2024-09-21T18:10:03.499 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:03.496+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:03.565 INFO:tasks.rados.rados.0.smithi160.stdout:3475: finishing copy_from to smithi16030472-438 2024-09-21T18:10:03.566 INFO:tasks.rados.rados.0.smithi160.stdout:3475: got expected ENOENT (src dne) 2024-09-21T18:10:03.566 INFO:tasks.rados.rados.0.smithi160.stdout:3475: finishing copy_from racing read to smithi16030472-438 2024-09-21T18:10:03.571 INFO:tasks.rados.rados.0.smithi160.stdout:3477: expect (ObjNum 1053 snap 0 seq_num 1053) 2024-09-21T18:10:03.585 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:03.577+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54438471200001, 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.586 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54414508799999, 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:10:03.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:10:03.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54410968100001, 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.597 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.601 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54407650799999, 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:10:03.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:10:03.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:10:03.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54404694799999, 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:10:03.606 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.610 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54401729200001, 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:03.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:10:03.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:10:03.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:10:03.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54353103599999, 2024-09-21T18:10:03.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:10:03.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.620 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.54349744300001, 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:10:03.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.44327975100001, 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:03.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:10:03.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:10:03.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:03.633 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.44268084699999, 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.634 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:03.635 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:10:03.636 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.637 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:10:03.638 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.639 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:10:03.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.643 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.644 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.44196622999999, 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.645 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.646 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.647 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:10:03.648 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.649 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.650 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.651 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:10:03.652 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.653 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.654 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.655 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.432713498, 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:03.656 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:10:03.657 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:10:03.658 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.659 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:10:03.660 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:10:03.661 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:10:03.662 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.43244879700001, 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.663 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:03.664 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.665 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.666 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.667 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:10:03.668 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.669 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.670 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:10:03.671 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.672 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.432354108, 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.673 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.674 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.675 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:10:03.676 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.677 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.678 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:10:03.679 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:10:03.680 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.681 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.682 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.43231281800001, 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:10:03.683 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.684 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.685 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.686 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.687 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:10:03.688 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.689 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.690 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.691 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:10:03.692 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.693 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.42104291499999, 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.694 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.695 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.696 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.696 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:03.696 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.696 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.696 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.697 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:10:03.698 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.699 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.700 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.701 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.411926682, 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.702 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:10:03.703 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:10:03.704 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:10:03.705 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:10:03.706 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.410863315, 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:10:03.707 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.708 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.709 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.710 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.711 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 152.410823005, 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:03.712 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:10:03.713 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:10:03.714 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:10:03.715 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 151.64818516299999, 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:03.716 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:03.717 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:03.718 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:03.719 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:03.720 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:03.721 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:03.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:10:03.722 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:03.717+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:03.748 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2337211 2024-09-21T18:10:03.773 INFO:tasks.rados.rados.0.smithi160.stdout:3472: finishing copy_from to smithi16030472-109 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 109 v 25 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3476: got (0) Success 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: finishing write tid 3 to smithi16030472-324 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: finishing write tid 4 to smithi16030472-324 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: oid 324 updating version 0 to 1052 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: oid 324 updating version 1052 to 1053 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: oid 324 version 1053 is already newer than 1018 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 324 v 1053 (ObjNum 1467 snap 0 seq_num 1467) dirty exists 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3454: left oid 324 (ObjNum 1467 snap 0 seq_num 1467) 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3468: finishing copy_from to smithi16030472-263 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 263 v 665 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:10:03.774 INFO:tasks.rados.rados.0.smithi160.stdout:3466: got (0) Success 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 216 v 0 (ObjNum 1461 snap 0 seq_num 1461) dirty exists 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3454: done (9 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3466: done (8 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3468: done (7 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3472: done (6 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3475: done (5 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3476: done (4 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3477: done (3 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3479: done (2 left) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3480: read oid 220 snap -1 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3480: expect (ObjNum 947 snap 0 seq_num 947) 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3481: copy_from oid 256 from oid 164 current snap is 0 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3478: finishing copy_from racing read to smithi16030472-179 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:3470: got (0) Success 2024-09-21T18:10:03.775 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 0 (ObjNum 1420 snap 0 seq_num 1420) dirty exists 2024-09-21T18:10:03.776 INFO:tasks.rados.rados.0.smithi160.stdout:3470: done (3 left) 2024-09-21T18:10:03.776 INFO:tasks.rados.rados.0.smithi160.stdout:3482: cache_try_flush oid 317 snap -1 2024-09-21T18:10:03.776 INFO:tasks.rados.rados.0.smithi160.stdout:3483: write oid 21 current snap is 0 2024-09-21T18:10:03.776 INFO:tasks.rados.rados.0.smithi160.stdout:3483: seq_num 1474 ranges {773518=573155,1855164=406520} 2024-09-21T18:10:03.795 INFO:tasks.rados.rados.0.smithi160.stdout:3483: writing smithi16030472-21 from 773518 to 1346673 tid 1 2024-09-21T18:10:03.810 INFO:tasks.rados.rados.0.smithi160.stdout:3483: writing smithi16030472-21 from 1855164 to 2261684 tid 2 2024-09-21T18:10:03.810 INFO:tasks.rados.rados.0.smithi160.stdout:3484: write (excl) oid 379 current snap is 0 2024-09-21T18:10:03.811 INFO:tasks.rados.rados.0.smithi160.stdout:3478: finishing copy_from to smithi16030472-179 2024-09-21T18:10:03.811 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 225 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:10:03.811 INFO:tasks.rados.rados.0.smithi160.stdout:3484: seq_num 1475 ranges {790325=764757,2029999=550178} 2024-09-21T18:10:03.819 INFO:tasks.rados.rados.0.smithi160.stdout:3484: writing smithi16030472-379 from 790325 to 1555082 tid 1 2024-09-21T18:10:03.822 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:03.820+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:03.831 INFO:tasks.rados.rados.0.smithi160.stdout:3484: writing smithi16030472-379 from 2029999 to 2580177 tid 2 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3478: done (5 left) 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3485: read oid 80 snap -1 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3485: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3486: cache_try_flush oid 52 snap -1 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3487: read oid 26 snap -1 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3487: expect (ObjNum 881 snap 0 seq_num 881) 2024-09-21T18:10:03.832 INFO:tasks.rados.rados.0.smithi160.stdout:3488: delete oid 108 current snap is 0 2024-09-21T18:10:03.900 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:03.900 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:03.900 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:03.914 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:10:03.959 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:03.956+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:10:04.022 INFO:tasks.rados.rados.0.smithi160.stdout:3487: expect (ObjNum 881 snap 0 seq_num 881) 2024-09-21T18:10:04.024 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:04.021+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:04.089 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:04.089 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:04.089 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:04.089 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:04.089 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:04.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:10:04.125 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:04.121+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:04.129 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.181 INFO:tasks.rados.rados.0.smithi160.stdout:3484: finishing write tid 1 to smithi16030472-379 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: finishing write tid 2 to smithi16030472-379 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: finishing write tid 3 to smithi16030472-379 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: finishing write tid 4 to smithi16030472-379 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: oid 379 updating version 0 to 1054 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: oid 379 updating version 1054 to 1056 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: oid 379 version 1056 is already newer than 1055 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 379 v 1056 (ObjNum 1475 snap 0 seq_num 1475) dirty exists 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3484: left oid 379 (ObjNum 1475 snap 0 seq_num 1475) 2024-09-21T18:10:04.182 INFO:tasks.rados.rados.0.smithi160.stdout:3480: expect (ObjNum 947 snap 0 seq_num 947) 2024-09-21T18:10:04.186 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:04.180+0000 7f0b6a212640 -1 osd.9 119 handle_osd_map injecting map crc failure 2024-09-21T18:10:04.226 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:04.224+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:04.237 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 199.935427557, 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.238 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:04.239 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.240 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.241 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.242 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 199.93421349799999, 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.243 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.244 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.245 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.246 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.247 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 199.932862578, 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:04.248 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.249 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.250 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.251 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:04.252 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 193.64528194100001, 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:04.253 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.254 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:10:04.255 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:10:04.256 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.24604270099999, 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:10:04.257 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.258 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.259 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:10:04.260 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.261 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14547330900001, 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:04.262 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:10:04.263 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:10:04.264 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:10:04.265 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.145115242, 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.266 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.267 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.268 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:10:04.269 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14507840900001, 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.270 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.271 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.272 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.273 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14461867700001, 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.274 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:10:04.275 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:10:04.276 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:10:04.277 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14418714999999, 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.278 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:10:04.279 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.280 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.281 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:04.282 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14325053499999, 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.283 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.284 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:10:04.285 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.286 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.14321185200001, 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.287 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.288 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.289 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:10:04.290 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.1419252, 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.291 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.292 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.293 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.294 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.03978073600001, 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.295 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:10:04.296 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:10:04.297 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.298 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 153.03905694299999, 2024-09-21T18:10:04.299 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.300 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.301 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:10:04.302 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.303 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 144.91334670399999, 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.304 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.305 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.306 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.307 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 120.273552217, 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:04.308 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:10:04.309 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:10:04.310 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:10:04.311 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8700 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'653, mlcod=116'650)", 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.362611825, 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31160568599999999, 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.312 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871871+0000", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5370000000000002e-06 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.313 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.872549+0000", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067808400000000002 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.872577+0000", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8140000000000002e-05 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.972708+0000", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100130698 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.314 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.073005+0000", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100297214 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.073072+0000", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7106000000000004e-05 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.183367+0000", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110295064 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.315 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.183473+0000", 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000105843 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8702 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'655, mlcod=116'651)", 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.260999729, 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21321606600000001, 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.316 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.317 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973481+0000", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669999999999999e-06 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974481+0000", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00099984100000000001 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974500+0000", 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9717000000000001e-05 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.318 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974511+0000", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0723000000000001e-05 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.179699+0000", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20518751700000001 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.179742+0000", 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3769000000000001e-05 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.319 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.186579+0000", 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0068362400000000004 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.186696+0000", 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117192 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.320 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8703 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'656, mlcod=116'651)", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 10.259951594, 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21259102999999999, 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:04.321 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974529+0000", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1030000000000001e-06 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974536+0000", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5129999999999999e-06 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974544+0000", 2024-09-21T18:10:04.322 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8199999999999997e-06 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974551+0000", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7039999999999997e-06 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.182573+0000", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20802183899999999 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.182647+0000", 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3903e-05 2024-09-21T18:10:04.323 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.187031+0000", 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043840160000000001 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.187119+0000", 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8132000000000003e-05 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:04.324 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:04.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:04.325 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.326 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:10:04.328 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:04.320+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:04.332 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:04.328+0000 7fd79a4fd640 -1 osd.1 119 handle_osd_map injecting map crc failure 2024-09-21T18:10:04.360 INFO:tasks.rados.rados.0.smithi160.stdout:3482: got (0) Success 2024-09-21T18:10:04.361 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 0 (ObjNum 129 snap 0 seq_num 3138257216) clean dne 2024-09-21T18:10:04.361 INFO:tasks.rados.rados.0.smithi160.stdout:3481: finishing copy_from to smithi16030472-256 2024-09-21T18:10:04.361 INFO:tasks.rados.rados.0.smithi160.stdout:3481: got expected ENOENT (src dne) 2024-09-21T18:10:04.361 INFO:tasks.rados.rados.0.smithi160.stdout:3485: expect (ObjNum 198 snap 0 seq_num 198) 2024-09-21T18:10:04.361 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.428 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:04.424+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3481: finishing copy_from racing read to smithi16030472-256 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3483: finishing write tid 1 to smithi16030472-21 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3483: finishing write tid 2 to smithi16030472-21 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3483: finishing write tid 3 to smithi16030472-21 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3480: done (8 left) 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3481: done (7 left) 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3482: done (6 left) 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3484: done (5 left) 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3485: done (4 left) 2024-09-21T18:10:04.465 INFO:tasks.rados.rados.0.smithi160.stdout:3487: done (3 left) 2024-09-21T18:10:04.466 INFO:tasks.rados.rados.0.smithi160.stdout:3488: done (2 left) 2024-09-21T18:10:04.466 INFO:tasks.rados.rados.0.smithi160.stdout:3489: copy_from oid 396 from oid 295 current snap is 0 2024-09-21T18:10:04.466 INFO:tasks.rados.rados.0.smithi160.stdout:3490: read oid 406 snap -1 2024-09-21T18:10:04.466 INFO:tasks.rados.rados.0.smithi160.stdout:3490: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:10:04.466 INFO:tasks.rados.rados.0.smithi160.stdout:3491: delete oid 480 current snap is 0 2024-09-21T18:10:04.466 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:04.466 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:04.466 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:04.466 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:04.479 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:10:04.483 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:04.483 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:04.503 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:04.525 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:04.525 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:04.529 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:04.521+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:04.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:04.546 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:04.563 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:04.563 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:3483: finishing write tid 4 to smithi16030472-21 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:3483: oid 21 updating version 0 to 972 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:3483: oid 21 updating version 972 to 974 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:3483: oid 21 version 974 is already newer than 973 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 21 v 974 (ObjNum 1474 snap 0 seq_num 1474) dirty exists 2024-09-21T18:10:04.566 INFO:tasks.rados.rados.0.smithi160.stdout:3483: left oid 21 (ObjNum 1474 snap 0 seq_num 1474) 2024-09-21T18:10:04.575 INFO:tasks.rados.rados.0.smithi160.stdout:3490: expect (ObjNum 172 snap 0 seq_num 172) 2024-09-21T18:10:04.575 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.581 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:04.581 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:04.603 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:04.603 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:04.603 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:04.605 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.622 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:04.622 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:04.629 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:04.625+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:04.644 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:04.644 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:04.647 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:04.648 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:04.660 INFO:tasks.rados.rados.0.smithi160.stdout:3486: got (0) Success 2024-09-21T18:10:04.660 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 52 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:10:04.662 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:10:04.667 INFO:tasks.rados.rados.0.smithi160.stdout:3489: finishing copy_from to smithi16030472-396 2024-09-21T18:10:04.667 INFO:tasks.rados.rados.0.smithi160.stdout:3489: got expected ENOENT (src dne) 2024-09-21T18:10:04.667 INFO:tasks.rados.rados.0.smithi160.stdout:3489: finishing copy_from racing read to smithi16030472-396 2024-09-21T18:10:04.668 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:04.668 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:04.669 INFO:tasks.rados.rados.0.smithi160.stdout:3483: done (4 left) 2024-09-21T18:10:04.669 INFO:tasks.rados.rados.0.smithi160.stdout:3486: done (3 left) 2024-09-21T18:10:04.669 INFO:tasks.rados.rados.0.smithi160.stdout:3489: done (2 left) 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3490: done (1 left) 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3491: done (0 left) 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3492: cache_flush oid 468 snap -1 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3493: read oid 409 snap -1 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3493: expect deleted 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3494: read oid 275 snap -1 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3494: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3495: cache_flush oid 350 snap -1 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3496: cache_evict oid 230 snap -1 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3497: copy_from oid 198 from oid 108 current snap is 0 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3498: write oid 381 current snap is 0 2024-09-21T18:10:04.670 INFO:tasks.rados.rados.0.smithi160.stdout:3498: seq_num 1478 ranges {440746=590504,1689092=697505} 2024-09-21T18:10:04.684 INFO:tasks.rados.rados.0.smithi160.stdout:3498: writing smithi16030472-381 from 440746 to 1031250 tid 1 2024-09-21T18:10:04.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:04.690 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:3498: writing smithi16030472-381 from 1689092 to 2386597 tid 2 2024-09-21T18:10:04.706 INFO:tasks.rados.rados.0.smithi160.stdout:3499: cache_evict oid 497 snap -1 2024-09-21T18:10:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:3500: cache_evict oid 197 snap -1 2024-09-21T18:10:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:3501: read oid 303 snap -1 2024-09-21T18:10:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:3501: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:04.707 INFO:tasks.rados.rados.0.smithi160.stdout:3502: delete oid 144 current snap is 0 2024-09-21T18:10:04.710 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:04.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:04.730 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:04.725+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:04.731 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:04.753 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:04.754 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:04.771 INFO:tasks.rados.rados.0.smithi160.stdout:3496: got (16) Device or resource busy 2024-09-21T18:10:04.780 INFO:tasks.rados.rados.0.smithi160.stdout:3494: expect (ObjNum 274 snap 0 seq_num 274) 2024-09-21T18:10:04.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:04.781 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:04.785 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:04.785 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:04.789 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:04.789 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:04.803 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:10:04.822 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.830 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:04.829+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:04.868 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:04.898 INFO:tasks.rados.rados.0.smithi160.stdout:3497: finishing copy_from to smithi16030472-198 2024-09-21T18:10:04.898 INFO:tasks.rados.rados.0.smithi160.stdout:3497: got expected ENOENT (src dne) 2024-09-21T18:10:04.898 INFO:tasks.rados.rados.0.smithi160.stdout:3497: finishing copy_from racing read to smithi16030472-198 2024-09-21T18:10:04.898 INFO:tasks.rados.rados.0.smithi160.stdout:3498: finishing write tid 1 to smithi16030472-381 2024-09-21T18:10:04.931 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:04.924+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:04.933 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:04.933 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:04.947 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:10:04.996 INFO:tasks.rados.rados.0.smithi160.stdout:3501: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:04.996 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:05.031 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:05.028+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:05.066 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.066 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.080 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:10:05.132 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:05.125+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:05.172 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3369373 2024-09-21T18:10:05.233 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:05.225+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:05.249 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.250 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:10:05.291 INFO:tasks.rados.rados.0.smithi160.stdout:3492: got (0) Success 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 468 v 0 (ObjNum 1412 snap 0 seq_num 1412) dirty exists 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3500: got (0) Success 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3495: got (0) Success 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 0 (ObjNum 1207 snap 0 seq_num 1207) dirty exists 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: finishing write tid 2 to smithi16030472-381 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: finishing write tid 3 to smithi16030472-381 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: finishing write tid 4 to smithi16030472-381 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: oid 381 updating version 0 to 1038 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: oid 381 updating version 1038 to 1039 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: oid 381 updating version 1039 to 1040 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 1040 (ObjNum 1478 snap 0 seq_num 1478) dirty exists 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3498: left oid 381 (ObjNum 1478 snap 0 seq_num 1478) 2024-09-21T18:10:05.292 INFO:tasks.rados.rados.0.smithi160.stdout:3499: got (0) Success 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3492: done (10 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3493: done (9 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3494: done (8 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3495: done (7 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3496: done (6 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3497: done (5 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3498: done (4 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3499: done (3 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3500: done (2 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3501: done (1 left) 2024-09-21T18:10:05.293 INFO:tasks.rados.rados.0.smithi160.stdout:3502: done (0 left) 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3503: copy_from oid 350 from oid 428 current snap is 0 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3504: cache_try_flush oid 295 snap -1 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3505: read oid 290 snap -1 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3505: expect (ObjNum 1316 snap 0 seq_num 1316) 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3506: cache_flush oid 123 snap -1 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3507: write oid 143 current snap is 0 2024-09-21T18:10:05.294 INFO:tasks.rados.rados.0.smithi160.stdout:3507: seq_num 1480 ranges {474360=741002,1806056=701174} 2024-09-21T18:10:05.303 INFO:tasks.rados.rados.0.smithi160.stdout:3507: writing smithi16030472-143 from 474360 to 1215362 tid 1 2024-09-21T18:10:05.312 INFO:tasks.rados.rados.0.smithi160.stdout:3507: writing smithi16030472-143 from 1806056 to 2507230 tid 2 2024-09-21T18:10:05.312 INFO:tasks.rados.rados.0.smithi160.stdout:3508: write (excl) oid 460 current snap is 0 2024-09-21T18:10:05.313 INFO:tasks.rados.rados.0.smithi160.stdout:3508: seq_num 1481 ranges {491167=416251,1464538=561185,2753570=137597} 2024-09-21T18:10:05.318 INFO:tasks.rados.rados.0.smithi160.stdout:3508: writing smithi16030472-460 from 491167 to 907418 tid 1 2024-09-21T18:10:05.326 INFO:tasks.rados.rados.0.smithi160.stdout:3508: writing smithi16030472-460 from 1464538 to 2025723 tid 2 2024-09-21T18:10:05.334 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:05.328+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:05.335 INFO:tasks.rados.rados.0.smithi160.stdout:3508: writing smithi16030472-460 from 2753570 to 2891167 tid 3 2024-09-21T18:10:05.335 INFO:tasks.rados.rados.0.smithi160.stdout:3509: read oid 344 snap -1 2024-09-21T18:10:05.335 INFO:tasks.rados.rados.0.smithi160.stdout:3509: expect (ObjNum 815 snap 0 seq_num 815) 2024-09-21T18:10:05.335 INFO:tasks.rados.rados.0.smithi160.stdout:3510: write oid 256 current snap is 0 2024-09-21T18:10:05.335 INFO:tasks.rados.rados.0.smithi160.stdout:3510: seq_num 1482 ranges {507974=491500,1406667=704843,2548287=359687} 2024-09-21T18:10:05.343 INFO:tasks.rados.rados.0.smithi160.stdout:3510: writing smithi16030472-256 from 507974 to 999474 tid 1 2024-09-21T18:10:05.351 INFO:tasks.rados.rados.0.smithi160.stdout:3510: writing smithi16030472-256 from 1406667 to 2111510 tid 2 2024-09-21T18:10:05.357 INFO:tasks.rados.rados.0.smithi160.stdout:3510: writing smithi16030472-256 from 2548287 to 2907974 tid 3 2024-09-21T18:10:05.357 INFO:tasks.rados.rados.0.smithi160.stdout:3511: delete oid 480 current snap is 0 2024-09-21T18:10:05.357 INFO:tasks.rados.rados.0.smithi160.stdout:3506: got (0) Success 2024-09-21T18:10:05.357 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 0 (ObjNum 1439 snap 0 seq_num 1439) dirty exists 2024-09-21T18:10:05.397 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.397 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.412 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:10:05.423 INFO:tasks.rados.rados.0.smithi160.stdout:3508: finishing write tid 1 to smithi16030472-460 2024-09-21T18:10:05.434 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:05.433+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:05.436 INFO:tasks.rados.rados.0.smithi160.stdout:3504: got (0) Success 2024-09-21T18:10:05.436 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 295 v 0 (ObjNum 257 snap 0 seq_num 3149867680) clean dne 2024-09-21T18:10:05.443 INFO:tasks.rados.rados.0.smithi160.stdout:3509: expect (ObjNum 815 snap 0 seq_num 815) 2024-09-21T18:10:05.443 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:05.536 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:05.532+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:05.538 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.538 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:10:05.637 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:05.633+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:05.641 INFO:tasks.rados.rados.0.smithi160.stdout:3505: expect (ObjNum 1316 snap 0 seq_num 1316) 2024-09-21T18:10:05.673 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.673 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.680 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:05.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3507: finishing write tid 1 to smithi16030472-143 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3503: finishing copy_from racing read to smithi16030472-350 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3508: finishing write tid 2 to smithi16030472-460 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3508: finishing write tid 3 to smithi16030472-460 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3508: finishing write tid 4 to smithi16030472-460 2024-09-21T18:10:05.702 INFO:tasks.rados.rados.0.smithi160.stdout:3508: finishing write tid 5 to smithi16030472-460 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: oid 460 updating version 0 to 1058 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: oid 460 updating version 1058 to 1061 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: oid 460 version 1061 is already newer than 1059 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: oid 460 version 1061 is already newer than 1060 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 460 v 1061 (ObjNum 1481 snap 0 seq_num 1481) dirty exists 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: left oid 460 (ObjNum 1481 snap 0 seq_num 1481) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3504: done (8 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3505: done (7 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3506: done (6 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3508: done (5 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3509: done (4 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3511: done (3 left) 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3512: copy_from oid 62 from oid 497 current snap is 0 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3513: read oid 467 snap -1 2024-09-21T18:10:05.703 INFO:tasks.rados.rados.0.smithi160.stdout:3513: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:10:05.704 INFO:tasks.rados.rados.0.smithi160.stdout:3514: write oid 289 current snap is 0 2024-09-21T18:10:05.704 INFO:tasks.rados.rados.0.smithi160.stdout:3514: seq_num 1484 ranges {541588=758351,2039984=708512} 2024-09-21T18:10:05.713 INFO:tasks.rados.rados.0.smithi160.stdout:3514: writing smithi16030472-289 from 541588 to 1299939 tid 1 2024-09-21T18:10:05.727 INFO:tasks.rados.rados.0.smithi160.stdout:3514: writing smithi16030472-289 from 2039984 to 2748496 tid 2 2024-09-21T18:10:05.727 INFO:tasks.rados.rados.0.smithi160.stdout:3515: write (excl) oid 105 current snap is 0 2024-09-21T18:10:05.727 INFO:tasks.rados.rados.0.smithi160.stdout:3515: seq_num 1485 ranges {558395=433600,1698466=568523,2863262=95133} 2024-09-21T18:10:05.736 INFO:tasks.rados.rados.0.smithi160.stdout:3515: writing smithi16030472-105 from 558395 to 991995 tid 1 2024-09-21T18:10:05.737 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:05.729+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:05.747 INFO:tasks.rados.rados.0.smithi160.stdout:3515: writing smithi16030472-105 from 1698466 to 2266989 tid 2 2024-09-21T18:10:05.753 INFO:tasks.rados.rados.0.smithi160.stdout:3515: writing smithi16030472-105 from 2863262 to 2958395 tid 3 2024-09-21T18:10:05.753 INFO:tasks.rados.rados.0.smithi160.stdout:3516: write oid 423 current snap is 0 2024-09-21T18:10:05.753 INFO:tasks.rados.rados.0.smithi160.stdout:3516: seq_num 1486 ranges {575202=508849,1640595=712181} 2024-09-21T18:10:05.762 INFO:tasks.rados.rados.0.smithi160.stdout:3516: writing smithi16030472-423 from 575202 to 1084051 tid 1 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3516: writing smithi16030472-423 from 1640595 to 2352776 tid 2 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3517: cache_evict oid 150 snap -1 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3518: read oid 138 snap -1 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3518: expect (ObjNum 1016 snap 0 seq_num 1016) 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3519: cache_flush oid 277 snap -1 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3520: delete oid 192 current snap is 0 2024-09-21T18:10:05.772 INFO:tasks.rados.rados.0.smithi160.stdout:3510: finishing write tid 1 to smithi16030472-256 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: finishing write tid 2 to smithi16030472-256 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3503: finishing copy_from to smithi16030472-350 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 350 v 10 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: finishing write tid 3 to smithi16030472-256 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: finishing write tid 4 to smithi16030472-256 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: finishing write tid 5 to smithi16030472-256 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: oid 256 updating version 0 to 1008 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: oid 256 version 1008 is already newer than 1007 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: oid 256 version 1008 is already newer than 1006 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: oid 256 updating version 1008 to 1009 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 256 v 1009 (ObjNum 1482 snap 0 seq_num 1482) dirty exists 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3510: left oid 256 (ObjNum 1482 snap 0 seq_num 1482) 2024-09-21T18:10:05.773 INFO:tasks.rados.rados.0.smithi160.stdout:3513: expect (ObjNum 862 snap 0 seq_num 862) 2024-09-21T18:10:05.813 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.813 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:05.827 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:10:05.838 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:05.833+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:05.877 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:05.931 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:05.938 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:05.937+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:05.984 INFO:tasks.rados.rados.0.smithi160.stdout:3507: finishing write tid 2 to smithi16030472-143 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: finishing write tid 1 to smithi16030472-289 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: finishing write tid 2 to smithi16030472-289 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: finishing write tid 3 to smithi16030472-289 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: finishing write tid 4 to smithi16030472-289 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: oid 289 updating version 0 to 1063 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: oid 289 version 1063 is already newer than 1062 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: oid 289 updating version 1063 to 1064 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 289 v 1064 (ObjNum 1484 snap 0 seq_num 1484) dirty exists 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3514: left oid 289 (ObjNum 1484 snap 0 seq_num 1484) 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3515: finishing write tid 1 to smithi16030472-105 2024-09-21T18:10:05.985 INFO:tasks.rados.rados.0.smithi160.stdout:3518: expect (ObjNum 1016 snap 0 seq_num 1016) 2024-09-21T18:10:05.994 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:05.994 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.010 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:10:06.039 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:06.032+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:06.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:10:06.054 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2549555 2024-09-21T18:10:06.067 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1441352 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3516: finishing write tid 1 to smithi16030472-423 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3517: got (0) Success 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3507: finishing write tid 3 to smithi16030472-143 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3507: finishing write tid 4 to smithi16030472-143 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3507: oid 143 updating version 0 to 976 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3507: oid 143 updating version 976 to 977 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:3507: oid 143 version 977 is already newer than 962 2024-09-21T18:10:06.101 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 143 v 977 (ObjNum 1480 snap 0 seq_num 1480) dirty exists 2024-09-21T18:10:06.102 INFO:tasks.rados.rados.0.smithi160.stdout:3507: left oid 143 (ObjNum 1480 snap 0 seq_num 1480) 2024-09-21T18:10:06.102 INFO:tasks.rados.rados.0.smithi160.stdout:3512: finishing copy_from to smithi16030472-62 2024-09-21T18:10:06.102 INFO:tasks.rados.rados.0.smithi160.stdout:3512: got expected ENOENT (src dne) 2024-09-21T18:10:06.102 INFO:tasks.rados.rados.0.smithi160.stdout:3512: finishing copy_from racing read to smithi16030472-62 2024-09-21T18:10:06.102 INFO:tasks.rados.rados.0.smithi160.stdout:3516: finishing write tid 2 to smithi16030472-423 2024-09-21T18:10:06.135 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.135 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.140 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:06.133+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:06.150 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: finishing write tid 3 to smithi16030472-423 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: finishing write tid 4 to smithi16030472-423 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: oid 423 updating version 0 to 1045 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: oid 423 version 1045 is already newer than 1012 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: oid 423 updating version 1045 to 1046 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 423 v 1046 (ObjNum 1486 snap 0 seq_num 1486) dirty exists 2024-09-21T18:10:06.194 INFO:tasks.rados.rados.0.smithi160.stdout:3516: left oid 423 (ObjNum 1486 snap 0 seq_num 1486) 2024-09-21T18:10:06.242 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:06.237+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3519: got (0) Success 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 277 v 0 (ObjNum 129 snap 0 seq_num 3136074016) clean dne 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3503: done (11 left) 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3507: done (10 left) 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3510: done (9 left) 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3512: done (8 left) 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3513: done (7 left) 2024-09-21T18:10:06.247 INFO:tasks.rados.rados.0.smithi160.stdout:3514: done (6 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3516: done (5 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3517: done (4 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3518: done (3 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3519: done (2 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3520: done (1 left) 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3521: cache_evict oid 492 snap -1 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3522: write oid 329 current snap is 0 2024-09-21T18:10:06.248 INFO:tasks.rados.rados.0.smithi160.stdout:3522: seq_num 1488 ranges {608816=659347,1757559=715850} 2024-09-21T18:10:06.257 INFO:tasks.rados.rados.0.smithi160.stdout:3522: writing smithi16030472-329 from 608816 to 1268163 tid 1 2024-09-21T18:10:06.265 INFO:tasks.rados.rados.0.smithi160.stdout:3522: writing smithi16030472-329 from 1757559 to 2473409 tid 2 2024-09-21T18:10:06.265 INFO:tasks.rados.rados.0.smithi160.stdout:3523: write (excl) oid 412 current snap is 0 2024-09-21T18:10:06.265 INFO:tasks.rados.rados.0.smithi160.stdout:3523: seq_num 1489 ranges {625623=734596,1816041=459508,2740248=285375} 2024-09-21T18:10:06.274 INFO:tasks.rados.rados.0.smithi160.stdout:3523: writing smithi16030472-412 from 625623 to 1360219 tid 1 2024-09-21T18:10:06.280 INFO:tasks.rados.rados.0.smithi160.stdout:3523: writing smithi16030472-412 from 1816041 to 2275549 tid 2 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3523: writing smithi16030472-412 from 2740248 to 3025623 tid 3 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3524: cache_evict oid 176 snap -1 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3525: read oid 401 snap -1 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3525: expect (ObjNum 400 snap 0 seq_num 400) 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3526: cache_flush oid 199 snap -1 2024-09-21T18:10:06.286 INFO:tasks.rados.rados.0.smithi160.stdout:3515: finishing write tid 2 to smithi16030472-105 2024-09-21T18:10:06.287 INFO:tasks.rados.rados.0.smithi160.stdout:3527: copy_from oid 326 from oid 484 current snap is 0 2024-09-21T18:10:06.287 INFO:tasks.rados.rados.0.smithi160.stdout:3528: write oid 80 current snap is 0 2024-09-21T18:10:06.287 INFO:tasks.rados.rados.0.smithi160.stdout:3528: seq_num 1490 ranges {642430=409845,1758170=719519} 2024-09-21T18:10:06.293 INFO:tasks.rados.rados.0.smithi160.stdout:3528: writing smithi16030472-80 from 642430 to 1052275 tid 1 2024-09-21T18:10:06.302 INFO:tasks.rados.rados.0.smithi160.stdout:3528: writing smithi16030472-80 from 1758170 to 2477689 tid 2 2024-09-21T18:10:06.303 INFO:tasks.rados.rados.0.smithi160.stdout:3529: write oid 91 current snap is 0 2024-09-21T18:10:06.303 INFO:tasks.rados.rados.0.smithi160.stdout:3521: got (16) Device or resource busy 2024-09-21T18:10:06.303 INFO:tasks.rados.rados.0.smithi160.stdout:3525: expect (ObjNum 400 snap 0 seq_num 400) 2024-09-21T18:10:06.326 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.326 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.340 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:10:06.342 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:06.336+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:06.410 INFO:tasks.rados.rados.0.smithi160.stdout:3522: finishing write tid 1 to smithi16030472-329 2024-09-21T18:10:06.411 INFO:tasks.rados.rados.0.smithi160.stdout:3529: seq_num 1491 ranges {659237=601447,1933005=463177} 2024-09-21T18:10:06.420 INFO:tasks.rados.rados.0.smithi160.stdout:3529: writing smithi16030472-91 from 659237 to 1260684 tid 1 2024-09-21T18:10:06.428 INFO:tasks.rados.rados.0.smithi160.stdout:3529: writing smithi16030472-91 from 1933005 to 2396182 tid 2 2024-09-21T18:10:06.428 INFO:tasks.rados.rados.0.smithi160.stdout:3521: done (9 left) 2024-09-21T18:10:06.428 INFO:tasks.rados.rados.0.smithi160.stdout:3525: done (8 left) 2024-09-21T18:10:06.429 INFO:tasks.rados.rados.0.smithi160.stdout:3530: cache_try_flush oid 296 snap -1 2024-09-21T18:10:06.429 INFO:tasks.rados.rados.0.smithi160.stdout:3531: write (excl) oid 273 current snap is 0 2024-09-21T18:10:06.429 INFO:tasks.rados.rados.0.smithi160.stdout:3531: seq_num 1492 ranges {676044=676696,1991487=723188} 2024-09-21T18:10:06.438 INFO:tasks.rados.rados.0.smithi160.stdout:3531: writing smithi16030472-273 from 676044 to 1352740 tid 1 2024-09-21T18:10:06.443 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:06.436+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3531: writing smithi16030472-273 from 1991487 to 2714675 tid 2 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3532: read oid 237 snap -1 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3532: expect (ObjNum 1295 snap 0 seq_num 1295) 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3533: read oid 381 snap -1 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3533: expect (ObjNum 1478 snap 0 seq_num 1478) 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3534: cache_flush oid 410 snap -1 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3535: cache_evict oid 265 snap -1 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3536: write oid 114 current snap is 0 2024-09-21T18:10:06.449 INFO:tasks.rados.rados.0.smithi160.stdout:3536: seq_num 1493 ranges {692851=751945,2049969=466846} 2024-09-21T18:10:06.459 INFO:tasks.rados.rados.0.smithi160.stdout:3536: writing smithi16030472-114 from 692851 to 1444796 tid 1 2024-09-21T18:10:06.467 INFO:tasks.rados.rados.0.smithi160.stdout:3536: writing smithi16030472-114 from 2049969 to 2516815 tid 2 2024-09-21T18:10:06.467 INFO:tasks.rados.rados.0.smithi160.stdout:3537: read oid 299 snap -1 2024-09-21T18:10:06.467 INFO:tasks.rados.rados.0.smithi160.stdout:3537: expect (ObjNum 1257 snap 0 seq_num 1257) 2024-09-21T18:10:06.467 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:06.492 INFO:tasks.rados.rados.0.smithi160.stdout:3522: finishing write tid 2 to smithi16030472-329 2024-09-21T18:10:06.493 INFO:tasks.rados.rados.0.smithi160.stdout:3522: finishing write tid 3 to smithi16030472-329 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: finishing write tid 4 to smithi16030472-329 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: oid 329 updating version 0 to 1065 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: oid 329 updating version 1065 to 1066 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: oid 329 updating version 1066 to 1067 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 329 v 1067 (ObjNum 1488 snap 0 seq_num 1488) dirty exists 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: left oid 329 (ObjNum 1488 snap 0 seq_num 1488) 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3522: done (15 left) 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout:3538: copy_from oid 453 from oid 481 current snap is 0 2024-09-21T18:10:06.494 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:06.508 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.508 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:10:06.544 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:06.541+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:06.557 INFO:tasks.rados.rados.0.smithi160.stdout:3532: expect (ObjNum 1295 snap 0 seq_num 1295) 2024-09-21T18:10:06.611 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:06.645 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:06.637+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:06.646 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.646 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.660 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:10:06.668 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 500362 2024-09-21T18:10:06.720 INFO:tasks.rados.rados.0.smithi160.stdout:3533: expect (ObjNum 1478 snap 0 seq_num 1478) 2024-09-21T18:10:06.720 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:06.746 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:06.744+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:06.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:06.829 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.829 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.839 INFO:tasks.rados.rados.0.smithi160.stdout:3523: finishing write tid 1 to smithi16030472-412 2024-09-21T18:10:06.839 INFO:tasks.rados.rados.0.smithi160.stdout:3523: finishing write tid 2 to smithi16030472-412 2024-09-21T18:10:06.839 INFO:tasks.rados.rados.0.smithi160.stdout:3523: finishing write tid 3 to smithi16030472-412 2024-09-21T18:10:06.839 INFO:tasks.rados.rados.0.smithi160.stdout:3527: finishing copy_from to smithi16030472-326 2024-09-21T18:10:06.839 INFO:tasks.rados.rados.0.smithi160.stdout:3527: got expected ENOENT (src dne) 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3527: finishing copy_from racing read to smithi16030472-326 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3523: finishing write tid 4 to smithi16030472-412 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3524: got (0) Success 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: finishing write tid 3 to smithi16030472-105 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: finishing write tid 4 to smithi16030472-105 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: finishing write tid 5 to smithi16030472-105 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: oid 105 updating version 0 to 964 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: oid 105 updating version 964 to 979 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: oid 105 updating version 979 to 980 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: oid 105 version 980 is already newer than 963 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 105 v 980 (ObjNum 1485 snap 0 seq_num 1485) dirty exists 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3515: left oid 105 (ObjNum 1485 snap 0 seq_num 1485) 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3528: finishing write tid 1 to smithi16030472-80 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3528: finishing write tid 2 to smithi16030472-80 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3528: finishing write tid 3 to smithi16030472-80 2024-09-21T18:10:06.840 INFO:tasks.rados.rados.0.smithi160.stdout:3528: finishing write tid 4 to smithi16030472-80 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3528: oid 80 updating version 0 to 1011 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3528: oid 80 version 1011 is already newer than 1010 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3528: oid 80 updating version 1011 to 1012 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 1012 (ObjNum 1490 snap 0 seq_num 1490) dirty exists 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3528: left oid 80 (ObjNum 1490 snap 0 seq_num 1490) 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3530: got (2) No such file or directory 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3534: got (0) Success 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 0 (ObjNum 129 snap 0 seq_num 3128284648) clean dne 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3529: finishing write tid 1 to smithi16030472-91 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3529: finishing write tid 2 to smithi16030472-91 2024-09-21T18:10:06.841 INFO:tasks.rados.rados.0.smithi160.stdout:3537: expect (ObjNum 1257 snap 0 seq_num 1257) 2024-09-21T18:10:06.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:10:06.846 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:06.840+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:06.888 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:06.944 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:06.944 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:06.947 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:06.941+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:06.968 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:06.968 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:06.982 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:10:06.986 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:06.986 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: finishing write tid 3 to smithi16030472-91 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3531: finishing write tid 1 to smithi16030472-273 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: finishing write tid 4 to smithi16030472-91 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: oid 91 updating version 0 to 1014 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: oid 91 updating version 1014 to 1015 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: oid 91 version 1015 is already newer than 1013 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 1015 (ObjNum 1491 snap 0 seq_num 1491) dirty exists 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3529: left oid 91 (ObjNum 1491 snap 0 seq_num 1491) 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3523: finishing write tid 5 to smithi16030472-412 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3523: oid 412 updating version 0 to 1071 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3523: oid 412 version 1071 is already newer than 1069 2024-09-21T18:10:06.988 INFO:tasks.rados.rados.0.smithi160.stdout:3523: oid 412 version 1071 is already newer than 1068 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3523: oid 412 version 1071 is already newer than 1070 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 412 v 1071 (ObjNum 1489 snap 0 seq_num 1489) dirty exists 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3523: left oid 412 (ObjNum 1489 snap 0 seq_num 1489) 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: finishing write tid 2 to smithi16030472-273 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: finishing write tid 3 to smithi16030472-273 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: finishing write tid 4 to smithi16030472-273 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: oid 273 updating version 0 to 981 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: oid 273 updating version 981 to 982 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: oid 273 updating version 982 to 983 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 273 v 983 (ObjNum 1492 snap 0 seq_num 1492) dirty exists 2024-09-21T18:10:06.989 INFO:tasks.rados.rados.0.smithi160.stdout:3531: left oid 273 (ObjNum 1492 snap 0 seq_num 1492) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3536: finishing write tid 1 to smithi16030472-114 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3535: got (0) Success 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3515: done (15 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3523: done (14 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3524: done (13 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3527: done (12 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3528: done (11 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3529: done (10 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3530: done (9 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3531: done (8 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3532: done (7 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3533: done (6 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3534: done (5 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3535: done (4 left) 2024-09-21T18:10:06.990 INFO:tasks.rados.rados.0.smithi160.stdout:3537: done (3 left) 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3539: delete oid 469 current snap is 0 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: finishing write tid 2 to smithi16030472-114 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: finishing write tid 3 to smithi16030472-114 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: finishing write tid 4 to smithi16030472-114 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: oid 114 updating version 0 to 986 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: oid 114 version 986 is already newer than 984 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: oid 114 version 986 is already newer than 985 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 986 (ObjNum 1493 snap 0 seq_num 1493) dirty exists 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3536: left oid 114 (ObjNum 1493 snap 0 seq_num 1493) 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3538: finishing copy_from racing read to smithi16030472-453 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:3538: finishing copy_from to smithi16030472-453 2024-09-21T18:10:06.991 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 453 v 837 (ObjNum 1152 snap 0 seq_num 1152) dirty exists 2024-09-21T18:10:07.006 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:07.006 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:07.010 INFO:tasks.rados.rados.0.smithi160.stdout:3526: got (0) Success 2024-09-21T18:10:07.010 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 0 (ObjNum 1378 snap 0 seq_num 1378) dirty exists 2024-09-21T18:10:07.025 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:07.026 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:07.031 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:10:07.048 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:07.069 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:07.072 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:07.064+0000 7fb25ec0a640 -1 osd.11 120 handle_osd_map injecting map crc failure 2024-09-21T18:10:07.074 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:07.069+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:07.087 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:07.087 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:07.090 INFO:tasks.rados.rados.0.smithi160.stdout:3526: done (3 left) 2024-09-21T18:10:07.090 INFO:tasks.rados.rados.0.smithi160.stdout:3536: done (2 left) 2024-09-21T18:10:07.091 INFO:tasks.rados.rados.0.smithi160.stdout:3538: done (1 left) 2024-09-21T18:10:07.091 INFO:tasks.rados.rados.0.smithi160.stdout:3539: done (0 left) 2024-09-21T18:10:07.091 INFO:tasks.rados.rados.0.smithi160.stdout:3540: write (excl) oid 341 current snap is 0 2024-09-21T18:10:07.091 INFO:tasks.rados.rados.0.smithi160.stdout:3540: seq_num 1495 ranges {726465=502443,1650580=470515,2788433=338032} 2024-09-21T18:10:07.107 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:07.108 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:07.110 INFO:tasks.rados.rados.0.smithi160.stdout:3540: writing smithi16030472-341 from 726465 to 1228908 tid 1 2024-09-21T18:10:07.125 INFO:tasks.rados.rados.0.smithi160.stdout:3540: writing smithi16030472-341 from 1650580 to 2121095 tid 2 2024-09-21T18:10:07.141 INFO:tasks.rados.rados.0.smithi160.stdout:3540: writing smithi16030472-341 from 2788433 to 3126465 tid 3 2024-09-21T18:10:07.141 INFO:tasks.rados.rados.0.smithi160.stdout:3541: copy_from oid 113 from oid 184 current snap is 0 2024-09-21T18:10:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:3540: finishing write tid 1 to smithi16030472-341 2024-09-21T18:10:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:3542: read oid 190 snap -1 2024-09-21T18:10:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:3542: expect (ObjNum 1150 snap 0 seq_num 1150) 2024-09-21T18:10:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:3543: write (excl) oid 76 current snap is 0 2024-09-21T18:10:07.142 INFO:tasks.rados.rados.0.smithi160.stdout:3543: seq_num 1496 ranges {743272=577692,2109062=730526} 2024-09-21T18:10:07.148 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:07.145+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:07.154 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:07.154 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:07.157 INFO:tasks.rados.rados.0.smithi160.stdout:3543: writing smithi16030472-76 from 743272 to 1320964 tid 1 2024-09-21T18:10:07.168 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3543: writing smithi16030472-76 from 2109062 to 2839588 tid 2 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3544: copy_from oid 319 from oid 345 current snap is 0 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3540: finishing write tid 2 to smithi16030472-341 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3540: finishing write tid 3 to smithi16030472-341 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3545: cache_flush oid 136 snap -1 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3546: write (excl) oid 99 current snap is 0 2024-09-21T18:10:07.172 INFO:tasks.rados.rados.0.smithi160.stdout:3546: seq_num 1497 ranges {760079=652941,2167544=474184} 2024-09-21T18:10:07.183 INFO:tasks.rados.rados.0.smithi160.stdout:3546: writing smithi16030472-99 from 760079 to 1413020 tid 1 2024-09-21T18:10:07.191 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:07.192 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:07.193 INFO:tasks.rados.rados.0.smithi160.stdout:3546: writing smithi16030472-99 from 2167544 to 2641728 tid 2 2024-09-21T18:10:07.193 INFO:tasks.rados.rados.0.smithi160.stdout:3547: write (excl) oid 301 current snap is 0 2024-09-21T18:10:07.193 INFO:tasks.rados.rados.0.smithi160.stdout:3547: seq_num 1498 ranges {776886=444543,1826026=734195,2987055=189831} 2024-09-21T18:10:07.203 INFO:tasks.rados.rados.0.smithi160.stdout:3547: writing smithi16030472-301 from 776886 to 1221429 tid 1 2024-09-21T18:10:07.214 INFO:tasks.rados.rados.0.smithi160.stdout:3547: writing smithi16030472-301 from 1826026 to 2560221 tid 2 2024-09-21T18:10:07.215 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:07.215 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:07.219 INFO:tasks.rados.rados.0.smithi160.stdout:3547: writing smithi16030472-301 from 2987055 to 3176886 tid 3 2024-09-21T18:10:07.219 INFO:tasks.rados.rados.0.smithi160.stdout:3548: write oid 224 current snap is 0 2024-09-21T18:10:07.219 INFO:tasks.rados.rados.0.smithi160.stdout:3548: seq_num 1499 ranges {793693=519792,1884508=477853,2898125=295568} 2024-09-21T18:10:07.228 INFO:tasks.rados.rados.0.smithi160.stdout:3548: writing smithi16030472-224 from 793693 to 1313485 tid 1 2024-09-21T18:10:07.234 INFO:tasks.rados.rados.0.smithi160.stdout:3548: writing smithi16030472-224 from 1884508 to 2362361 tid 2 2024-09-21T18:10:07.239 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:07.239 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:07.240 INFO:tasks.rados.rados.0.smithi160.stdout:3548: writing smithi16030472-224 from 2898125 to 3193693 tid 3 2024-09-21T18:10:07.240 INFO:tasks.rados.rados.0.smithi160.stdout:3549: read oid 130 snap -1 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3549: expect (ObjNum 1428 snap 0 seq_num 1428) 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3550: read oid 232 snap -1 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3550: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3551: cache_flush oid 108 snap -1 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: finishing write tid 4 to smithi16030472-341 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: finishing write tid 5 to smithi16030472-341 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: oid 341 updating version 0 to 1051 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: oid 341 version 1051 is already newer than 1050 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: oid 341 updating version 1051 to 1052 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: oid 341 version 1052 is already newer than 1049 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 341 v 1052 (ObjNum 1495 snap 0 seq_num 1495) dirty exists 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: left oid 341 (ObjNum 1495 snap 0 seq_num 1495) 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3540: done (11 left) 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3552: cache_try_flush oid 488 snap -1 2024-09-21T18:10:07.241 INFO:tasks.rados.rados.0.smithi160.stdout:3553: write (excl) oid 428 current snap is 0 2024-09-21T18:10:07.242 INFO:tasks.rados.rados.0.smithi160.stdout:3553: seq_num 1500 ranges {410500=595041,1542990=621511,2925548=284952} 2024-09-21T18:10:07.247 INFO:tasks.rados.rados.0.smithi160.stdout:3553: writing smithi16030472-428 from 410500 to 1005541 tid 1 2024-09-21T18:10:07.248 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:07.241+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:07.254 INFO:tasks.rados.rados.0.smithi160.stdout:3553: writing smithi16030472-428 from 1542990 to 2164501 tid 2 2024-09-21T18:10:07.261 INFO:tasks.rados.rados.0.smithi160.stdout:3553: writing smithi16030472-428 from 2925548 to 3210500 tid 3 2024-09-21T18:10:07.261 INFO:tasks.rados.rados.0.smithi160.stdout:3554: write (excl) oid 338 current snap is 0 2024-09-21T18:10:07.261 INFO:tasks.rados.rados.0.smithi160.stdout:3554: seq_num 1501 ranges {427307=670290,1601472=481522,2552971=483056} 2024-09-21T18:10:07.267 INFO:tasks.rados.rados.0.smithi160.stdout:3554: writing smithi16030472-338 from 427307 to 1097597 tid 1 2024-09-21T18:10:07.269 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:07.273 INFO:tasks.rados.rados.0.smithi160.stdout:3554: writing smithi16030472-338 from 1601472 to 2082994 tid 2 2024-09-21T18:10:07.278 INFO:tasks.rados.rados.0.smithi160.stdout:3554: writing smithi16030472-338 from 2552971 to 3036027 tid 3 2024-09-21T18:10:07.278 INFO:tasks.rados.rados.0.smithi160.stdout:3555: read oid 201 snap -1 2024-09-21T18:10:07.278 INFO:tasks.rados.rados.0.smithi160.stdout:3555: expect (ObjNum 1009 snap 0 seq_num 1009) 2024-09-21T18:10:07.278 INFO:tasks.rados.rados.0.smithi160.stdout:3556: read oid 66 snap -1 2024-09-21T18:10:07.279 INFO:tasks.rados.rados.0.smithi160.stdout:3556: expect (ObjNum 1365 snap 0 seq_num 1365) 2024-09-21T18:10:07.279 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:07.279 INFO:tasks.rados.rados.0.smithi160.stdout:3549: expect (ObjNum 1428 snap 0 seq_num 1428) 2024-09-21T18:10:07.279 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:07.296 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:07.296 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:07.302 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:07.302 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:07.302 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:10:07.302 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:07.315 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:10:07.349 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:07.345+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:07.364 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 785337 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:3545: got (0) Success 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 136 v 0 (ObjNum 525 snap 0 seq_num 525) dirty exists 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:3541: finishing copy_from racing read to smithi16030472-113 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:3541: finishing copy_from to smithi16030472-113 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 113 v 496 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:3544: finishing copy_from to smithi16030472-319 2024-09-21T18:10:07.381 INFO:tasks.rados.rados.0.smithi160.stdout:3544: got expected ENOENT (src dne) 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3544: finishing copy_from racing read to smithi16030472-319 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3541: done (15 left) 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3544: done (14 left) 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3545: done (13 left) 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3549: done (12 left) 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3557: write oid 307 current snap is 0 2024-09-21T18:10:07.382 INFO:tasks.rados.rados.0.smithi160.stdout:3557: seq_num 1502 ranges {444114=745539,1943601=625180} 2024-09-21T18:10:07.389 INFO:tasks.rados.rados.0.smithi160.stdout:3557: writing smithi16030472-307 from 444114 to 1189653 tid 1 2024-09-21T18:10:07.399 INFO:tasks.rados.rados.0.smithi160.stdout:3557: writing smithi16030472-307 from 1943601 to 2568781 tid 2 2024-09-21T18:10:07.399 INFO:tasks.rados.rados.0.smithi160.stdout:3558: cache_evict oid 208 snap -1 2024-09-21T18:10:07.399 INFO:tasks.rados.rados.0.smithi160.stdout:3559: cache_evict oid 408 snap -1 2024-09-21T18:10:07.400 INFO:tasks.rados.rados.0.smithi160.stdout:3560: write oid 83 current snap is 0 2024-09-21T18:10:07.400 INFO:tasks.rados.rados.0.smithi160.stdout:3560: seq_num 1503 ranges {460921=420788,1602083=485191,2491464=621953} 2024-09-21T18:10:07.400 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:07.400 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:07.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:07.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:07.405 INFO:tasks.rados.rados.0.smithi160.stdout:3560: writing smithi16030472-83 from 460921 to 881709 tid 1 2024-09-21T18:10:07.414 INFO:tasks.rados.rados.0.smithi160.stdout:3560: writing smithi16030472-83 from 1602083 to 2087274 tid 2 2024-09-21T18:10:07.421 INFO:tasks.rados.rados.0.smithi160.stdout:3560: writing smithi16030472-83 from 2491464 to 3113417 tid 3 2024-09-21T18:10:07.421 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:07.450 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:07.448+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:07.479 INFO:tasks.rados.rados.0.smithi160.stdout:3546: finishing write tid 1 to smithi16030472-99 2024-09-21T18:10:07.479 INFO:tasks.rados.rados.0.smithi160.stdout:3546: finishing write tid 2 to smithi16030472-99 2024-09-21T18:10:07.479 INFO:tasks.rados.rados.0.smithi160.stdout:3546: finishing write tid 3 to smithi16030472-99 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: finishing write tid 4 to smithi16030472-99 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: oid 99 updating version 0 to 1074 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: oid 99 version 1074 is already newer than 1073 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: oid 99 version 1074 is already newer than 1072 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 1074 (ObjNum 1497 snap 0 seq_num 1497) dirty exists 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: left oid 99 (ObjNum 1497 snap 0 seq_num 1497) 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3546: done (15 left) 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout:3561: cache_try_flush oid 287 snap -1 2024-09-21T18:10:07.480 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:07.483 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:07.483 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:07.483 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:07.491 INFO:tasks.rados.rados.0.smithi160.stdout:3560: finishing write tid 1 to smithi16030472-83 2024-09-21T18:10:07.491 INFO:tasks.rados.rados.0.smithi160.stdout:3560: finishing write tid 2 to smithi16030472-83 2024-09-21T18:10:07.493 INFO:tasks.rados.rados.0.smithi160.stdout:3560: finishing write tid 3 to smithi16030472-83 2024-09-21T18:10:07.497 INFO:tasks.rados.rados.0.smithi160.stdout:3557: finishing write tid 1 to smithi16030472-307 2024-09-21T18:10:07.499 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:07.503 INFO:tasks.rados.rados.0.smithi160.stdout:3557: finishing write tid 2 to smithi16030472-307 2024-09-21T18:10:07.544 INFO:tasks.rados.rados.0.smithi160.stdout:3542: expect (ObjNum 1150 snap 0 seq_num 1150) 2024-09-21T18:10:07.552 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:07.545+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:07.563 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3120283 2024-09-21T18:10:07.571 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 852837 2024-09-21T18:10:07.622 INFO:tasks.rados.rados.0.smithi160.stdout:3543: finishing write tid 1 to smithi16030472-76 2024-09-21T18:10:07.623 INFO:tasks.rados.rados.0.smithi160.stdout:3555: expect (ObjNum 1009 snap 0 seq_num 1009) 2024-09-21T18:10:07.630 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:07.630 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:07.630 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:07.630 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:10:07.630 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:07.644 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:07.645 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:07.689 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:07.684+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3558: got (16) Device or resource busy 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3543: finishing write tid 2 to smithi16030472-76 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3542: done (15 left) 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3555: done (14 left) 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3558: done (13 left) 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3562: write (excl) oid 26 current snap is 0 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3543: finishing write tid 3 to smithi16030472-76 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3543: finishing write tid 4 to smithi16030472-76 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3543: oid 76 updating version 0 to 1018 2024-09-21T18:10:07.710 INFO:tasks.rados.rados.0.smithi160.stdout:3543: oid 76 version 1018 is already newer than 1017 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:3543: oid 76 version 1018 is already newer than 1016 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 1018 (ObjNum 1496 snap 0 seq_num 1496) dirty exists 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:3543: left oid 76 (ObjNum 1496 snap 0 seq_num 1496) 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:3547: finishing write tid 1 to smithi16030472-301 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:3547: finishing write tid 2 to smithi16030472-301 2024-09-21T18:10:07.711 INFO:tasks.rados.rados.0.smithi160.stdout:3562: seq_num 1504 ranges {477728=496037,1660565=628849,2918887=358841} 2024-09-21T18:10:07.719 INFO:tasks.rados.rados.0.smithi160.stdout:3562: writing smithi16030472-26 from 477728 to 973765 tid 1 2024-09-21T18:10:07.730 INFO:tasks.rados.rados.0.smithi160.stdout:3562: writing smithi16030472-26 from 1660565 to 2289414 tid 2 2024-09-21T18:10:07.738 INFO:tasks.rados.rados.0.smithi160.stdout:3562: writing smithi16030472-26 from 2918887 to 3277728 tid 3 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3543: done (13 left) 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3563: delete oid 209 current snap is 0 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3560: finishing write tid 4 to smithi16030472-83 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: finishing write tid 3 to smithi16030472-307 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: finishing write tid 4 to smithi16030472-307 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: oid 307 updating version 0 to 1056 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: oid 307 version 1056 is already newer than 1054 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: oid 307 version 1056 is already newer than 1055 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 1056 (ObjNum 1502 snap 0 seq_num 1502) dirty exists 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3557: left oid 307 (ObjNum 1502 snap 0 seq_num 1502) 2024-09-21T18:10:07.739 INFO:tasks.rados.rados.0.smithi160.stdout:3556: expect (ObjNum 1365 snap 0 seq_num 1365) 2024-09-21T18:10:07.753 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:07.749+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:07.791 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1153939 2024-09-21T18:10:07.825 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 592796 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9359 4.1 4:ad7a0a8c:::smithi16030472-428:head [setxattr _header (57) in=64b,truncate 3210500] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541384+0000", 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28293685299999999, 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28297296999999999, 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.829 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9359 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541384+0000", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541384+0000", 2024-09-21T18:10:07.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541385+0000", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7000000000000005e-07 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541388+0000", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.163e-06 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541392+0000", 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9539999999999998e-06 2024-09-21T18:10:07.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541396+0000", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7400000000000002e-06 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.569991+0000", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.028594986999999999 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570065+0000", 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3857999999999997e-05 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570152+0000", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7138000000000005e-05 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667239+0000", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.097087584000000005 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9360 4.1 4:ad7a0a8c:::smithi16030472-428:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541400+0000", 2024-09-21T18:10:07.833 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28292076399999999, 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28303094899999998, 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9360 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541400+0000", 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541400+0000", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541401+0000", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6800000000000001e-07 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541403+0000", 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8029999999999999e-06 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541408+0000", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0440000000000003e-06 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541412+0000", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9230000000000004e-06 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570538+0000", 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029126446 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570603+0000", 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4784999999999997e-05 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9361 4.1 4:b3a70d5e:::smithi16030472-338:head [write 427307~670290 in=670290b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541417+0000", 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28290380900000001, 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.283069285, 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.837 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9361 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541417+0000", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541417+0000", 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541418+0000", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9900000000000001e-07 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541606+0000", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00018783 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541612+0000", 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7349999999999999e-06 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541617+0000", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7199999999999997e-06 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570631+0000", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.029013988000000001 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570698+0000", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6750999999999998e-05 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.570801+0000", 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000103502 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667246+0000", 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.096444674999999994 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9362 4.1 4:b3a70d5e:::smithi16030472-338:head [write 1601472~481522 in=481522b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541623+0000", 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28269807899999999, 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28292811699999998, 2024-09-21T18:10:07.841 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9362 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541623+0000", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541623+0000", 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541624+0000", 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0699999999999997e-07 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541886+0000", 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026187399999999999 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541892+0000", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.528e-06 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541896+0000", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7799999999999998e-06 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.571997+0000", 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.03010146 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.572078+0000", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1170999999999999e-05 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.572175+0000", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.6706000000000005e-05 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667253+0000", 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.095077514000000002 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.845 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9363 4.1 4:b3a70d5e:::smithi16030472-338:head [write 2552971~483056 in=483056b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541902+0000", 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28241942799999997, 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.282709286, 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9363 2024-09-21T18:10:07.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541902+0000", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541902+0000", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541902+0000", 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9800000000000004e-07 2024-09-21T18:10:07.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542207+0000", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030458599999999997 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542215+0000", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2870000000000004e-06 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542220+0000", 2024-09-21T18:10:07.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8119999999999998e-06 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.573642+0000", 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.031421530000000003 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.573727+0000", 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5594999999999996e-05 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.573817+0000", 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9905000000000001e-05 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667259+0000", 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.093441922999999996 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9364 4.1 4:b3a70d5e:::smithi16030472-338:head [setxattr _header (57) in=64b,truncate 3227307] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.542228+0000", 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.282093498, 2024-09-21T18:10:07.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28244468499999997, 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9364 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542228+0000", 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542228+0000", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542228+0000", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3399999999999999e-07 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542231+0000", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.807e-06 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542235+0000", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542239+0000", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5750000000000001e-06 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.574995+0000", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.032756368000000001 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.575089+0000", 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3472000000000002e-05 2024-09-21T18:10:07.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.575178+0000", 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9608e-05 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667265+0000", 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.092087143999999996 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9365 4.1 4:b3a70d5e:::smithi16030472-338:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.542243+0000", 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28207791799999998, 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28249053499999999, 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9365 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542243+0000", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542243+0000", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542244+0000", 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7399999999999999e-07 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542245+0000", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7209999999999999e-06 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542249+0000", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8430000000000003e-06 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542253+0000", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.4860000000000002e-06 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.575582+0000", 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.033329507000000001 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.575653+0000", 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0477999999999998e-05 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9341 4.1 4:84cadebc:::smithi16030472-301:head [setxattr _header (57) in=64b,truncate 3176886] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.218741+0000", 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.605579957, 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.60607169000000005, 2024-09-21T18:10:07.858 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9341 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218741+0000", 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218741+0000", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218742+0000", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0809999999999999e-06 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218748+0000", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5260000000000001e-06 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218757+0000", 2024-09-21T18:10:07.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2539999999999998e-06 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218764+0000", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5470000000000002e-06 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.466437+0000", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.247672858 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.466685+0000", 2024-09-21T18:10:07.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024821799999999997 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.467034+0000", 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000349274 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.556986+0000", 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.089951960999999997 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9377 4.3 4:dac2a3d5:::smithi16030472-408:head [cache-evict] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.542421+0000", 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28190018500000003, 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28245619100000002, 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9377 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542421+0000", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542421+0000", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542421+0000", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.72e-07 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542423+0000", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.629e-06 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542427+0000", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6260000000000001e-06 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.542430+0000", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6200000000000001e-06 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.646392+0000", 2024-09-21T18:10:07.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.103961395 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.646630+0000", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023861100000000001 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.646780+0000", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014949200000000001 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.648497+0000", 2024-09-21T18:10:07.866 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0017172330000000001 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9342 4.1 4:84cadebc:::smithi16030472-301:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.218774+0000", 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.60554681600000004, 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.60616625700000004, 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.867 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9342 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218774+0000", 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218774+0000", 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218775+0000", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218778+0000", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.1939999999999998e-06 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218787+0000", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7649999999999999e-06 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.218793+0000", 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7339999999999996e-06 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.557011+0000", 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33821822200000001 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.557074+0000", 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2509999999999996e-05 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.870 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:1051 3.2 3:7c14251d:::smithi16030472-108:head [delete] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.543867+0000", 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28045385299999998, 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28112837699999998, 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543867+0000", 2024-09-21T18:10:07.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543867+0000", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543870+0000", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.199e-06 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543876+0000", 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0569999999999998e-06 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543887+0000", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1182999999999999e-05 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543896+0000", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7280000000000001e-06 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.644102+0000", 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10020657099999999 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.644343+0000", 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024044499999999999 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9343 4.1 4:837e1005:::smithi16030472-224:head [write 793693~519792 in=519792b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.227665+0000", 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.59665589299999999, 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.59737986300000001, 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.874 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9343 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.227665+0000", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.227665+0000", 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.227667+0000", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.046e-06 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.228029+0000", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00036180899999999999 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.228049+0000", 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9775e-05 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.228060+0000", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1433e-05 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.557096+0000", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32903592199999998 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.557201+0000", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000104924 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.557323+0000", 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012139 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.561657+0000", 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0043343690000000002 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:1052 3.2 3:634afb93:::smithi16030472-488:head [copy-from ver 986 in=56b] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.543925+0000", 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.280396113, 2024-09-21T18:10:07.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28120165000000003, 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543925+0000", 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543925+0000", 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543926+0000", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.0800000000000004e-07 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543930+0000", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1690000000000002e-06 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543942+0000", 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1667e-05 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543948+0000", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3910000000000003e-06 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.745316+0000", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20136827800000001 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.745404+0000", 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7604999999999999e-05 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.750687+0000", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0052825959999999996 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.758755+0000", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080683290000000008 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9344 4.1 4:837e1005:::smithi16030472-224:head [write 1884508~477853 in=477853b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.234301+0000", 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.590020557, 2024-09-21T18:10:07.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.59090470500000003, 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9344 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234301+0000", 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234301+0000", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234303+0000", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7219999999999999e-06 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234473+0000", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016952800000000001 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234495+0000", 2024-09-21T18:10:07.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2234000000000001e-05 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.234508+0000", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2893e-05 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.558891+0000", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32438263499999997 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.559001+0000", 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011024899999999999 2024-09-21T18:10:07.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.559105+0000", 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010357999999999999 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562259+0000", 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0031545929999999998 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.15.23:1053 3.2 3:4b5e6b7f:::smithi16030472-287:head [delete] snapc 0=[] ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.543957+0000", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.280364528, 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28131678999999998, 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543957+0000", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543957+0000", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543957+0000", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7700000000000004e-07 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543960+0000", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.8270000000000002e-06 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543967+0000", 2024-09-21T18:10:07.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.5200000000000003e-06 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.543973+0000", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7179999999999999e-06 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.745571+0000", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20159777000000001 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.745718+0000", 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014652999999999999 2024-09-21T18:10:07.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.745834+0000", 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000116202 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.747134+0000", 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001299955 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.890 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9345 4.1 4:837e1005:::smithi16030472-224:head [write 2898125~295568 in=295568b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.239409+0000", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.58491252400000004, 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.58592657699999995, 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9345 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239409+0000", 2024-09-21T18:10:07.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239409+0000", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239411+0000", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6460000000000002e-06 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239585+0000", 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000173819 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239607+0000", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2238999999999998e-05 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239619+0000", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2002e-05 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.560374+0000", 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.32075503999999999 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.560517+0000", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014263899999999999 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.560609+0000", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1452999999999993e-05 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562360+0000", 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001751636 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.894 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9346 4.1 4:837e1005:::smithi16030472-224:head [setxattr _header (57) in=64b,truncate 3193693] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.239636+0000", 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.58468477299999999, 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.58576841300000004, 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9346 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.895 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239636+0000", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239636+0000", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239637+0000", 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000002e-07 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239641+0000", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8229999999999997e-06 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.239647+0000", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1809999999999998e-06 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339756+0000", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10010902200000001 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.561716+0000", 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.221959246 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.561792+0000", 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6586000000000002e-05 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.561878+0000", 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6213999999999998e-05 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.565839+0000", 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003960374 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:9341 4.1 e121/33)", 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.768195+0000", 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.056126611999999999, 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057286761999999998, 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768195+0000", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768195+0000", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768195+0000", 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8800000000000002e-07 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.900 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768198+0000", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.14e-06 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768205+0000", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9090000000000003e-06 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768211+0000", 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9279999999999998e-06 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:9343 4.1 e121/33)", 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.768219+0000", 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.056102094999999998, 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057298443999999997, 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768219+0000", 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768219+0000", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768220+0000", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6499999999999999e-07 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768223+0000", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.976e-06 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768229+0000", 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8029999999999999e-06 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768235+0000", 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3179999999999997e-06 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:9344 4.1 e121/33)", 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.768254+0000", 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.056067725999999998, 2024-09-21T18:10:07.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057300243000000001, 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768254+0000", 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768254+0000", 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.905 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768254+0000", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.9800000000000003e-07 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768256+0000", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8819999999999999e-06 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768262+0000", 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5990000000000001e-06 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768267+0000", 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0560000000000004e-06 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9347 4.1 4:837e1005:::smithi16030472-224:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.339798+0000", 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.48452335899999999, 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.485796389, 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.907 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9347 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339798+0000", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339798+0000", 2024-09-21T18:10:07.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339801+0000", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5899999999999999e-06 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339815+0000", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3030000000000001e-05 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.339836+0000", 2024-09-21T18:10:07.909 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1046000000000001e-05 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.439980+0000", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100144075 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562287+0000", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12230757 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562352+0000", 2024-09-21T18:10:07.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4986000000000004e-05 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:9345 4.1 e121/33)", 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.768272+0000", 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.056049403999999997, 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.057373358999999999, 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768272+0000", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768272+0000", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768272+0000", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2499999999999999e-07 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768274+0000", 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1919999999999999e-06 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.768280+0000", 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3380000000000004e-06 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9352 4.3 4:cafbb15d:::smithi16030472-232:head [sparse-read 0~3159947] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.440111+0000", 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38421071499999998, 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38557169299999999, 2024-09-21T18:10:07.913 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9352 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440111+0000", 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440111+0000", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440112+0000", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3230000000000001e-06 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440117+0000", 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.9529999999999998e-06 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440128+0000", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1654000000000001e-05 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440137+0000", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0930000000000003e-06 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.441039+0000", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00090263099999999998 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:07.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.441105+0000", 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6279999999999996e-05 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.749743+0000", 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30863780099999999 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.749915+0000", 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017144299999999999 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9353 4.3 4:cafbb15d:::smithi16030472-232:head [sparse-read 1466422~1693525] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.440196+0000", 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38412489599999999, 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38555076100000002, 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9353 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.918 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440196+0000", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440196+0000", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440197+0000", 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440201+0000", 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0740000000000003e-06 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440210+0000", 2024-09-21T18:10:07.921 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7800000000000006e-06 2024-09-21T18:10:07.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440218+0000", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0360000000000006e-06 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.441186+0000", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00096722599999999996 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.441309+0000", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000123819 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.749975+0000", 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.308665829 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.750116+0000", 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014076799999999999 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9356 4.1 4:ad7a0a8c:::smithi16030472-428:head [write 410500~595041 in=595041b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.440229+0000", 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.38409178300000002, 2024-09-21T18:10:07.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.38558259900000003, 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9356 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440229+0000", 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440229+0000", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440230+0000", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8299999999999995e-07 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440485+0000", 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025472800000000002 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.440499+0000", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3923999999999999e-05 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.540641+0000", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100141985 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562381+0000", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.021739745000000001 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562442+0000", 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.1518000000000003e-05 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.562547+0000", 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010507599999999999 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667203+0000", 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.104655846 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9357 4.1 4:ad7a0a8c:::smithi16030472-428:head [write 1542990~621511 in=621511b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.540682+0000", 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28363919999999998, 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28519466700000001, 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9357 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.540682+0000", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.540682+0000", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.540685+0000", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.286e-06 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541160+0000", 2024-09-21T18:10:07.929 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047434400000000002 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541177+0000", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.766e-05 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541185+0000", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.3529999999999997e-06 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.564235+0000", 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.023049907000000001 2024-09-21T18:10:07.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.564359+0000", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012430699999999999 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.564454+0000", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4732e-05 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667224+0000", 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102770103 2024-09-21T18:10:07.931 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9358 4.1 4:ad7a0a8c:::smithi16030472-428:head [write 2925548~284952 in=284952b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:07.541196+0000", 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.28312534099999997, 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.28478778300000002, 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9358 2024-09-21T18:10:07.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541196+0000", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541196+0000", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541197+0000", 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2399999999999997e-07 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541366+0000", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016903400000000001 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541373+0000", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6559999999999992e-06 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.541378+0000", 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2629999999999997e-06 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.568780+0000", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.027402029000000001 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.568876+0000", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5925000000000001e-05 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.568984+0000", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010836300000000001 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:07.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:07.667231+0000", 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098247078000000002 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 27 2024-09-21T18:10:07.936 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:07.937 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: finishing write tid 5 to smithi16030472-83 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: oid 83 updating version 0 to 1076 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: oid 83 updating version 1076 to 1078 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: oid 83 version 1078 is already newer than 1077 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: oid 83 version 1078 is already newer than 1075 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 83 v 1078 (ObjNum 1503 snap 0 seq_num 1503) dirty exists 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3560: left oid 83 (ObjNum 1503 snap 0 seq_num 1503) 2024-09-21T18:10:07.938 INFO:tasks.rados.rados.0.smithi160.stdout:3547: finishing write tid 3 to smithi16030472-301 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: finishing write tid 1 to smithi16030472-26 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: finishing write tid 2 to smithi16030472-26 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: finishing write tid 3 to smithi16030472-26 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: finishing write tid 4 to smithi16030472-26 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: finishing write tid 5 to smithi16030472-26 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: oid 26 updating version 0 to 1057 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: oid 26 updating version 1057 to 1060 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: oid 26 version 1060 is already newer than 1058 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: oid 26 version 1060 is already newer than 1059 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 1060 (ObjNum 1504 snap 0 seq_num 1504) dirty exists 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3562: left oid 26 (ObjNum 1504 snap 0 seq_num 1504) 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3559: got (0) Success 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3561: got (0) Success 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 287 v 0 (ObjNum 49 snap 0 seq_num 3111024432) clean dne 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:3551: got (0) Success 2024-09-21T18:10:07.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 108 v 0 (ObjNum 0 snap 0 seq_num 304) clean dne 2024-09-21T18:10:07.940 INFO:tasks.rados.rados.0.smithi160.stdout:3550: expect (ObjNum 1021 snap 0 seq_num 1021) 2024-09-21T18:10:07.940 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:07.936+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: finishing write tid 4 to smithi16030472-301 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: finishing write tid 5 to smithi16030472-301 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: oid 301 updating version 0 to 1019 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: oid 301 updating version 1019 to 1021 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: oid 301 version 1021 is already newer than 1020 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:3547: oid 301 updating version 1021 to 1022 2024-09-21T18:10:07.994 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 301 v 1022 (ObjNum 1498 snap 0 seq_num 1498) dirty exists 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3547: left oid 301 (ObjNum 1498 snap 0 seq_num 1498) 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: finishing write tid 1 to smithi16030472-224 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3552: got (0) Success 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 488 v 0 (ObjNum 1376 snap 0 seq_num 1376) dirty exists 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: finishing write tid 2 to smithi16030472-224 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: finishing write tid 3 to smithi16030472-224 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: finishing write tid 4 to smithi16030472-224 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: finishing write tid 5 to smithi16030472-224 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: oid 224 updating version 0 to 1024 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: oid 224 updating version 1024 to 1025 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: oid 224 updating version 1025 to 1026 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: oid 224 version 1026 is already newer than 1023 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 224 v 1026 (ObjNum 1499 snap 0 seq_num 1499) dirty exists 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3548: left oid 224 (ObjNum 1499 snap 0 seq_num 1499) 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3553: finishing write tid 1 to smithi16030472-428 2024-09-21T18:10:07.995 INFO:tasks.rados.rados.0.smithi160.stdout:3553: finishing write tid 2 to smithi16030472-428 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3547: done (13 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3548: done (12 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3550: done (11 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3551: done (10 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3552: done (9 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3556: done (8 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3557: done (7 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3559: done (6 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3560: done (5 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3561: done (4 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3562: done (3 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3563: done (2 left) 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3564: write oid 442 current snap is 0 2024-09-21T18:10:07.996 INFO:tasks.rados.rados.0.smithi160.stdout:3564: seq_num 1506 ranges {511342=762888,1777529=632518,2973733=337609} 2024-09-21T18:10:08.005 INFO:tasks.rados.rados.0.smithi160.stdout:3564: writing smithi16030472-442 from 511342 to 1274230 tid 1 2024-09-21T18:10:08.013 INFO:tasks.rados.rados.0.smithi160.stdout:3564: writing smithi16030472-442 from 1777529 to 2410047 tid 2 2024-09-21T18:10:08.020 INFO:tasks.rados.rados.0.smithi160.stdout:3564: writing smithi16030472-442 from 2973733 to 3311342 tid 3 2024-09-21T18:10:08.020 INFO:tasks.rados.rados.0.smithi160.stdout:3565: read oid 66 snap -1 2024-09-21T18:10:08.020 INFO:tasks.rados.rados.0.smithi160.stdout:3565: expect (ObjNum 1365 snap 0 seq_num 1365) 2024-09-21T18:10:08.020 INFO:tasks.rados.rados.0.smithi160.stdout:3566: write (excl) oid 480 current snap is 0 2024-09-21T18:10:08.020 INFO:tasks.rados.rados.0.smithi160.stdout:3566: seq_num 1507 ranges {528149=438137,1436011=492529,2601156=667041} 2024-09-21T18:10:08.027 INFO:tasks.rados.rados.0.smithi160.stdout:3566: writing smithi16030472-480 from 528149 to 966286 tid 1 2024-09-21T18:10:08.034 INFO:tasks.rados.rados.0.smithi160.stdout:3566: writing smithi16030472-480 from 1436011 to 1928540 tid 2 2024-09-21T18:10:08.038 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:08.033+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:08.044 INFO:tasks.rados.rados.0.smithi160.stdout:3566: writing smithi16030472-480 from 2601156 to 3268197 tid 3 2024-09-21T18:10:08.044 INFO:tasks.rados.rados.0.smithi160.stdout:3567: cache_try_flush oid 253 snap -1 2024-09-21T18:10:08.044 INFO:tasks.rados.rados.0.smithi160.stdout:3568: cache_flush oid 352 snap -1 2024-09-21T18:10:08.044 INFO:tasks.rados.rados.0.smithi160.stdout:3569: delete oid 363 current snap is 0 2024-09-21T18:10:08.049 INFO:tasks.rados.rados.0.smithi160.stdout:3569: done (7 left) 2024-09-21T18:10:08.049 INFO:tasks.rados.rados.0.smithi160.stdout:3570: write oid 165 current snap is 0 2024-09-21T18:10:08.049 INFO:tasks.rados.rados.0.smithi160.stdout:3570: seq_num 1509 ranges {561763=588635,1552975=496198,2656002=405938} 2024-09-21T18:10:08.057 INFO:tasks.rados.rados.0.smithi160.stdout:3570: writing smithi16030472-165 from 561763 to 1150398 tid 1 2024-09-21T18:10:08.063 INFO:tasks.rados.rados.0.smithi160.stdout:3570: writing smithi16030472-165 from 1552975 to 2049173 tid 2 2024-09-21T18:10:08.071 INFO:tasks.rados.rados.0.smithi160.stdout:3570: writing smithi16030472-165 from 2656002 to 3061940 tid 3 2024-09-21T18:10:08.071 INFO:tasks.rados.rados.0.smithi160.stdout:3571: cache_flush oid 467 snap -1 2024-09-21T18:10:08.071 INFO:tasks.rados.rados.0.smithi160.stdout:3572: read oid 186 snap -1 2024-09-21T18:10:08.072 INFO:tasks.rados.rados.0.smithi160.stdout:3572: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:10:08.072 INFO:tasks.rados.rados.0.smithi160.stdout:3573: cache_flush oid 84 snap -1 2024-09-21T18:10:08.072 INFO:tasks.rados.rados.0.smithi160.stdout:3574: write oid 413 current snap is 0 2024-09-21T18:10:08.072 INFO:tasks.rados.rados.0.smithi160.stdout:3574: seq_num 1510 ranges {578570=663884,1895104=639856,2967072=411498} 2024-09-21T18:10:08.080 INFO:tasks.rados.rados.0.smithi160.stdout:3574: writing smithi16030472-413 from 578570 to 1242454 tid 1 2024-09-21T18:10:08.089 INFO:tasks.rados.rados.0.smithi160.stdout:3574: writing smithi16030472-413 from 1895104 to 2534960 tid 2 2024-09-21T18:10:08.094 INFO:tasks.rados.rados.0.smithi160.stdout:3574: writing smithi16030472-413 from 2967072 to 3378570 tid 3 2024-09-21T18:10:08.094 INFO:tasks.rados.rados.0.smithi160.stdout:3575: cache_try_flush oid 97 snap -1 2024-09-21T18:10:08.094 INFO:tasks.rados.rados.0.smithi160.stdout:3576: write oid 356 current snap is 0 2024-09-21T18:10:08.094 INFO:tasks.rados.rados.0.smithi160.stdout:3576: seq_num 1511 ranges {595377=739133,1953586=499867,3110848=284529} 2024-09-21T18:10:08.102 INFO:tasks.rados.rados.0.smithi160.stdout:3576: writing smithi16030472-356 from 595377 to 1334510 tid 1 2024-09-21T18:10:08.103 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:08.104 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:08.104 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:08.104 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:08.104 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:08.108 INFO:tasks.rados.rados.0.smithi160.stdout:3576: writing smithi16030472-356 from 1953586 to 2453453 tid 2 2024-09-21T18:10:08.114 INFO:tasks.rados.rados.0.smithi160.stdout:3576: writing smithi16030472-356 from 3110848 to 3395377 tid 3 2024-09-21T18:10:08.114 INFO:tasks.rados.rados.0.smithi160.stdout:3577: cache_evict oid 342 snap -1 2024-09-21T18:10:08.114 INFO:tasks.rados.rados.0.smithi160.stdout:3578: write (excl) oid 379 current snap is 0 2024-09-21T18:10:08.114 INFO:tasks.rados.rados.0.smithi160.stdout:3578: seq_num 1512 ranges {612184=414382,1612068=643525,3021918=390266} 2024-09-21T18:10:08.119 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:08.120 INFO:tasks.rados.rados.0.smithi160.stdout:3578: writing smithi16030472-379 from 612184 to 1026566 tid 1 2024-09-21T18:10:08.127 INFO:tasks.rados.rados.0.smithi160.stdout:3578: writing smithi16030472-379 from 1612068 to 2255593 tid 2 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3578: writing smithi16030472-379 from 3021918 to 3412184 tid 3 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3577: got (0) Success 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3577: done (15 left) 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3579: copy_from oid 275 from oid 486 current snap is 0 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3578: finishing write tid 1 to smithi16030472-379 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3553: finishing write tid 3 to smithi16030472-428 2024-09-21T18:10:08.134 INFO:tasks.rados.rados.0.smithi160.stdout:3553: finishing write tid 4 to smithi16030472-428 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: finishing write tid 5 to smithi16030472-428 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: oid 428 updating version 0 to 1027 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: oid 428 updating version 1027 to 1029 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: oid 428 updating version 1029 to 1030 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: oid 428 version 1030 is already newer than 1028 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 428 v 1030 (ObjNum 1500 snap 0 seq_num 1500) dirty exists 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: left oid 428 (ObjNum 1500 snap 0 seq_num 1500) 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3554: finishing write tid 1 to smithi16030472-338 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3554: finishing write tid 2 to smithi16030472-338 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3578: finishing write tid 2 to smithi16030472-379 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3553: done (15 left) 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3580: write oid 419 current snap is 0 2024-09-21T18:10:08.135 INFO:tasks.rados.rados.0.smithi160.stdout:3580: seq_num 1513 ranges {628991=605984,1786903=787183,3165694=263297} 2024-09-21T18:10:08.138 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:08.136+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:08.141 INFO:tasks.rados.rados.0.smithi160.stdout:3580: writing smithi16030472-419 from 628991 to 1234975 tid 1 2024-09-21T18:10:08.148 INFO:tasks.rados.rados.0.smithi160.stdout:3580: writing smithi16030472-419 from 1786903 to 2574086 tid 2 2024-09-21T18:10:08.153 INFO:tasks.rados.rados.0.smithi160.stdout:3580: writing smithi16030472-419 from 3165694 to 3428991 tid 3 2024-09-21T18:10:08.153 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.234 INFO:tasks.rados.rados.0.smithi160.stdout:3578: finishing write tid 3 to smithi16030472-379 2024-09-21T18:10:08.240 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:08.237+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:08.246 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:08.246 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:08.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:08.246 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:08.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 175.097819945, 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:10:08.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:10:08.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:10:08.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:10:08.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 175.097773636, 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.253 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:10:08.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:10:08.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:10:08.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:10:08.258 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 175.09567772299999, 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:10:08.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:10:08.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:10:08.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:10:08.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:10:08.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:10:08.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 175.09501460800001, 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.265 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:10:08.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:10:08.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:10:08.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 175.09497810900001, 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:10:08.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:10:08.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 174.96133993699999, 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.276 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:10:08.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 174.960628146, 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:10:08.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:10:08.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:08.287 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 174.960590583, 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.163204584, 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:10:08.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:10:08.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.296 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.16301003800001, 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:08.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:10:08.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:10:08.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.300 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.16272365, 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.16270757000001, 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:10:08.304 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:10:08.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.307 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.13378031799999, 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.310 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.127561243, 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:10:08.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.315 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.12245964799999, 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:10:08.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.1221257, 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:10:08.319 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:10:08.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:10:08.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.122105221, 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.323 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:08.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:10:08.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:10:08.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.03577669399999, 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:10:08.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.02731650800001, 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:10:08.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.02138880300001, 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:10:08.334 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.335 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.338 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: finishing write tid 3 to smithi16030472-338 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: finishing write tid 4 to smithi16030472-338 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: finishing write tid 5 to smithi16030472-338 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: oid 338 updating version 0 to 1032 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: oid 338 updating version 1032 to 1034 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: oid 338 version 1034 is already newer than 1031 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: oid 338 version 1034 is already newer than 1033 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 338 v 1034 (ObjNum 1501 snap 0 seq_num 1501) dirty exists 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: left oid 338 (ObjNum 1501 snap 0 seq_num 1501) 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3554: done (15 left) 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3581: write oid 316 current snap is 0 2024-09-21T18:10:08.339 INFO:tasks.rados.rados.0.smithi160.stdout:3581: seq_num 1514 ranges {645798=681233,1729032=647194,3076764=369034} 2024-09-21T18:10:08.340 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout:3581: writing smithi16030472-316 from 645798 to 1327031 tid 1 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout:3581: writing smithi16030472-316 from 1729032 to 2376226 tid 2 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout:3581: writing smithi16030472-316 from 3076764 to 3445798 tid 3 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout:3565: expect (ObjNum 1365 snap 0 seq_num 1365) 2024-09-21T18:10:08.341 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:08.342 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:08.341+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:08.396 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1825510 2024-09-21T18:10:08.413 INFO:tasks.rados.rados.0.smithi160.stdout:3568: got (0) Success 2024-09-21T18:10:08.413 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 352 v 0 (ObjNum 129 snap 0 seq_num 3127838216) clean dne 2024-09-21T18:10:08.414 INFO:tasks.rados.rados.0.smithi160.stdout:3565: done (15 left) 2024-09-21T18:10:08.414 INFO:tasks.rados.rados.0.smithi160.stdout:3568: done (14 left) 2024-09-21T18:10:08.414 INFO:tasks.rados.rados.0.smithi160.stdout:3582: write (excl) oid 294 current snap is 0 2024-09-21T18:10:08.414 INFO:tasks.rados.rados.0.smithi160.stdout:3582: seq_num 1515 ranges {662605=756482,2187514=790852} 2024-09-21T18:10:08.421 INFO:tasks.rados.rados.0.smithi160.stdout:3582: writing smithi16030472-294 from 662605 to 1419087 tid 1 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout:3582: writing smithi16030472-294 from 2187514 to 2978366 tid 2 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout:3583: cache_evict oid 19 snap -1 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout:3578: finishing write tid 4 to smithi16030472-379 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout:3564: finishing write tid 1 to smithi16030472-442 2024-09-21T18:10:08.431 INFO:tasks.rados.rados.0.smithi160.stdout:3564: finishing write tid 2 to smithi16030472-442 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: finishing write tid 5 to smithi16030472-379 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: oid 379 updating version 0 to 1082 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: oid 379 version 1082 is already newer than 1080 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: oid 379 version 1082 is already newer than 1081 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: oid 379 version 1082 is already newer than 1079 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 379 v 1082 (ObjNum 1512 snap 0 seq_num 1512) dirty exists 2024-09-21T18:10:08.437 INFO:tasks.rados.rados.0.smithi160.stdout:3578: left oid 379 (ObjNum 1512 snap 0 seq_num 1512) 2024-09-21T18:10:08.438 INFO:tasks.rados.rados.0.smithi160.stdout:3578: done (15 left) 2024-09-21T18:10:08.438 INFO:tasks.rados.rados.0.smithi160.stdout:3584: copy_from oid 197 from oid 152 current snap is 0 2024-09-21T18:10:08.438 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.443 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:08.440+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3574: finishing write tid 1 to smithi16030472-413 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3574: finishing write tid 2 to smithi16030472-413 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3574: finishing write tid 3 to smithi16030472-413 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3567: got (0) Success 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 253 v 0 (ObjNum -1166666496 snap 21993 seq_num 304) clean dne 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3564: finishing write tid 3 to smithi16030472-442 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3567: done (15 left) 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3585: write oid 429 current snap is 0 2024-09-21T18:10:08.468 INFO:tasks.rados.rados.0.smithi160.stdout:3585: seq_num 1516 ranges {679412=431731,1845996=650863,3131610=347802} 2024-09-21T18:10:08.479 INFO:tasks.rados.rados.0.smithi160.stdout:3585: writing smithi16030472-429 from 679412 to 1111143 tid 1 2024-09-21T18:10:08.492 INFO:tasks.rados.rados.0.smithi160.stdout:3585: writing smithi16030472-429 from 1845996 to 2496859 tid 2 2024-09-21T18:10:08.502 INFO:tasks.rados.rados.0.smithi160.stdout:3585: writing smithi16030472-429 from 3131610 to 3479412 tid 3 2024-09-21T18:10:08.502 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.505 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:08.505 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:08.505 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.505 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9406 3.3 3:c3d868ee:::smithi16030472-165:head [write 561763~588635 in=588635b] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.505 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.483398+0000", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.020417523999999999, 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020481632999999999, 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483398+0000", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483398+0000", 2024-09-21T18:10:08.506 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483401+0000", 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6459999999999999e-06 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483772+0000", 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037063900000000003 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.507 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483793+0000", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0596000000000001e-05 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483805+0000", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2446000000000001e-05 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483959+0000", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000153871 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.484153+0000", 2024-09-21T18:10:08.508 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019423899999999999 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 2", 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.484383+0000", 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000230008 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.487873+0000", 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034898360000000001 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.509 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:2829 3.3 3:c3d868ee:::smithi16030472-165:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.483837+0000", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019978151999999999, 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020155013999999999, 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483837+0000", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.510 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483837+0000", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483838+0000", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7199999999999997e-07 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483842+0000", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0589999999999996e-06 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.511 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483852+0000", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0344e-05 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483860+0000", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9349999999999994e-06 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.487222+0000", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0033617 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.512 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.487360+0000", 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013836600000000001 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(osd.11.22:2830 3.3 3:c3d868ee:::smithi16030472-165:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.483935+0000", 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.019880482000000001, 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.020146265999999999, 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:08.513 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483935+0000", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483935+0000", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483936+0000", 2024-09-21T18:10:08.514 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9699999999999994e-07 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483941+0000", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8350000000000003e-06 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483949+0000", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4419999999999998e-06 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.483959+0000", 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0127e-05 2024-09-21T18:10:08.515 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.487453+0000", 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0034941740000000001 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.487668+0000", 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000214742 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.516 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_repop_reply(client.4876.0:9406 3.3 e121/31)", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.488802+0000", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.015013238999999999, 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.015354173, 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "no flag points reached", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.488802+0000", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.517 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.488802+0000", 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.488804+0000", 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8980000000000001e-06 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.488810+0000", 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4699999999999999e-06 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.518 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.488830+0000", 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9653e-05 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9445 4.2 4.e14b9e42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.421659+0000", 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.082156385999999998, 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.082557606000000006, 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.519 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9445 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.421659+0000", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.421659+0000", 2024-09-21T18:10:08.520 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.421662+0000", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.96e-06 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.422081+0000", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000419648 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.422103+0000", 2024-09-21T18:10:08.521 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1458e-05 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.422115+0000", 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2588999999999999e-05 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9446 4.2 4.e14b9e42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.430702+0000", 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.073112707999999998, 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.073590638, 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.522 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9446 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.430702+0000", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.523 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.430702+0000", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.430705+0000", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0619999999999999e-06 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431086+0000", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038143600000000003 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431100+0000", 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3740000000000001e-05 2024-09-21T18:10:08.524 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431108+0000", 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6969999999999999e-06 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9447 4.2 4.e14b9e42 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.431123+0000", 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.072692039, 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.073239884000000005, 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.525 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9447 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431123+0000", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431123+0000", 2024-09-21T18:10:08.526 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431124+0000", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.3e-07 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431127+0000", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9059999999999998e-06 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431131+0000", 2024-09-21T18:10:08.527 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8750000000000002e-06 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431134+0000", 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0769999999999999e-06 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:9448 4.2 4.e14b9e42 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:10:08.431138+0000", 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 0.072677021999999994, 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.073295360000000004, 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.528 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 9448 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431138+0000", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431138+0000", 2024-09-21T18:10:08.529 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431139+0000", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5600000000000001e-07 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431140+0000", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7150000000000001e-06 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431146+0000", 2024-09-21T18:10:08.530 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4009999999999996e-06 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:10:08.431149+0000", 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3179999999999998e-06 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: ], 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 8 2024-09-21T18:10:08.531 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:08.532 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:08.573 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:08.568+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: finishing write tid 4 to smithi16030472-413 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: finishing write tid 5 to smithi16030472-413 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: oid 413 updating version 0 to 1035 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: oid 413 updating version 1035 to 1036 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: oid 413 updating version 1036 to 1037 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: oid 413 updating version 1037 to 1038 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 413 v 1038 (ObjNum 1510 snap 0 seq_num 1510) dirty exists 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3574: left oid 413 (ObjNum 1510 snap 0 seq_num 1510) 2024-09-21T18:10:08.576 INFO:tasks.rados.rados.0.smithi160.stdout:3583: got (0) Success 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout:3574: done (15 left) 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout:3583: done (14 left) 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout:3586: read oid 452 snap -1 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout:3586: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout:3587: copy_from oid 381 from oid 395 current snap is 0 2024-09-21T18:10:08.577 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:08.644 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:08.636+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:08.680 INFO:tasks.rados.rados.0.smithi160.stdout:3572: expect (ObjNum 185 snap 0 seq_num 185) 2024-09-21T18:10:08.695 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:08.695 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:08.695 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:08.695 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:08.696 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:08.710 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:08.746 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:08.740+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:3573: got (0) Success 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 84 v 0 (ObjNum 1217 snap 0 seq_num 1217) dirty exists 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:3582: finishing write tid 1 to smithi16030472-294 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:3582: finishing write tid 2 to smithi16030472-294 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:3587: finishing copy_from racing read to smithi16030472-381 2024-09-21T18:10:08.831 INFO:tasks.rados.rados.0.smithi160.stdout:3564: finishing write tid 4 to smithi16030472-442 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: finishing write tid 5 to smithi16030472-442 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: oid 442 updating version 0 to 989 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: oid 442 version 989 is already newer than 988 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: oid 442 updating version 989 to 990 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: oid 442 version 990 is already newer than 987 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 990 (ObjNum 1506 snap 0 seq_num 1506) dirty exists 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: left oid 442 (ObjNum 1506 snap 0 seq_num 1506) 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3566: finishing write tid 1 to smithi16030472-480 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3566: finishing write tid 2 to smithi16030472-480 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3570: finishing write tid 1 to smithi16030472-165 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3564: done (15 left) 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3572: done (14 left) 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3573: done (13 left) 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3588: read oid 310 snap -1 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3588: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-21T18:10:08.832 INFO:tasks.rados.rados.0.smithi160.stdout:3589: write (excl) oid 69 current snap is 0 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: finishing write tid 3 to smithi16030472-480 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: finishing write tid 4 to smithi16030472-480 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: finishing write tid 5 to smithi16030472-480 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: oid 480 updating version 0 to 992 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: oid 480 version 992 is already newer than 991 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: oid 480 updating version 992 to 993 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: oid 480 updating version 993 to 994 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 480 v 994 (ObjNum 1507 snap 0 seq_num 1507) dirty exists 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3566: left oid 480 (ObjNum 1507 snap 0 seq_num 1507) 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3587: finishing copy_from to smithi16030472-381 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 844 (ObjNum 1202 snap 0 seq_num 1202) dirty exists 2024-09-21T18:10:08.833 INFO:tasks.rados.rados.0.smithi160.stdout:3589: seq_num 1517 ranges {696219=506980,1904478=794521,3159033=337186} 2024-09-21T18:10:08.842 INFO:tasks.rados.rados.0.smithi160.stdout:3589: writing smithi16030472-69 from 696219 to 1203199 tid 1 2024-09-21T18:10:08.845 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:08.841+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:08.863 INFO:tasks.rados.rados.0.smithi160.stdout:3589: writing smithi16030472-69 from 1904478 to 2698999 tid 2 2024-09-21T18:10:08.874 INFO:tasks.rados.rados.0.smithi160.stdout:3589: writing smithi16030472-69 from 3159033 to 3496219 tid 3 2024-09-21T18:10:08.874 INFO:tasks.rados.rados.0.smithi160.stdout:3566: done (14 left) 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3587: done (13 left) 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3590: cache_try_flush oid 401 snap -1 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3591: read oid 125 snap -1 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3591: expect (ObjNum 1333 snap 0 seq_num 1333) 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3592: copy_from oid 12 from oid 230 current snap is 0 2024-09-21T18:10:08.875 INFO:tasks.rados.rados.0.smithi160.stdout:3588: expect (ObjNum 145 snap 0 seq_num 145) 2024-09-21T18:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:08.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.20467409, 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:10:08.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.204603728, 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:10:08.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.192802293, 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.888 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:10:08.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.18493591000001, 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.892 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.893 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:10:08.895 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.184550266, 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:10:08.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:10:08.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.898 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:10:08.899 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 170.184504817, 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:10:08.900 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:10:08.901 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.902 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.903 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:10:08.904 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 169.1366069, 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:10:08.905 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:10:08.906 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:10:08.907 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:10:08.908 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 169.13649925799999, 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:10:08.909 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:10:08.910 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.911 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 163.56071664300001, 2024-09-21T18:10:08.912 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:08.913 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:10:08.914 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.915 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.916 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.917 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 163.56066470799999, 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.918 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.919 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:10:08.920 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:10:08.921 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:10:08.922 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 162.79396032, 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.923 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.924 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.925 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:10:08.926 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:10:08.927 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.928 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.929 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.929 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.929 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 162.56485706199999, 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:08.930 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.931 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:10:08.932 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:10:08.933 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.934 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 162.564429072, 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.935 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:10:08.936 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:10:08.937 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.938 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.939 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.940 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 162.56438250299999, 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.941 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.942 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.943 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.944 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:10:08.944 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.945 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:10:08.946 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 158.06371344300001, 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:10:08.947 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:10:08.948 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.949 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.950 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.951 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.05618164699999, 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.952 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.953 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.954 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:10:08.955 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:10:08.956 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.957 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 151.05610732700001, 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.958 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.959 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:10:08.960 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.961 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.962 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 150.65430408200001, 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.963 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:10:08.964 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.965 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:08.966 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 150.55195419399999, 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.967 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.968 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.969 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.970 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.971 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6670 4.2 4:574ca510:::smithi16030472-466:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 58.239325796000003, 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4286385400000001, 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6670 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:08.972 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637674+0000", 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.973 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637680+0000", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6629999999999998e-06 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637691+0000", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1097000000000001e-05 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637700+0000", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.974 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054141+0000", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41644096200000003 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054299+0000", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157572 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.065938+0000", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.011639553 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066192+0000", 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253222 2024-09-21T18:10:08.975 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066312+0000", 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120285 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:08.976 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:08.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:10:08.977 INFO:tasks.rados.rados.0.smithi160.stdout:3589: finishing write tid 1 to smithi16030472-69 2024-09-21T18:10:08.977 INFO:tasks.rados.rados.0.smithi160.stdout:3586: expect (ObjNum 1123 snap 0 seq_num 1123) 2024-09-21T18:10:08.978 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:08.973+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:08.981 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1798919 2024-09-21T18:10:09.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:10:09.079 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:09.072+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3570: finishing write tid 2 to smithi16030472-165 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: finishing write tid 1 to smithi16030472-429 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: finishing write tid 2 to smithi16030472-429 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: finishing write tid 3 to smithi16030472-429 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: finishing write tid 4 to smithi16030472-429 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: finishing write tid 5 to smithi16030472-429 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: oid 429 updating version 0 to 1039 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: oid 429 updating version 1039 to 1042 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: oid 429 version 1042 is already newer than 1041 2024-09-21T18:10:09.088 INFO:tasks.rados.rados.0.smithi160.stdout:3585: oid 429 version 1042 is already newer than 1040 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 1042 (ObjNum 1516 snap 0 seq_num 1516) dirty exists 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3585: left oid 429 (ObjNum 1516 snap 0 seq_num 1516) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: finishing write tid 3 to smithi16030472-294 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: finishing write tid 4 to smithi16030472-294 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: oid 294 updating version 0 to 1084 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: oid 294 updating version 1084 to 1085 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: oid 294 version 1085 is already newer than 1083 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 294 v 1085 (ObjNum 1515 snap 0 seq_num 1515) dirty exists 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: left oid 294 (ObjNum 1515 snap 0 seq_num 1515) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3582: done (15 left) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3585: done (14 left) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3586: done (13 left) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3588: done (12 left) 2024-09-21T18:10:09.089 INFO:tasks.rados.rados.0.smithi160.stdout:3589: finishing write tid 2 to smithi16030472-69 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3589: finishing write tid 3 to smithi16030472-69 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3584: finishing copy_from to smithi16030472-197 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 35 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3590: got (0) Success 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 401 v 0 (ObjNum 400 snap 0 seq_num 400) dirty exists 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3584: finishing copy_from racing read to smithi16030472-197 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: finishing write tid 3 to smithi16030472-165 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: finishing write tid 4 to smithi16030472-165 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: finishing write tid 5 to smithi16030472-165 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: oid 165 updating version 0 to 996 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: oid 165 version 996 is already newer than 995 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: oid 165 version 996 is already newer than 965 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:3570: oid 165 updating version 996 to 997 2024-09-21T18:10:09.090 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 165 v 997 (ObjNum 1509 snap 0 seq_num 1509) dirty exists 2024-09-21T18:10:09.091 INFO:tasks.rados.rados.0.smithi160.stdout:3570: left oid 165 (ObjNum 1509 snap 0 seq_num 1509) 2024-09-21T18:10:09.091 INFO:tasks.rados.rados.0.smithi160.stdout:3580: finishing write tid 1 to smithi16030472-419 2024-09-21T18:10:09.091 INFO:tasks.rados.rados.0.smithi160.stdout:3591: expect (ObjNum 1333 snap 0 seq_num 1333) 2024-09-21T18:10:09.091 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:09.115 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:09.115 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:09.116 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:09.116 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:09.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:10:09.160 INFO:tasks.rados.rados.0.smithi160.stdout:3589: finishing write tid 4 to smithi16030472-69 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3580: finishing write tid 2 to smithi16030472-419 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3593: read oid 454 snap -1 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3593: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3570: done (12 left) 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3584: done (11 left) 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3590: done (10 left) 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3591: done (9 left) 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3594: cache_flush oid 410 snap -1 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3595: delete oid 243 current snap is 0 2024-09-21T18:10:09.161 INFO:tasks.rados.rados.0.smithi160.stdout:3594: got (0) Success 2024-09-21T18:10:09.162 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 0 (ObjNum 129 snap 0 seq_num 3128284648) clean dne 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: finishing write tid 5 to smithi16030472-69 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: oid 69 updating version 0 to 1070 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: oid 69 version 1070 is already newer than 1067 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: oid 69 version 1070 is already newer than 1069 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: oid 69 version 1070 is already newer than 1068 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 69 v 1070 (ObjNum 1517 snap 0 seq_num 1517) dirty exists 2024-09-21T18:10:09.177 INFO:tasks.rados.rados.0.smithi160.stdout:3589: left oid 69 (ObjNum 1517 snap 0 seq_num 1517) 2024-09-21T18:10:09.180 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:09.172+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:09.240 INFO:tasks.rados.rados.0.smithi160.stdout:3580: finishing write tid 3 to smithi16030472-419 2024-09-21T18:10:09.240 INFO:tasks.rados.rados.0.smithi160.stdout:3580: finishing write tid 4 to smithi16030472-419 2024-09-21T18:10:09.241 INFO:tasks.rados.rados.0.smithi160.stdout:3576: finishing write tid 1 to smithi16030472-356 2024-09-21T18:10:09.248 INFO:tasks.rados.rados.0.smithi160.stdout:3571: got (0) Success 2024-09-21T18:10:09.248 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 467 v 0 (ObjNum 862 snap 0 seq_num 862) dirty exists 2024-09-21T18:10:09.259 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:09.259 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:09.259 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:09.265 INFO:tasks.rados.rados.0.smithi160.stdout:3571: done (10 left) 2024-09-21T18:10:09.265 INFO:tasks.rados.rados.0.smithi160.stdout:3589: done (9 left) 2024-09-21T18:10:09.265 INFO:tasks.rados.rados.0.smithi160.stdout:3594: done (8 left) 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3595: done (7 left) 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3596: cache_flush oid 304 snap -1 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3597: cache_flush oid 123 snap -1 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3598: read oid 115 snap -1 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3598: expect (ObjNum 895 snap 0 seq_num 895) 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3599: read oid 168 snap -1 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3599: expect (ObjNum 1113 snap 0 seq_num 1113) 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3600: cache_evict oid 104 snap -1 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3601: copy_from oid 355 from oid 98 current snap is 0 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3602: write oid 264 current snap is 0 2024-09-21T18:10:09.266 INFO:tasks.rados.rados.0.smithi160.stdout:3602: seq_num 1519 ranges {729833=657478,1905089=798190,3497526=32307} 2024-09-21T18:10:09.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:10:09.281 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:09.276+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:09.288 INFO:tasks.rados.rados.0.smithi160.stdout:3602: writing smithi16030472-264 from 729833 to 1387311 tid 1 2024-09-21T18:10:09.309 INFO:tasks.rados.rados.0.smithi160.stdout:3602: writing smithi16030472-264 from 1905089 to 2703279 tid 2 2024-09-21T18:10:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:3602: writing smithi16030472-264 from 3497526 to 3529833 tid 3 2024-09-21T18:10:09.321 INFO:tasks.rados.rados.0.smithi160.stdout:3603: cache_flush oid 461 snap -1 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3597: got (0) Success 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 123 v 0 (ObjNum 1439 snap 0 seq_num 1439) dirty exists 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3597: done (14 left) 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3604: read oid 105 snap -1 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3604: expect (ObjNum 1485 snap 0 seq_num 1485) 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3596: got (0) Success 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 304 v 0 (ObjNum 1251 snap 0 seq_num 1251) dirty exists 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3605: cache_try_flush oid 114 snap -1 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3596: done (15 left) 2024-09-21T18:10:09.322 INFO:tasks.rados.rados.0.smithi160.stdout:3606: delete oid 377 current snap is 0 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: finishing write tid 1 to smithi16030472-316 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: finishing write tid 2 to smithi16030472-316 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: finishing write tid 3 to smithi16030472-316 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: finishing write tid 4 to smithi16030472-316 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: finishing write tid 5 to smithi16030472-316 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: oid 316 updating version 0 to 1005 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: oid 316 version 1005 is already newer than 1004 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: oid 316 version 1005 is already newer than 1003 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: oid 316 version 1005 is already newer than 1002 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 316 v 1005 (ObjNum 1514 snap 0 seq_num 1514) dirty exists 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3581: left oid 316 (ObjNum 1514 snap 0 seq_num 1514) 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3592: finishing copy_from to smithi16030472-12 2024-09-21T18:10:09.354 INFO:tasks.rados.rados.0.smithi160.stdout:3592: got expected ENOENT (src dne) 2024-09-21T18:10:09.382 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:09.381+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:09.394 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:09.394 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:09.395 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:10:09.395 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:09.396 INFO:tasks.rados.rados.0.smithi160.stdout:3602: finishing write tid 1 to smithi16030472-264 2024-09-21T18:10:09.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:09.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:10:09.483 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:09.480+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:09.506 INFO:tasks.rados.rados.0.smithi160.stdout:3602: finishing write tid 2 to smithi16030472-264 2024-09-21T18:10:09.506 INFO:tasks.rados.rados.0.smithi160.stdout:3602: finishing write tid 3 to smithi16030472-264 2024-09-21T18:10:09.552 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:09.553 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:09.573 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:09.573 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:09.576 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:09.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:9490 4.0 4:3d365130:::smithi16030472-355:head [copy-from ver 0 in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:09.265332+0000", 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.31059933200000001, 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.31063584399999999, 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 9490 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265332+0000", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265332+0000", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265335+0000", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5569999999999998e-06 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:09.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265345+0000", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8160000000000005e-06 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265363+0000", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8774999999999999e-05 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265367+0000", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0419999999999996e-06 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265627+0000", 2024-09-21T18:10:09.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025990399999999998 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265753+0000", 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012584599999999999 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:9491 4.0 4:3d365130:::smithi16030472-355:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e121)", 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:09.265378+0000", 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.31055395400000002, 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.31065731400000002, 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:09.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for blocked object", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 9491 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265378+0000", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265378+0000", 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265378+0000", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.61e-07 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265381+0000", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8169999999999999e-06 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265389+0000", 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0220000000000004e-06 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265395+0000", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8819999999999998e-06 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.265916+0000", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00052148500000000003 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:09.266000+0000", 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3195999999999994e-05 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:09.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:10:09.584 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 2 2024-09-21T18:10:09.584 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:09.585 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:09.581+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:09.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:10:09.594 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:09.607 INFO:tasks.rados.rados.0.smithi160.stdout:3602: finishing write tid 4 to smithi16030472-264 2024-09-21T18:10:09.617 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:09.617 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:09.634 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:09.634 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:09.655 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:09.655 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:09.657 INFO:tasks.rados.rados.0.smithi160.stdout:3576: finishing write tid 2 to smithi16030472-356 2024-09-21T18:10:09.657 INFO:tasks.rados.rados.0.smithi160.stdout:3575: got (0) Success 2024-09-21T18:10:09.657 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 97 v 0 (ObjNum 49 snap 0 seq_num 1953066355) clean dne 2024-09-21T18:10:09.660 INFO:tasks.rados.rados.0.smithi160.stdout:3598: expect (ObjNum 895 snap 0 seq_num 895) 2024-09-21T18:10:09.676 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:09.686 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:09.680+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:09.697 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:09.697 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:09.708 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:09.708 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:09.719 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:09.719 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:09.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:10:09.741 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:09.753 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:09.763 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:09.763 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:09.784 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:09.784 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:09.786 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:09.785+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:09.805 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:09.812 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:09.828 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:09.851 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:09.851 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:09.855 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:09.855 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:09.864 INFO:tasks.rados.rados.0.smithi160.stdout:3599: expect (ObjNum 1113 snap 0 seq_num 1113) 2024-09-21T18:10:09.865 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:09.871 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:09.872 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:09.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:10:09.887 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:09.880+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:09.939 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:09.981 INFO:tasks.rados.rados.0.smithi160.stdout:3580: finishing write tid 5 to smithi16030472-419 2024-09-21T18:10:09.981 INFO:tasks.rados.rados.0.smithi160.stdout:3580: oid 419 updating version 0 to 999 2024-09-21T18:10:09.981 INFO:tasks.rados.rados.0.smithi160.stdout:3580: oid 419 updating version 999 to 1001 2024-09-21T18:10:09.981 INFO:tasks.rados.rados.0.smithi160.stdout:3580: oid 419 version 1001 is already newer than 1000 2024-09-21T18:10:09.981 INFO:tasks.rados.rados.0.smithi160.stdout:3580: oid 419 version 1001 is already newer than 998 2024-09-21T18:10:09.982 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 419 v 1001 (ObjNum 1513 snap 0 seq_num 1513) dirty exists 2024-09-21T18:10:09.982 INFO:tasks.rados.rados.0.smithi160.stdout:3580: left oid 419 (ObjNum 1513 snap 0 seq_num 1513) 2024-09-21T18:10:09.982 INFO:tasks.rados.rados.0.smithi160.stdout:3592: finishing copy_from racing read to smithi16030472-12 2024-09-21T18:10:09.982 INFO:tasks.rados.rados.0.smithi160.stdout:3593: expect (ObjNum 823 snap 0 seq_num 823) 2024-09-21T18:10:09.989 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:09.984+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:10.011 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 658462 2024-09-21T18:10:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:10:10.046 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:10.040+0000 7f440980f640 -1 osd.13 121 handle_osd_map injecting map crc failure 2024-09-21T18:10:10.046 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:10.045+0000 7f728550b640 -1 osd.14 121 handle_osd_map injecting map crc failure 2024-09-21T18:10:10.055 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.055 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:10:10.076 INFO:tasks.rados.rados.0.smithi160.stdout:3600: got (2) No such file or directory 2024-09-21T18:10:10.076 INFO:tasks.rados.rados.0.smithi160.stdout:3604: expect (ObjNum 1485 snap 0 seq_num 1485) 2024-09-21T18:10:10.090 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:10.089+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:10.123 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: finishing write tid 5 to smithi16030472-264 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: oid 264 updating version 0 to 1088 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: oid 264 updating version 1088 to 1089 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: oid 264 version 1089 is already newer than 1087 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: oid 264 updating version 1089 to 1090 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 1090 (ObjNum 1519 snap 0 seq_num 1519) dirty exists 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3602: left oid 264 (ObjNum 1519 snap 0 seq_num 1519) 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3601: finishing copy_from to smithi16030472-355 2024-09-21T18:10:10.157 INFO:tasks.rados.rados.0.smithi160.stdout:3601: got expected ENOENT (src dne) 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: finishing write tid 3 to smithi16030472-356 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: finishing write tid 4 to smithi16030472-356 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3579: finishing copy_from racing read to smithi16030472-275 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3601: finishing copy_from racing read to smithi16030472-355 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: finishing write tid 5 to smithi16030472-356 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: oid 356 updating version 0 to 967 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: oid 356 updating version 967 to 1007 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: oid 356 version 1007 is already newer than 1006 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: oid 356 updating version 1007 to 1008 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 356 v 1008 (ObjNum 1511 snap 0 seq_num 1511) dirty exists 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3576: left oid 356 (ObjNum 1511 snap 0 seq_num 1511) 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:3579: finishing copy_from to smithi16030472-275 2024-09-21T18:10:10.158 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 275 v 370 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3575: done (15 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3576: done (14 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3579: done (13 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3580: done (12 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3581: done (11 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3592: done (10 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3593: done (9 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3598: done (8 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3599: done (7 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3600: done (6 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3601: done (5 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3602: done (4 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3604: done (3 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3606: done (2 left) 2024-09-21T18:10:10.159 INFO:tasks.rados.rados.0.smithi160.stdout:3607: copy_from oid 199 from oid 53 current snap is 0 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:3603: got (0) Success 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 461 v 0 (ObjNum 1112 snap 0 seq_num 1112) dirty exists 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:3605: got (0) Success 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 114 v 0 (ObjNum 1493 snap 0 seq_num 1493) dirty exists 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:3603: done (2 left) 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:3605: done (1 left) 2024-09-21T18:10:10.160 INFO:tasks.rados.rados.0.smithi160.stdout:3608: delete oid 321 current snap is 0 2024-09-21T18:10:10.190 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:10.189+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:10.239 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.239 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:10:10.258 INFO:tasks.rados.rados.0.smithi160.stdout:3607: finishing copy_from racing read to smithi16030472-199 2024-09-21T18:10:10.278 INFO:tasks.rados.rados.0.smithi160.stdout:3607: finishing copy_from to smithi16030472-199 2024-09-21T18:10:10.278 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 762 (ObjNum 1115 snap 0 seq_num 1115) dirty exists 2024-09-21T18:10:10.291 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:10.289+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3607: done (1 left) 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3608: done (0 left) 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3609: read oid 247 snap -1 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3609: expect (ObjNum 1393 snap 0 seq_num 1393) 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3610: cache_try_flush oid 320 snap -1 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3611: copy_from oid 328 from oid 72 current snap is 0 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3612: read oid 451 snap -1 2024-09-21T18:10:10.364 INFO:tasks.rados.rados.0.smithi160.stdout:3612: expect deleted 2024-09-21T18:10:10.365 INFO:tasks.rados.rados.0.smithi160.stdout:3613: write oid 293 current snap is 0 2024-09-21T18:10:10.365 INFO:tasks.rados.rados.0.smithi160.stdout:3613: seq_num 1522 ranges {780254=599578,1796888=661870,2896148=684106} 2024-09-21T18:10:10.389 INFO:tasks.rados.rados.0.smithi160.stdout:3613: writing smithi16030472-293 from 780254 to 1379832 tid 1 2024-09-21T18:10:10.392 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:10.387+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:10.400 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:10.395+0000 7f4976023640 -1 osd.5 121 handle_osd_map injecting map crc failure 2024-09-21T18:10:10.406 INFO:tasks.rados.rados.0.smithi160.stdout:3613: writing smithi16030472-293 from 1796888 to 2458758 tid 2 2024-09-21T18:10:10.422 INFO:tasks.rados.rados.0.smithi160.stdout:3613: writing smithi16030472-293 from 2896148 to 3580254 tid 3 2024-09-21T18:10:10.423 INFO:tasks.rados.rados.0.smithi160.stdout:3614: write (excl) oid 85 current snap is 0 2024-09-21T18:10:10.423 INFO:tasks.rados.rados.0.smithi160.stdout:3614: seq_num 1523 ranges {797061=674827,2139017=405528,3207218=389843} 2024-09-21T18:10:10.438 INFO:tasks.rados.rados.0.smithi160.stdout:3614: writing smithi16030472-85 from 797061 to 1471888 tid 1 2024-09-21T18:10:10.443 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.443 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.447 INFO:tasks.rados.rados.0.smithi160.stdout:3614: writing smithi16030472-85 from 2139017 to 2544545 tid 2 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3614: writing smithi16030472-85 from 3207218 to 3597061 tid 3 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3615: cache_try_flush oid 99 snap -1 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3610: got (0) Success 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 320 v 0 (ObjNum 289 snap 0 seq_num 1) clean dne 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3610: done (6 left) 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3616: read oid 396 snap -1 2024-09-21T18:10:10.456 INFO:tasks.rados.rados.0.smithi160.stdout:3616: expect (ObjNum 1349 snap 0 seq_num 1349) 2024-09-21T18:10:10.457 INFO:tasks.rados.rados.0.smithi160.stdout:3617: delete oid 305 current snap is 0 2024-09-21T18:10:10.457 INFO:tasks.rados.rados.0.smithi160.stdout:3611: finishing copy_from racing read to smithi16030472-328 2024-09-21T18:10:10.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:10:10.466 INFO:tasks.rados.rados.0.smithi160.stdout:3609: expect (ObjNum 1393 snap 0 seq_num 1393) 2024-09-21T18:10:10.467 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:10.493 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:10.489+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:10.504 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:10.577 INFO:tasks.rados.rados.0.smithi160.stdout:3616: expect (ObjNum 1349 snap 0 seq_num 1349) 2024-09-21T18:10:10.578 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 411256 2024-09-21T18:10:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.594 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:10.591+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:10.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:10:10.695 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:10.693+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:10.731 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.731 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:3614: finishing write tid 1 to smithi16030472-85 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:3614: finishing write tid 2 to smithi16030472-85 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:3611: finishing copy_from to smithi16030472-328 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 328 v 877 (ObjNum 1425 snap 0 seq_num 1425) dirty exists 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:3613: finishing write tid 1 to smithi16030472-293 2024-09-21T18:10:10.733 INFO:tasks.rados.rados.0.smithi160.stdout:3613: finishing write tid 2 to smithi16030472-293 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3613: finishing write tid 3 to smithi16030472-293 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: finishing write tid 3 to smithi16030472-85 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: finishing write tid 4 to smithi16030472-85 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: finishing write tid 5 to smithi16030472-85 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: oid 85 updating version 0 to 1045 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: oid 85 version 1045 is already newer than 1043 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: oid 85 version 1045 is already newer than 1044 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: oid 85 updating version 1045 to 1046 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 1046 (ObjNum 1523 snap 0 seq_num 1523) dirty exists 2024-09-21T18:10:10.734 INFO:tasks.rados.rados.0.smithi160.stdout:3614: left oid 85 (ObjNum 1523 snap 0 seq_num 1523) 2024-09-21T18:10:10.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:10:10.758 INFO:tasks.rados.rados.0.smithi160.stdout:3613: finishing write tid 4 to smithi16030472-293 2024-09-21T18:10:10.758 INFO:tasks.rados.rados.0.smithi160.stdout:3613: finishing write tid 5 to smithi16030472-293 2024-09-21T18:10:10.758 INFO:tasks.rados.rados.0.smithi160.stdout:3613: oid 293 updating version 0 to 1010 2024-09-21T18:10:10.758 INFO:tasks.rados.rados.0.smithi160.stdout:3613: oid 293 updating version 1010 to 1011 2024-09-21T18:10:10.758 INFO:tasks.rados.rados.0.smithi160.stdout:3613: oid 293 updating version 1011 to 1013 2024-09-21T18:10:10.759 INFO:tasks.rados.rados.0.smithi160.stdout:3613: oid 293 version 1013 is already newer than 1012 2024-09-21T18:10:10.759 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 293 v 1013 (ObjNum 1522 snap 0 seq_num 1522) dirty exists 2024-09-21T18:10:10.759 INFO:tasks.rados.rados.0.smithi160.stdout:3613: left oid 293 (ObjNum 1522 snap 0 seq_num 1522) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3609: done (7 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3611: done (6 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3612: done (5 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3613: done (4 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3614: done (3 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3616: done (2 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3617: done (1 left) 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3618: cache_flush oid 164 snap -1 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3619: cache_try_flush oid 156 snap -1 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3620: read oid 67 snap -1 2024-09-21T18:10:10.760 INFO:tasks.rados.rados.0.smithi160.stdout:3620: expect deleted 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3621: copy_from oid 18 from oid 299 current snap is 0 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3622: cache_evict oid 34 snap -1 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3623: read oid 297 snap -1 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3623: expect (ObjNum 1230 snap 0 seq_num 1230) 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3624: cache_flush oid 98 snap -1 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3625: read oid 107 snap -1 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3625: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3626: read oid 175 snap -1 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3626: expect deleted 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3627: write (excl) oid 448 current snap is 0 2024-09-21T18:10:10.761 INFO:tasks.rados.rados.0.smithi160.stdout:3627: seq_num 1525 ranges {430675=425325,1455981=409197,2462064=535349,3463076=167599} 2024-09-21T18:10:10.766 INFO:tasks.rados.rados.0.smithi160.stdout:3627: writing smithi16030472-448 from 430675 to 856000 tid 1 2024-09-21T18:10:10.775 INFO:tasks.rados.rados.0.smithi160.stdout:3627: writing smithi16030472-448 from 1455981 to 1865178 tid 2 2024-09-21T18:10:10.785 INFO:tasks.rados.rados.0.smithi160.stdout:3627: writing smithi16030472-448 from 2462064 to 2997413 tid 3 2024-09-21T18:10:10.790 INFO:tasks.rados.rados.0.smithi160.stdout:3627: writing smithi16030472-448 from 3463076 to 3630675 tid 4 2024-09-21T18:10:10.790 INFO:tasks.rados.rados.0.smithi160.stdout:3628: write (excl) oid 484 current snap is 0 2024-09-21T18:10:10.790 INFO:tasks.rados.rados.0.smithi160.stdout:3628: seq_num 1526 ranges {447482=500574,1514463=552855,2489487=462974,3645668=1814} 2024-09-21T18:10:10.796 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:10.793+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:10.798 INFO:tasks.rados.rados.0.smithi160.stdout:3628: writing smithi16030472-484 from 447482 to 948056 tid 1 2024-09-21T18:10:10.808 INFO:tasks.rados.rados.0.smithi160.stdout:3628: writing smithi16030472-484 from 1514463 to 2067318 tid 2 2024-09-21T18:10:10.815 INFO:tasks.rados.rados.0.smithi160.stdout:3628: writing smithi16030472-484 from 2489487 to 2952461 tid 3 2024-09-21T18:10:10.820 INFO:tasks.rados.rados.0.smithi160.stdout:3628: writing smithi16030472-484 from 3645668 to 3647482 tid 4 2024-09-21T18:10:10.820 INFO:tasks.rados.rados.0.smithi160.stdout:3629: copy_from oid 82 from oid 451 current snap is 0 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3618: got (0) Success 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 164 v 0 (ObjNum 129 snap 0 seq_num 3128784304) clean dne 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3619: got (0) Success 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 156 v 0 (ObjNum 1019 snap 0 seq_num 1019) dirty exists 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3618: done (12 left) 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3619: done (11 left) 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3620: done (10 left) 2024-09-21T18:10:10.821 INFO:tasks.rados.rados.0.smithi160.stdout:3630: delete oid 470 current snap is 0 2024-09-21T18:10:10.861 INFO:tasks.rados.rados.0.smithi160.stdout:3624: got (0) Success 2024-09-21T18:10:10.862 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 98 v 0 (ObjNum 97217 snap 0 seq_num 3135953984) clean dne 2024-09-21T18:10:10.868 INFO:tasks.rados.rados.0.smithi160.stdout:3625: expect (ObjNum 887 snap 0 seq_num 887) 2024-09-21T18:10:10.897 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:10.893+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:10.925 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:10.926 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:10.931 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 611541 2024-09-21T18:10:10.941 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:10:10.998 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:10.993+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:11.029 INFO:tasks.rados.rados.0.smithi160.stdout:3623: expect (ObjNum 1230 snap 0 seq_num 1230) 2024-09-21T18:10:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.079 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.093 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:10:11.099 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:11.092+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:11.166 INFO:tasks.rados.rados.0.smithi160.stdout:3622: got (0) Success 2024-09-21T18:10:11.166 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 1 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 2 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3629: finishing copy_from to smithi16030472-82 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3629: got expected ENOENT (src dne) 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3629: finishing copy_from racing read to smithi16030472-82 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3615: got (0) Success 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 0 (ObjNum 1497 snap 0 seq_num 1497) dirty exists 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 1 to smithi16030472-484 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 2 to smithi16030472-484 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 3 to smithi16030472-484 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 4 to smithi16030472-484 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 5 to smithi16030472-484 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 3 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 4 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 5 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: finishing write tid 6 to smithi16030472-448 2024-09-21T18:10:11.167 INFO:tasks.rados.rados.0.smithi160.stdout:3627: oid 448 updating version 0 to 1050 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3627: oid 448 updating version 1050 to 1051 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3627: oid 448 version 1051 is already newer than 1047 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3627: oid 448 version 1051 is already newer than 1048 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3627: oid 448 version 1051 is already newer than 1049 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 1051 (ObjNum 1525 snap 0 seq_num 1525) dirty exists 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3627: left oid 448 (ObjNum 1525 snap 0 seq_num 1525) 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: finishing write tid 6 to smithi16030472-484 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: oid 484 updating version 0 to 1094 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: oid 484 updating version 1094 to 1095 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: oid 484 version 1095 is already newer than 1093 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: oid 484 version 1095 is already newer than 1092 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: oid 484 version 1095 is already newer than 1091 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 1095 (ObjNum 1526 snap 0 seq_num 1526) dirty exists 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3628: left oid 484 (ObjNum 1526 snap 0 seq_num 1526) 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3615: done (10 left) 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3622: done (9 left) 2024-09-21T18:10:11.168 INFO:tasks.rados.rados.0.smithi160.stdout:3623: done (8 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3624: done (7 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3625: done (6 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3626: done (5 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3627: done (4 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3628: done (3 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3629: done (2 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3630: done (1 left) 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3631: delete oid 478 current snap is 0 2024-09-21T18:10:11.169 INFO:tasks.rados.rados.0.smithi160.stdout:3631: done (1 left) 2024-09-21T18:10:11.170 INFO:tasks.rados.rados.0.smithi160.stdout:3632: write (excl) oid 481 current snap is 0 2024-09-21T18:10:11.170 INFO:tasks.rados.rados.0.smithi160.stdout:3632: seq_num 1529 ranges {497903=442674,1689909=416535,2571756=696790} 2024-09-21T18:10:11.182 INFO:tasks.rados.rados.0.smithi160.stdout:3632: writing smithi16030472-481 from 497903 to 940577 tid 1 2024-09-21T18:10:11.199 INFO:tasks.rados.rados.0.smithi160.stdout:3632: writing smithi16030472-481 from 1689909 to 2106444 tid 2 2024-09-21T18:10:11.200 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:11.197+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3632: writing smithi16030472-481 from 2571756 to 3268546 tid 3 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3633: cache_try_flush oid 335 snap -1 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3632: finishing write tid 1 to smithi16030472-481 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3634: copy_from oid 487 from oid 254 current snap is 0 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3621: finishing copy_from racing read to smithi16030472-18 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3621: finishing copy_from to smithi16030472-18 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 18 v 867 (ObjNum 1257 snap 0 seq_num 1257) dirty exists 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3621: done (3 left) 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3635: read oid 102 snap -1 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3635: expect (ObjNum 1327 snap 0 seq_num 1327) 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3636: read oid 143 snap -1 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3636: expect (ObjNum 1480 snap 0 seq_num 1480) 2024-09-21T18:10:11.217 INFO:tasks.rados.rados.0.smithi160.stdout:3637: copy_from oid 261 from oid 402 current snap is 0 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3638: copy_from oid 230 from oid 339 current snap is 0 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3639: read oid 427 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3639: expect (ObjNum 1405 snap 0 seq_num 1405) 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3640: read oid 362 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3640: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3641: copy_from oid 42 from oid 97 current snap is 0 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3642: read oid 291 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3642: expect (ObjNum 683 snap 0 seq_num 683) 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3643: cache_try_flush oid 189 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3644: cache_try_flush oid 178 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3645: read oid 224 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3645: expect (ObjNum 1499 snap 0 seq_num 1499) 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3646: read oid 170 snap -1 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3646: expect (ObjNum 1451 snap 0 seq_num 1451) 2024-09-21T18:10:11.218 INFO:tasks.rados.rados.0.smithi160.stdout:3647: cache_evict oid 106 snap -1 2024-09-21T18:10:11.219 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:11.221 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.221 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:10:11.301 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:11.295+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:11.317 INFO:tasks.rados.rados.0.smithi160.stdout:3633: got (0) Success 2024-09-21T18:10:11.317 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 335 v 0 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:10:11.317 INFO:tasks.rados.rados.0.smithi160.stdout:3633: done (15 left) 2024-09-21T18:10:11.317 INFO:tasks.rados.rados.0.smithi160.stdout:3648: copy_from oid 65 from oid 25 current snap is 0 2024-09-21T18:10:11.317 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:11.328 INFO:tasks.rados.rados.0.smithi160.stdout:3635: expect (ObjNum 1327 snap 0 seq_num 1327) 2024-09-21T18:10:11.357 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.357 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.372 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:10:11.402 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:11.396+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:11.404 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:11.470 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1248172 2024-09-21T18:10:11.498 INFO:tasks.rados.rados.0.smithi160.stdout:3642: expect (ObjNum 683 snap 0 seq_num 683) 2024-09-21T18:10:11.503 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:11.501+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:11.539 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:11.543 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.544 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.558 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:10:11.607 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:11.603+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:11.648 INFO:tasks.rados.rados.0.smithi160.stdout:3647: got (2) No such file or directory 2024-09-21T18:10:11.648 INFO:tasks.rados.rados.0.smithi160.stdout:3637: finishing copy_from to smithi16030472-261 2024-09-21T18:10:11.648 INFO:tasks.rados.rados.0.smithi160.stdout:3637: got expected ENOENT (src dne) 2024-09-21T18:10:11.648 INFO:tasks.rados.rados.0.smithi160.stdout:3637: finishing copy_from racing read to smithi16030472-261 2024-09-21T18:10:11.648 INFO:tasks.rados.rados.0.smithi160.stdout:3636: expect (ObjNum 1480 snap 0 seq_num 1480) 2024-09-21T18:10:11.676 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2317609 2024-09-21T18:10:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.686 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.695 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:11.701 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:10:11.705 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:11.700+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:11.779 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2111262 2024-09-21T18:10:11.800 INFO:tasks.rados.rados.0.smithi160.stdout:3643: got (0) Success 2024-09-21T18:10:11.801 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 189 v 0 (ObjNum 624 snap 0 seq_num 624) dirty exists 2024-09-21T18:10:11.801 INFO:tasks.rados.rados.0.smithi160.stdout:3639: expect (ObjNum 1405 snap 0 seq_num 1405) 2024-09-21T18:10:11.806 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:11.800+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:11.816 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2802036 2024-09-21T18:10:11.834 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1843793 2024-09-21T18:10:11.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:11.868 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3589190 2024-09-21T18:10:11.870 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:11.870 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3632: finishing write tid 2 to smithi16030472-481 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3632: finishing write tid 3 to smithi16030472-481 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3635: done (15 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3636: done (14 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3637: done (13 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3639: done (12 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3642: done (11 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3643: done (10 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3647: done (9 left) 2024-09-21T18:10:11.873 INFO:tasks.rados.rados.0.smithi160.stdout:3649: cache_evict oid 15 snap -1 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3634: finishing copy_from to smithi16030472-487 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3634: got expected ENOENT (src dne) 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3634: finishing copy_from racing read to smithi16030472-487 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3641: finishing copy_from to smithi16030472-42 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3641: got expected ENOENT (src dne) 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3641: finishing copy_from racing read to smithi16030472-42 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: finishing write tid 4 to smithi16030472-481 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: finishing write tid 5 to smithi16030472-481 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: oid 481 updating version 0 to 1056 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: oid 481 version 1056 is already newer than 1055 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: oid 481 version 1056 is already newer than 1054 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: oid 481 updating version 1056 to 1057 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 481 v 1057 (ObjNum 1529 snap 0 seq_num 1529) dirty exists 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3632: left oid 481 (ObjNum 1529 snap 0 seq_num 1529) 2024-09-21T18:10:11.874 INFO:tasks.rados.rados.0.smithi160.stdout:3640: expect (ObjNum 964 snap 0 seq_num 964) 2024-09-21T18:10:11.885 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:10:11.931 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:11.927+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:11.932 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1013876 2024-09-21T18:10:11.962 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 27025 2024-09-21T18:10:12.008 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:12.003+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:12.008 INFO:tasks.rados.rados.0.smithi160.stdout:3645: expect (ObjNum 1499 snap 0 seq_num 1499) 2024-09-21T18:10:12.009 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 292544 2024-09-21T18:10:12.011 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:12.011 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:12.018 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:12.018 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:12.026 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:10:12.040 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:12.041 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:12.064 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:12.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:10:12.087 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:12.087 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:12.104 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:12.104 INFO:tasks.rados.rados.0.smithi160.stdout:3646: expect (ObjNum 1451 snap 0 seq_num 1451) 2024-09-21T18:10:12.109 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:12.105+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:12.122 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:12.122 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:12.129 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:12.144 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:12.145 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:12.145 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:12.164 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:12.173 INFO:tasks.rados.rados.0.smithi160.stdout:3648: finishing copy_from to smithi16030472-65 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3648: got expected ENOENT (src dne) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3648: finishing copy_from racing read to smithi16030472-65 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3644: got (0) Success 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 178 v 0 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3638: finishing copy_from to smithi16030472-230 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 323 (ObjNum 491 snap 0 seq_num 491) dirty exists 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3638: finishing copy_from racing read to smithi16030472-230 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3632: done (9 left) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3634: done (8 left) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3638: done (7 left) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3640: done (6 left) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3641: done (5 left) 2024-09-21T18:10:12.174 INFO:tasks.rados.rados.0.smithi160.stdout:3644: done (4 left) 2024-09-21T18:10:12.175 INFO:tasks.rados.rados.0.smithi160.stdout:3645: done (3 left) 2024-09-21T18:10:12.175 INFO:tasks.rados.rados.0.smithi160.stdout:3646: done (2 left) 2024-09-21T18:10:12.175 INFO:tasks.rados.rados.0.smithi160.stdout:3648: done (1 left) 2024-09-21T18:10:12.175 INFO:tasks.rados.rados.0.smithi160.stdout:3650: write oid 95 current snap is 0 2024-09-21T18:10:12.175 INFO:tasks.rados.rados.0.smithi160.stdout:3650: seq_num 1530 ranges {514710=517923,1748391=560193,2999179=624415} 2024-09-21T18:10:12.179 INFO:tasks.rados.rados.0.smithi160.stdout:3650: writing smithi16030472-95 from 514710 to 1032633 tid 1 2024-09-21T18:10:12.185 INFO:tasks.rados.rados.0.smithi160.stdout:3650: writing smithi16030472-95 from 1748391 to 2308584 tid 2 2024-09-21T18:10:12.186 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3650: writing smithi16030472-95 from 2999179 to 3623594 tid 3 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3651: cache_try_flush oid 420 snap -1 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3649: got (0) Success 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3649: done (2 left) 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3652: cache_evict oid 479 snap -1 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3653: delete oid 34 current snap is 0 2024-09-21T18:10:12.193 INFO:tasks.rados.rados.0.smithi160.stdout:3652: got (16) Device or resource busy 2024-09-21T18:10:12.202 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:12.202 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:12.202 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:12.210 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:12.203+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:12.212 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:12.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:12.217 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:10:12.246 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:12.247 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:12.264 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:12.283 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:12.297 INFO:tasks.rados.rados.0.smithi160.stdout:3651: got (0) Success 2024-09-21T18:10:12.297 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 420 v 0 (ObjNum 129 snap 0 seq_num 3130406384) clean dne 2024-09-21T18:10:12.311 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:12.309+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:12.319 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:12.339 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:12.340 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:12.343 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:12.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:12.387 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:12.387 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:12.387 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:12.387 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1508 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'227, mlcod=50'223)", 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 161.35375196499999, 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22474888500000001, 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.388 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030587+0000", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030588+0000", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1930000000000001e-06 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030916+0000", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00032839099999999998 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030953+0000", 2024-09-21T18:10:12.389 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6452999999999998e-05 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.030978+0000", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.552e-05 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131280+0000", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100301508 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.131322+0000", 2024-09-21T18:10:12.390 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2530000000000001e-05 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255224+0000", 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.123902094 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.255336+0000", 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111196 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.391 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1509 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'228, mlcod=50'225)", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 161.35014141400001, 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.223574049, 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034197+0000", 2024-09-21T18:10:12.392 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.034201+0000", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3969999999999998e-06 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035071+0000", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087023899999999997 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035104+0000", 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3346000000000002e-05 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.393 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.035125+0000", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1104e-05 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.132989+0000", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097863793000000004 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.133019+0000", 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0256e-05 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.394 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257674+0000", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.124654209 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257771+0000", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7705000000000001e-05 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1510 3.0 e51/31 3:0382d963:::smithi16030472-337:head v 51'229, mlcod=50'225)", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:10:12.395 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 161.347334875, 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22085444300000001, 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037004+0000", 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.396 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037007+0000", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.36e-06 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037522+0000", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051534200000000001 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037560+0000", 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7047e-05 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.397 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037589+0000", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9448999999999999e-05 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135213+0000", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.097624488999999995 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.135256+0000", 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2972999999999997e-05 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.398 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257779+0000", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.122522187 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257858+0000", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9596000000000002e-05 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1523 3.0 e51/31 3:3d0b4c2b:::smithi16030472-340:head v 51'230, mlcod=50'225)", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 161.34358667500001, 2024-09-21T18:10:12.399 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217204175, 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040752+0000", 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.400 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.040755+0000", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.593e-06 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041142+0000", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00038734400000000002 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041168+0000", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.5599999999999999e-05 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.401 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.041183+0000", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5803000000000001e-05 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144904+0000", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10372047700000001 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.144961+0000", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7073000000000001e-05 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.402 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257863+0000", 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.112902419 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257956+0000", 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2866000000000002e-05 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1551 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'241, mlcod=51'229)", 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.722735871, 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.314347867, 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.403 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661603+0000", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.661604+0000", 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6300000000000004e-07 2024-09-21T18:10:12.404 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662236+0000", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00063211099999999998 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.662245+0000", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7759999999999992e-06 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762416+0000", 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100170779 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.405 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762609+0000", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000192315 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762652+0000", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3409999999999999e-05 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975881+0000", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21322941400000001 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.406 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975951+0000", 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9299000000000002e-05 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1552 3.0 e51/31 3:05986648:::smithi16030472-345:head v 51'242, mlcod=51'229)", 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.62188179899999, 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21354068100000001, 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.407 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762457+0000", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762460+0000", 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.667e-06 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.408 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762473+0000", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3953e-05 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762490+0000", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.649e-05 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.762505+0000", 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5367e-05 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.409 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764796+0000", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0022911519999999999 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764848+0000", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1307000000000002e-05 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975959+0000", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.211111196 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.975998+0000", 2024-09-21T18:10:12.410 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8549e-05 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1565 3.0 e51/31 3:03b3ee07:::smithi16030472-348:head v 51'244, mlcod=51'230)", 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.621125519, 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21673527000000001, 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:10:12.411 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763213+0000", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763214+0000", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2299999999999999e-07 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763571+0000", 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00035669100000000002 2024-09-21T18:10:12.412 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763582+0000", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1603000000000001e-05 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763592+0000", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9369999999999999e-06 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867861+0000", 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10426825200000001 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.413 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867924+0000", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3409000000000002e-05 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979883+0000", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111958928 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979948+0000", 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5526999999999998e-05 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.414 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1585 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'245, mlcod=51'234)", 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.620733962, 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21639325500000001, 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.415 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763605+0000", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763606+0000", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.6300000000000004e-07 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763869+0000", 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026385500000000001 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.416 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763878+0000", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0559999999999994e-06 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763885+0000", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9929999999999998e-06 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970915+0000", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20702981200000001 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.970977+0000", 2024-09-21T18:10:12.417 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1995999999999994e-05 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979958+0000", 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0089806350000000007 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979998+0000", 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0045e-05 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.418 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1586 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'246, mlcod=51'234)", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.620432838, 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.41830489599999998, 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763906+0000", 2024-09-21T18:10:12.419 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.763907+0000", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5600000000000005e-07 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764585+0000", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067825800000000005 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764599+0000", 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3906e-05 2024-09-21T18:10:12.420 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764606+0000", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7570000000000004e-06 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973638+0000", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20903128100000001 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.973684+0000", 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.5738999999999997e-05 2024-09-21T18:10:12.421 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.181962+0000", 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20827869800000001 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182211+0000", 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00024850099999999998 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1587 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'247, mlcod=51'234)", 2024-09-21T18:10:12.422 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.61970898800001, 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.417625792, 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764630+0000", 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.423 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.764631+0000", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765349+0000", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00071851300000000005 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.765368+0000", 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8743e-05 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.424 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865494+0000", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012591699999999 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976133+0000", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110639259 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.976188+0000", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.4570000000000001e-05 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.425 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182220+0000", 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.206032416 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182255+0000", 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5497000000000003e-05 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1588 3.0 e51/31 3:114d54bf:::smithi16030472-352:head v 51'248, mlcod=51'234)", 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:10:12.426 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.51881214900001, 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31959522099999998, 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865526+0000", 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.427 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865529+0000", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1069999999999999e-06 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865544+0000", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5726000000000002e-05 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865559+0000", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4377e-05 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.428 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865569+0000", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0370000000000001e-05 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979435+0000", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113865771 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.979475+0000", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.0485999999999998e-05 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185055+0000", 2024-09-21T18:10:12.429 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205579231 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185122+0000", 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7152999999999999e-05 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1600 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'249, mlcod=51'234)", 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.518717866, 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31955013399999999, 2024-09-21T18:10:12.430 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865621+0000", 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.431 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.865622+0000", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5400000000000001e-06 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866463+0000", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00084071199999999997 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866500+0000", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6727000000000003e-05 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.432 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866513+0000", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3443e-05 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980100+0000", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.113586927 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.980153+0000", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.2626000000000001e-05 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.433 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185126+0000", 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20497362 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185171+0000", 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4539000000000002e-05 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1601 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'250, mlcod=51'234)", 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.51780467099999, 2024-09-21T18:10:12.434 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31867633899999998, 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866534+0000", 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.435 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.866535+0000", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867070+0000", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00053475199999999999 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867081+0000", 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1287000000000001e-05 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.436 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867089+0000", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8699999999999992e-06 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983084+0000", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.115995579 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.983143+0000", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8386e-05 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.437 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185176+0000", 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20203295199999999 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185210+0000", 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4520000000000002e-05 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1602 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'251, mlcod=51'234)", 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.51723498300001, 2024-09-21T18:10:12.438 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31815525700000002, 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867104+0000", 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.439 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867105+0000", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5700000000000001e-07 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867672+0000", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000567908 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867687+0000", 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4183e-05 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.440 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867695+0000", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.5629999999999992e-06 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180254+0000", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31255896999999999 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.180288+0000", 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3847000000000003e-05 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.441 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185216+0000", 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0049277390000000004 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185259+0000", 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3189999999999998e-05 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1603 3.0 e51/31 3:3d365130:::smithi16030472-355:head v 51'252, mlcod=51'234)", 2024-09-21T18:10:12.442 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.516507499, 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31746343900000001, 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867831+0000", 2024-09-21T18:10:12.443 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867832+0000", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867839+0000", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.9439999999999999e-06 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867850+0000", 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1136e-05 2024-09-21T18:10:12.444 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.867858+0000", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3039999999999999e-06 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182391+0000", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31453379199999998 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.182457+0000", 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.5506000000000006e-05 2024-09-21T18:10:12.445 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185261+0000", 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.002804422 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185295+0000", 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3299000000000001e-05 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.446 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1615 3.0 e51/31 3:385a7674:::smithi16030472-358:head v 51'253, mlcod=51'234)", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.45032559000001, 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25186676800000002, 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.447 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934013+0000", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934017+0000", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5700000000000001e-06 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934894+0000", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00087770199999999995 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934925+0000", 2024-09-21T18:10:12.448 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.065e-05 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934940+0000", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4557e-05 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183129+0000", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.24818902000000001 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.183165+0000", 2024-09-21T18:10:12.449 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6053000000000002e-05 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185830+0000", 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026653710000000001 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.185880+0000", 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.9845000000000001e-05 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.450 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1732 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'276, mlcod=51'258)", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 159.19427978499999, 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.217203019, 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.451 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190059+0000", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190060+0000", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.283e-06 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190287+0000", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022724799999999999 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190305+0000", 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7646999999999999e-05 2024-09-21T18:10:12.452 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190322+0000", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.702e-05 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296004+0000", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.105681492 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.296016+0000", 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2520999999999999e-05 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.453 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407071+0000", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111054469 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407262+0000", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019133900000000001 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1733 3.0 e51/31 3:1d299e7f:::smithi16030472-387:head v 51'277, mlcod=51'258)", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:10:12.454 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 159.19399698699999, 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21696367999999999, 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190342+0000", 2024-09-21T18:10:12.455 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190343+0000", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.153e-06 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190352+0000", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8999999999999995e-06 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190388+0000", 2024-09-21T18:10:12.456 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5803000000000003e-05 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190402+0000", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4636e-05 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298723+0000", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10832126 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.298750+0000", 2024-09-21T18:10:12.457 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6421e-05 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407269+0000", 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108519296 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407305+0000", 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6211000000000003e-05 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.458 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1735 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'278, mlcod=51'258)", 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 159.193918002, 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.216919258, 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.459 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190421+0000", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190422+0000", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2669999999999999e-06 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190690+0000", 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026783500000000001 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.460 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190707+0000", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7050000000000001e-05 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.190720+0000", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3076e-05 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299112+0000", 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10839254299999999 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.461 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.299150+0000", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7119999999999997e-05 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407308+0000", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10815836400000001 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407340+0000", 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2002999999999998e-05 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.462 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1736 3.0 e51/31 3:1f60435c:::smithi16030472-388:head v 51'279, mlcod=51'258)", 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 159.191299671, 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.214336687, 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.463 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193039+0000", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193042+0000", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6800000000000002e-06 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193416+0000", 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00037407299999999999 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.464 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193445+0000", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9730000000000002e-05 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.193467+0000", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1725999999999998e-05 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402222+0000", 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20875459099999999 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.465 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.402293+0000", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1125999999999996e-05 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407343+0000", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050497290000000002 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:33.407376+0000", 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3031999999999998e-05 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.466 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.467 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:12.467 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:12.467 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:12.468 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:10:12.469 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:12.465+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:12.493 INFO:tasks.rados.rados.0.smithi160.stdout:3650: finishing write tid 1 to smithi16030472-95 2024-09-21T18:10:12.524 INFO:tasks.rados.rados.0.smithi160.stdout:3651: done (3 left) 2024-09-21T18:10:12.524 INFO:tasks.rados.rados.0.smithi160.stdout:3652: done (2 left) 2024-09-21T18:10:12.524 INFO:tasks.rados.rados.0.smithi160.stdout:3653: done (1 left) 2024-09-21T18:10:12.524 INFO:tasks.rados.rados.0.smithi160.stdout:3654: cache_evict oid 65 snap -1 2024-09-21T18:10:12.524 INFO:tasks.rados.rados.0.smithi160.stdout:3655: read oid 204 snap -1 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:3655: expect (ObjNum 1429 snap 0 seq_num 1429) 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:3656: cache_try_flush oid 458 snap -1 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:3657: cache_flush oid 61 snap -1 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:3658: delete oid 230 current snap is 0 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:3656: got (0) Success 2024-09-21T18:10:12.525 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 458 v 0 (ObjNum -1166044592 snap 21993 seq_num 304) clean dne 2024-09-21T18:10:12.569 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:12.563+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:12.598 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:12.598 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:12.598 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:12.598 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:12.600 INFO:tasks.rados.rados.0.smithi160.stdout:3654: got (0) Success 2024-09-21T18:10:12.613 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:10:12.624 INFO:tasks.rados.rados.0.smithi160.stdout:3650: finishing write tid 2 to smithi16030472-95 2024-09-21T18:10:12.699 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:12.695+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:12.701 INFO:tasks.rados.rados.0.smithi160.stdout:3654: done (5 left) 2024-09-21T18:10:12.701 INFO:tasks.rados.rados.0.smithi160.stdout:3656: done (4 left) 2024-09-21T18:10:12.701 INFO:tasks.rados.rados.0.smithi160.stdout:3658: done (3 left) 2024-09-21T18:10:12.701 INFO:tasks.rados.rados.0.smithi160.stdout:3659: write oid 427 current snap is 0 2024-09-21T18:10:12.701 INFO:tasks.rados.rados.0.smithi160.stdout:3659: seq_num 1533 ranges {565131=743670,1807484=423873,2965095=458231} 2024-09-21T18:10:12.722 INFO:tasks.rados.rados.0.smithi160.stdout:3659: writing smithi16030472-427 from 565131 to 1308801 tid 1 2024-09-21T18:10:12.737 INFO:tasks.rados.rados.0.smithi160.stdout:3659: writing smithi16030472-427 from 1807484 to 2231357 tid 2 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3659: writing smithi16030472-427 from 2965095 to 3423326 tid 3 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3660: read oid 127 snap -1 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3660: expect deleted 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3661: cache_flush oid 424 snap -1 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3657: got (0) Success 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 61 v 0 (ObjNum 898 snap 0 seq_num 898) dirty exists 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: finishing write tid 3 to smithi16030472-95 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: finishing write tid 4 to smithi16030472-95 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: finishing write tid 5 to smithi16030472-95 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: oid 95 updating version 0 to 1075 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: oid 95 updating version 1075 to 1096 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: oid 95 updating version 1096 to 1097 2024-09-21T18:10:12.756 INFO:tasks.rados.rados.0.smithi160.stdout:3650: oid 95 updating version 1097 to 1098 2024-09-21T18:10:12.757 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 95 v 1098 (ObjNum 1530 snap 0 seq_num 1530) dirty exists 2024-09-21T18:10:12.757 INFO:tasks.rados.rados.0.smithi160.stdout:3650: left oid 95 (ObjNum 1530 snap 0 seq_num 1530) 2024-09-21T18:10:12.757 INFO:tasks.rados.rados.0.smithi160.stdout:3650: done (5 left) 2024-09-21T18:10:12.757 INFO:tasks.rados.rados.0.smithi160.stdout:3657: done (4 left) 2024-09-21T18:10:12.757 INFO:tasks.rados.rados.0.smithi160.stdout:3662: delete oid 2 current snap is 0 2024-09-21T18:10:12.759 INFO:tasks.rados.rados.0.smithi160.stdout:3655: expect (ObjNum 1429 snap 0 seq_num 1429) 2024-09-21T18:10:12.771 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:12.769+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:12.782 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:12.782 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:12.782 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:12.796 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:10:12.809 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:12.883 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:12.879+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:12.929 INFO:tasks.rados.rados.0.smithi160.stdout:3661: got (0) Success 2024-09-21T18:10:12.929 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:10:12.957 INFO:tasks.rados.rados.0.smithi160.stdout:3659: finishing write tid 1 to smithi16030472-427 2024-09-21T18:10:12.957 INFO:tasks.rados.rados.0.smithi160.stdout:3659: finishing write tid 2 to smithi16030472-427 2024-09-21T18:10:12.961 INFO:tasks.rados.rados.0.smithi160.stdout:3655: done (4 left) 2024-09-21T18:10:12.961 INFO:tasks.rados.rados.0.smithi160.stdout:3661: done (3 left) 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3662: done (2 left) 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3663: copy_from oid 327 from oid 9 current snap is 0 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3664: copy_from oid 225 from oid 138 current snap is 0 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3665: read oid 84 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3665: expect (ObjNum 1217 snap 0 seq_num 1217) 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3666: cache_evict oid 257 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3667: cache_evict oid 431 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3668: cache_evict oid 450 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3669: read oid 224 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3669: expect (ObjNum 1499 snap 0 seq_num 1499) 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3670: cache_evict oid 313 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3671: cache_evict oid 395 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3672: cache_flush oid 236 snap -1 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3673: write (excl) oid 313 current snap is 0 2024-09-21T18:10:12.962 INFO:tasks.rados.rados.0.smithi160.stdout:3673: seq_num 1535 ranges {598745=494168,1808095=427542,2903588=597128} 2024-09-21T18:10:12.966 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:12.967 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:12.967 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:12.967 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:10:12.967 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:12.972 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:12.967+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:12.976 INFO:tasks.rados.rados.0.smithi160.stdout:3673: writing smithi16030472-313 from 598745 to 1092913 tid 1 2024-09-21T18:10:12.981 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:10:12.993 INFO:tasks.rados.rados.0.smithi160.stdout:3673: writing smithi16030472-313 from 1808095 to 2235637 tid 2 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3673: writing smithi16030472-313 from 2903588 to 3500716 tid 3 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3674: cache_evict oid 463 snap -1 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3670: got (16) Device or resource busy 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3670: done (13 left) 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3675: cache_try_flush oid 99 snap -1 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3676: cache_try_flush oid 217 snap -1 2024-09-21T18:10:13.011 INFO:tasks.rados.rados.0.smithi160.stdout:3665: expect (ObjNum 1217 snap 0 seq_num 1217) 2024-09-21T18:10:13.065 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:10:13.068 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2177862 2024-09-21T18:10:13.072 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:13.069+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:13.083 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:13.114 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.115 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:13.115 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:10:13.115 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.130 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3668: got (2) No such file or directory 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3673: finishing write tid 1 to smithi16030472-313 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3673: finishing write tid 2 to smithi16030472-313 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3663: finishing copy_from to smithi16030472-327 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3663: got expected ENOENT (src dne) 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3663: finishing copy_from racing read to smithi16030472-327 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3675: got (0) Success 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 99 v 0 (ObjNum 1497 snap 0 seq_num 1497) dirty exists 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3663: done (14 left) 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3665: done (13 left) 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3668: done (12 left) 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3675: done (11 left) 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3677: read oid 263 snap -1 2024-09-21T18:10:13.159 INFO:tasks.rados.rados.0.smithi160.stdout:3677: expect (ObjNum 898 snap 0 seq_num 898) 2024-09-21T18:10:13.160 INFO:tasks.rados.rados.0.smithi160.stdout:3678: write (excl) oid 188 current snap is 0 2024-09-21T18:10:13.160 INFO:tasks.rados.rados.0.smithi160.stdout:3678: seq_num 1536 ranges {615552=685770,1982930=571200,3047364=408400} 2024-09-21T18:10:13.174 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:13.173+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:13.178 INFO:tasks.rados.rados.0.smithi160.stdout:3678: writing smithi16030472-188 from 615552 to 1301322 tid 1 2024-09-21T18:10:13.196 INFO:tasks.rados.rados.0.smithi160.stdout:3678: writing smithi16030472-188 from 1982930 to 2554130 tid 2 2024-09-21T18:10:13.208 INFO:tasks.rados.rados.0.smithi160.stdout:3678: writing smithi16030472-188 from 3047364 to 3455764 tid 3 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3679: read oid 189 snap -1 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3679: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3680: delete oid 14 current snap is 0 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: finishing write tid 3 to smithi16030472-427 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: finishing write tid 4 to smithi16030472-427 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: finishing write tid 5 to smithi16030472-427 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: oid 427 updating version 0 to 1059 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: oid 427 updating version 1059 to 1060 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: oid 427 updating version 1060 to 1061 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: oid 427 version 1061 is already newer than 1058 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 427 v 1061 (ObjNum 1533 snap 0 seq_num 1533) dirty exists 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3659: left oid 427 (ObjNum 1533 snap 0 seq_num 1533) 2024-09-21T18:10:13.209 INFO:tasks.rados.rados.0.smithi160.stdout:3667: got (16) Device or resource busy 2024-09-21T18:10:13.210 INFO:tasks.rados.rados.0.smithi160.stdout:3669: expect (ObjNum 1499 snap 0 seq_num 1499) 2024-09-21T18:10:13.210 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:13.249 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.249 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:13.249 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:13.249 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:13.249 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.263 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:10:13.275 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:13.273+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:13.314 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2879892 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3671: got (16) Device or resource busy 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3672: got (0) Success 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 236 v 0 (ObjNum 235 snap 0 seq_num 235) dirty exists 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3666: got (0) Success 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: finishing write tid 3 to smithi16030472-313 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: finishing write tid 4 to smithi16030472-313 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: finishing write tid 5 to smithi16030472-313 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: oid 313 updating version 0 to 1099 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: oid 313 updating version 1099 to 1101 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: oid 313 version 1101 is already newer than 1100 2024-09-21T18:10:13.323 INFO:tasks.rados.rados.0.smithi160.stdout:3673: oid 313 updating version 1101 to 1102 2024-09-21T18:10:13.324 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 313 v 1102 (ObjNum 1535 snap 0 seq_num 1535) dirty exists 2024-09-21T18:10:13.324 INFO:tasks.rados.rados.0.smithi160.stdout:3673: left oid 313 (ObjNum 1535 snap 0 seq_num 1535) 2024-09-21T18:10:13.324 INFO:tasks.rados.rados.0.smithi160.stdout:3677: expect (ObjNum 898 snap 0 seq_num 898) 2024-09-21T18:10:13.339 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1750196 2024-09-21T18:10:13.376 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:13.371+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:13.412 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2537234 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 175.100142573, 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:10:13.434 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:10:13.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.26029057700001, 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:10:13.438 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.15657012899999, 2024-09-21T18:10:13.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:10:13.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:10:13.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.15579538599999, 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.448 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.15514736700001, 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:10:13.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.14127567200001, 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:10:13.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.457 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:13.458 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.13833465600001, 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.459 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.460 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:10:13.461 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.135317808, 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.462 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:10:13.463 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.464 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.465 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.133106226, 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.466 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:10:13.467 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.468 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.469 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.02776913400001, 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.470 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.471 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:10:13.472 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:10:13.473 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 168.33634676, 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:13.474 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:10:13.475 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:10:13.476 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:10:13.477 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 168.33331676, 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:13.478 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:10:13.479 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.480 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:13.481 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 163.36675279799999, 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.482 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.483 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:10:13.484 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 161.699917363, 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.485 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:10:13.486 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:10:13.487 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.488 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.39642628199999, 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.489 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.490 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:10:13.491 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.492 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.394129249, 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.493 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.494 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:10:13.495 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:10:13.496 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.392078801, 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.497 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.498 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.499 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.500 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.38992343999999, 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:13.501 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:10:13.502 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.503 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:13.504 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.38916068200001, 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.505 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.506 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.507 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 154.28725563699999, 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:10:13.508 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:10:13.509 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:10:13.510 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:13.511 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:13.512 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.513 INFO:tasks.rados.rados.0.smithi160.stdout:3679: expect (ObjNum 624 snap 0 seq_num 624) 2024-09-21T18:10:13.513 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:13.514 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:10:13.514 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:13.509+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:13.519 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1855336 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3664: finishing copy_from racing read to smithi16030472-225 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3676: got (2) No such file or directory 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3674: got (0) Success 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3664: finishing copy_from to smithi16030472-225 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 225 v 727 (ObjNum 1016 snap 0 seq_num 1016) dirty exists 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3678: finishing write tid 1 to smithi16030472-188 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3678: finishing write tid 2 to smithi16030472-188 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3678: finishing write tid 3 to smithi16030472-188 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3659: done (14 left) 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3660: done (13 left) 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3664: done (12 left) 2024-09-21T18:10:13.554 INFO:tasks.rados.rados.0.smithi160.stdout:3666: done (11 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3667: done (10 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3669: done (9 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3671: done (8 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3672: done (7 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3673: done (6 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3674: done (5 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3676: done (4 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3677: done (3 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3679: done (2 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3680: done (1 left) 2024-09-21T18:10:13.555 INFO:tasks.rados.rados.0.smithi160.stdout:3681: delete oid 338 current snap is 0 2024-09-21T18:10:13.557 INFO:tasks.rados.rados.0.smithi160.stdout:3681: done (1 left) 2024-09-21T18:10:13.557 INFO:tasks.rados.rados.0.smithi160.stdout:3682: read oid 399 snap -1 2024-09-21T18:10:13.557 INFO:tasks.rados.rados.0.smithi160.stdout:3682: expect (ObjNum 962 snap 0 seq_num 962) 2024-09-21T18:10:13.558 INFO:tasks.rados.rados.0.smithi160.stdout:3683: read oid 281 snap -1 2024-09-21T18:10:13.558 INFO:tasks.rados.rados.0.smithi160.stdout:3683: expect (ObjNum 974 snap 0 seq_num 974) 2024-09-21T18:10:13.558 INFO:tasks.rados.rados.0.smithi160.stdout:3684: write (excl) oid 298 current snap is 0 2024-09-21T18:10:13.558 INFO:tasks.rados.rados.0.smithi160.stdout:3684: seq_num 1539 ranges {665973=511517,1642023=718527,2896927=642216} 2024-09-21T18:10:13.577 INFO:tasks.rados.rados.0.smithi160.stdout:3684: writing smithi16030472-298 from 665973 to 1177490 tid 1 2024-09-21T18:10:13.596 INFO:tasks.rados.rados.0.smithi160.stdout:3684: writing smithi16030472-298 from 1642023 to 2360550 tid 2 2024-09-21T18:10:13.614 INFO:tasks.rados.rados.0.smithi160.stdout:3684: writing smithi16030472-298 from 2896927 to 3539143 tid 3 2024-09-21T18:10:13.614 INFO:tasks.rados.rados.0.smithi160.stdout:3685: cache_try_flush oid 410 snap -1 2024-09-21T18:10:13.614 INFO:tasks.rados.rados.0.smithi160.stdout:3686: write oid 240 current snap is 0 2024-09-21T18:10:13.614 INFO:tasks.rados.rados.0.smithi160.stdout:3682: expect (ObjNum 962 snap 0 seq_num 962) 2024-09-21T18:10:13.614 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:13.611+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:13.617 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 610262 2024-09-21T18:10:13.641 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:13.641 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:10:13.641 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.652 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:13.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:10:13.704 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1423417 2024-09-21T18:10:13.715 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:13.708+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:13.717 INFO:tasks.rados.rados.0.smithi160.stdout:3683: expect (ObjNum 974 snap 0 seq_num 974) 2024-09-21T18:10:13.717 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:13.780 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.780 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:13.780 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:13.780 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:13.780 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.782 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1901267 2024-09-21T18:10:13.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3684: finishing write tid 1 to smithi16030472-298 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: finishing write tid 4 to smithi16030472-188 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: finishing write tid 5 to smithi16030472-188 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: oid 188 updating version 0 to 1103 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: oid 188 updating version 1103 to 1105 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: oid 188 version 1105 is already newer than 1104 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:3678: oid 188 updating version 1105 to 1106 2024-09-21T18:10:13.837 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 188 v 1106 (ObjNum 1536 snap 0 seq_num 1536) dirty exists 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3678: left oid 188 (ObjNum 1536 snap 0 seq_num 1536) 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3685: got (0) Success 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 410 v 0 (ObjNum 129 snap 0 seq_num 3128284648) clean dne 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: finishing write tid 2 to smithi16030472-298 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: finishing write tid 3 to smithi16030472-298 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: finishing write tid 4 to smithi16030472-298 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: finishing write tid 5 to smithi16030472-298 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: oid 298 updating version 0 to 1065 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: oid 298 updating version 1065 to 1066 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: oid 298 version 1066 is already newer than 1064 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: oid 298 updating version 1066 to 1067 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 298 v 1067 (ObjNum 1539 snap 0 seq_num 1539) dirty exists 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3684: left oid 298 (ObjNum 1539 snap 0 seq_num 1539) 2024-09-21T18:10:13.838 INFO:tasks.rados.rados.0.smithi160.stdout:3686: seq_num 1540 ranges {682780=586766,1700505=578538,3040703=569841} 2024-09-21T18:10:13.839 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:13.837+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:13.857 INFO:tasks.rados.rados.0.smithi160.stdout:3686: writing smithi16030472-240 from 682780 to 1269546 tid 1 2024-09-21T18:10:13.860 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:13.852+0000 7f94db546640 -1 osd.15 122 handle_osd_map injecting map crc failure 2024-09-21T18:10:13.875 INFO:tasks.rados.rados.0.smithi160.stdout:3686: writing smithi16030472-240 from 1700505 to 2279043 tid 2 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3686: writing smithi16030472-240 from 3040703 to 3610544 tid 3 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3678: done (5 left) 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3682: done (4 left) 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3683: done (3 left) 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3684: done (2 left) 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3685: done (1 left) 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3687: cache_flush oid 355 snap -1 2024-09-21T18:10:13.895 INFO:tasks.rados.rados.0.smithi160.stdout:3686: finishing write tid 1 to smithi16030472-240 2024-09-21T18:10:13.896 INFO:tasks.rados.rados.0.smithi160.stdout:3688: read oid 202 snap -1 2024-09-21T18:10:13.896 INFO:tasks.rados.rados.0.smithi160.stdout:3688: expect deleted 2024-09-21T18:10:13.896 INFO:tasks.rados.rados.0.smithi160.stdout:3689: write oid 175 current snap is 0 2024-09-21T18:10:13.896 INFO:tasks.rados.rados.0.smithi160.stdout:3689: seq_num 1541 ranges {699587=662015,2158987=722196,3351773=547814} 2024-09-21T18:10:13.911 INFO:tasks.rados.rados.0.smithi160.stdout:3689: writing smithi16030472-175 from 699587 to 1361602 tid 1 2024-09-21T18:10:13.917 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:13.917 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:13.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:13.917 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:10:13.917 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:13.918 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:13.917+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:13.928 INFO:tasks.rados.rados.0.smithi160.stdout:3689: writing smithi16030472-175 from 2158987 to 2881183 tid 2 2024-09-21T18:10:13.932 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:10:13.938 INFO:tasks.rados.rados.0.smithi160.stdout:3689: writing smithi16030472-175 from 3351773 to 3899587 tid 3 2024-09-21T18:10:13.939 INFO:tasks.rados.rados.0.smithi160.stdout:3690: cache_flush oid 366 snap -1 2024-09-21T18:10:13.939 INFO:tasks.rados.rados.0.smithi160.stdout:3691: copy_from oid 314 from oid 388 current snap is 0 2024-09-21T18:10:13.939 INFO:tasks.rados.rados.0.smithi160.stdout:3692: cache_flush oid 364 snap -1 2024-09-21T18:10:13.939 INFO:tasks.rados.rados.0.smithi160.stdout:3693: write oid 357 current snap is 0 2024-09-21T18:10:13.939 INFO:tasks.rados.rados.0.smithi160.stdout:3693: seq_num 1542 ranges {716394=737264,2217469=582207,3495549=420845} 2024-09-21T18:10:13.951 INFO:tasks.rados.rados.0.smithi160.stdout:3693: writing smithi16030472-357 from 716394 to 1453658 tid 1 2024-09-21T18:10:13.960 INFO:tasks.rados.rados.0.smithi160.stdout:3693: writing smithi16030472-357 from 2217469 to 2799676 tid 2 2024-09-21T18:10:13.968 INFO:tasks.rados.rados.0.smithi160.stdout:3693: writing smithi16030472-357 from 3495549 to 3916394 tid 3 2024-09-21T18:10:13.968 INFO:tasks.rados.rados.0.smithi160.stdout:3694: delete oid 100 current snap is 0 2024-09-21T18:10:13.968 INFO:tasks.rados.rados.0.smithi160.stdout:3692: got (0) Success 2024-09-21T18:10:13.969 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 364 v 0 (ObjNum 1407 snap 0 seq_num 1407) dirty exists 2024-09-21T18:10:13.969 INFO:tasks.rados.rados.0.smithi160.stdout:3690: got (0) Success 2024-09-21T18:10:13.969 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 366 v 0 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T18:10:14.019 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:14.016+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:14.083 INFO:tasks.rados.rados.0.smithi160.stdout:3686: finishing write tid 2 to smithi16030472-240 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:9689 4.0 e123/33 4:2e7ef9ac:::smithi16030472-175:head v 123'1078, mlcod=122'1075)", 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:14.101091+0000", 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0044857489999999998, 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0045048850000000001, 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:10:14.106 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.101091+0000", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.101091+0000", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.101095+0000", 2024-09-21T18:10:14.107 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8990000000000003e-06 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102226+0000", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001131319 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102256+0000", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9913e-05 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102279+0000", 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2912999999999999e-05 2024-09-21T18:10:14.108 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102443+0000", 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016447900000000001 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102498+0000", 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4709000000000001e-05 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.109 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:9690 4.0 e123/33)", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:14.102329+0000", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.003247429, 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0032984749999999999, 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102329+0000", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.110 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102329+0000", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.102331+0000", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.216e-06 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103148+0000", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081709899999999995 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.111 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103174+0000", 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5999e-05 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103192+0000", 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7257000000000001e-05 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:9691 4.0 e123/33)", 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:14.103221+0000", 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.0023559430000000001, 2024-09-21T18:10:14.112 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0024284710000000002, 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103221+0000", 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103221+0000", 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.113 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103222+0000", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.604e-06 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103232+0000", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0302999999999999e-05 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103247+0000", 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4797000000000001e-05 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.114 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:14.103259+0000", 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1946e-05 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 3 2024-09-21T18:10:14.115 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.120 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:14.117+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:14.121 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:10:14.140 INFO:tasks.rados.rados.0.smithi160.stdout:3687: got (0) Success 2024-09-21T18:10:14.140 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 355 v 0 (ObjNum 1274 snap 0 seq_num 1274) dirty exists 2024-09-21T18:10:14.140 INFO:tasks.rados.rados.0.smithi160.stdout:3689: finishing write tid 1 to smithi16030472-175 2024-09-21T18:10:14.209 INFO:tasks.rados.rados.0.smithi160.stdout:3689: finishing write tid 2 to smithi16030472-175 2024-09-21T18:10:14.221 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:14.215+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:14.238 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.238 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:14.238 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:14.238 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:14.239 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:10:14.299 INFO:tasks.rados.rados.0.smithi160.stdout:3691: finishing copy_from racing read to smithi16030472-314 2024-09-21T18:10:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:3689: finishing write tid 3 to smithi16030472-175 2024-09-21T18:10:14.310 INFO:tasks.rados.rados.0.smithi160.stdout:3689: finishing write tid 4 to smithi16030472-175 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: finishing write tid 5 to smithi16030472-175 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: oid 175 updating version 0 to 1077 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: oid 175 updating version 1077 to 1078 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: oid 175 updating version 1078 to 1079 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: oid 175 updating version 1079 to 1080 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 175 v 1080 (ObjNum 1541 snap 0 seq_num 1541) dirty exists 2024-09-21T18:10:14.311 INFO:tasks.rados.rados.0.smithi160.stdout:3689: left oid 175 (ObjNum 1541 snap 0 seq_num 1541) 2024-09-21T18:10:14.314 INFO:tasks.rados.rados.0.smithi160.stdout:3687: done (8 left) 2024-09-21T18:10:14.314 INFO:tasks.rados.rados.0.smithi160.stdout:3688: done (7 left) 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3689: done (6 left) 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3690: done (5 left) 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3692: done (4 left) 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3694: done (3 left) 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3695: cache_evict oid 422 snap -1 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3696: cache_try_flush oid 6 snap -1 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3697: cache_try_flush oid 85 snap -1 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3698: cache_flush oid 238 snap -1 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3699: delete oid 156 current snap is 0 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:3698: got (0) Success 2024-09-21T18:10:14.315 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 238 v 0 (ObjNum 684 snap 0 seq_num 684) dirty exists 2024-09-21T18:10:14.316 INFO:tasks.rados.rados.0.smithi160.stdout:3695: got (16) Device or resource busy 2024-09-21T18:10:14.322 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:14.316+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:14.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:14.372 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.372 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:14.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:14.372 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 162.77484708399999, 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:10:14.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:10:14.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:10:14.376 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.344877137, 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.24735676200001, 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.381 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:10:14.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:10:14.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.246672512, 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.385 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:14.389 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.24583915700001, 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:10:14.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:10:14.393 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.243120527, 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:10:14.396 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.397 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.147312121, 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.398 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.399 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.400 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:10:14.401 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.14500758, 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.402 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.403 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.404 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.405 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.16737895999999, 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.406 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:10:14.407 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:10:14.408 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:10:14.409 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.066223278, 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:10:14.410 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.411 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.412 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.413 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:14.414 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.06542761599999, 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.415 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:10:14.416 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.417 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.418 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.04357420300001, 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.419 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:10:14.420 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:10:14.421 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.422 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.034221519, 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.423 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:10:14.424 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:10:14.425 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:10:14.426 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.028846871, 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:10:14.427 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.428 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.429 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.430 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.431 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 121.026298596, 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.432 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.433 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.434 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.435 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 114.798598059, 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.436 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.437 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.438 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.439 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 114.797931766, 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.440 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.441 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.442 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:10:14.443 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 114.797839654, 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.444 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.445 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.446 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.447 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:6894 4.2 e96/33 4:60b99da3:::smithi16030472-227:head v 96'728, mlcod=96'726)", 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 58.527199211000003, 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.303226251, 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.448 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843452+0000", 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0429999999999999e-06 2024-09-21T18:10:14.449 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844181+0000", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072923900000000002 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844209+0000", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7313000000000001e-05 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944352+0000", 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100143339 2024-09-21T18:10:14.450 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944508+0000", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015568600000000001 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944543+0000", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5649999999999999e-05 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146570+0000", 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20202621900000001 2024-09-21T18:10:14.451 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146675+0000", 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010576199999999999 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:9447 4.2 e121/33 4:4279d287:::smithi16030472-294:head v 121'1069, mlcod=121'1065)", 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 5.7226291280000003, 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30225417700000001, 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.452 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.453 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648022+0000", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1599999999999998e-06 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648035+0000", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2643000000000001e-05 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648063+0000", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8229999999999999e-05 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.454 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.748203+0000", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10013965800000001 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.948607+0000", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20040434600000001 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.948647+0000", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9344999999999997e-05 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.455 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.950195+0000", 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001548787 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.950273+0000", 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8008000000000003e-05 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.456 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.458 INFO:tasks.rados.rados.0.smithi160.stdout:3686: finishing write tid 3 to smithi16030472-240 2024-09-21T18:10:14.458 INFO:tasks.rados.rados.0.smithi160.stdout:3693: finishing write tid 1 to smithi16030472-357 2024-09-21T18:10:14.458 INFO:tasks.rados.rados.0.smithi160.stdout:3693: finishing write tid 2 to smithi16030472-357 2024-09-21T18:10:14.458 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:14.452+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:14.459 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:10:14.502 INFO:tasks.rados.rados.0.smithi160.stdout:3686: finishing write tid 4 to smithi16030472-240 2024-09-21T18:10:14.504 INFO:tasks.rados.rados.0.smithi160.stdout:3693: finishing write tid 3 to smithi16030472-357 2024-09-21T18:10:14.505 INFO:tasks.rados.rados.0.smithi160.stdout:3693: finishing write tid 4 to smithi16030472-357 2024-09-21T18:10:14.505 INFO:tasks.rados.rados.0.smithi160.stdout:3693: finishing write tid 5 to smithi16030472-357 2024-09-21T18:10:14.505 INFO:tasks.rados.rados.0.smithi160.stdout:3693: oid 357 updating version 0 to 1070 2024-09-21T18:10:14.505 INFO:tasks.rados.rados.0.smithi160.stdout:3693: oid 357 version 1070 is already newer than 1068 2024-09-21T18:10:14.505 INFO:tasks.rados.rados.0.smithi160.stdout:3693: oid 357 version 1070 is already newer than 1069 2024-09-21T18:10:14.506 INFO:tasks.rados.rados.0.smithi160.stdout:3693: oid 357 updating version 1070 to 1071 2024-09-21T18:10:14.506 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 357 v 1071 (ObjNum 1542 snap 0 seq_num 1542) dirty exists 2024-09-21T18:10:14.506 INFO:tasks.rados.rados.0.smithi160.stdout:3693: left oid 357 (ObjNum 1542 snap 0 seq_num 1542) 2024-09-21T18:10:14.506 INFO:tasks.rados.rados.0.smithi160.stdout:3691: finishing copy_from to smithi16030472-314 2024-09-21T18:10:14.506 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 314 v 683 (ObjNum 930 snap 0 seq_num 930) dirty exists 2024-09-21T18:10:14.506 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:14.507 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:14.515 INFO:tasks.rados.rados.0.smithi160.stdout:3696: got (0) Success 2024-09-21T18:10:14.515 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 0 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:10:14.526 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:14.527 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:14.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:14.546 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:14.559 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:14.557+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:14.568 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:14.568 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:14.585 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.585 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:14.585 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:10:14.585 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.599 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:10:14.602 INFO:tasks.rados.rados.0.smithi160.stdout:3686: finishing write tid 5 to smithi16030472-240 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:3686: oid 240 updating version 0 to 988 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:3686: oid 240 version 988 is already newer than 987 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:3686: oid 240 updating version 988 to 1016 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:3686: oid 240 updating version 1016 to 1017 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 240 v 1017 (ObjNum 1540 snap 0 seq_num 1540) dirty exists 2024-09-21T18:10:14.603 INFO:tasks.rados.rados.0.smithi160.stdout:3686: left oid 240 (ObjNum 1540 snap 0 seq_num 1540) 2024-09-21T18:10:14.612 INFO:tasks.rados.rados.0.smithi160.stdout:3686: done (7 left) 2024-09-21T18:10:14.612 INFO:tasks.rados.rados.0.smithi160.stdout:3691: done (6 left) 2024-09-21T18:10:14.612 INFO:tasks.rados.rados.0.smithi160.stdout:3693: done (5 left) 2024-09-21T18:10:14.612 INFO:tasks.rados.rados.0.smithi160.stdout:3695: done (4 left) 2024-09-21T18:10:14.612 INFO:tasks.rados.rados.0.smithi160.stdout:3696: done (3 left) 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3698: done (2 left) 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3699: done (1 left) 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3700: cache_flush oid 157 snap -1 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3701: cache_try_flush oid 327 snap -1 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3702: copy_from oid 25 from oid 123 current snap is 0 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3703: delete oid 392 current snap is 0 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:3700: got (0) Success 2024-09-21T18:10:14.613 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 157 v 0 (ObjNum 946 snap 0 seq_num 946) dirty exists 2024-09-21T18:10:14.626 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:14.626 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:14.632 INFO:tasks.rados.rados.0.smithi160.stdout:3697: got (0) Success 2024-09-21T18:10:14.632 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 85 v 0 (ObjNum 1523 snap 0 seq_num 1523) dirty exists 2024-09-21T18:10:14.649 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:14.650 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:14.660 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:14.655+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:14.674 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:14.675 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:14.675 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:14.697 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:14.720 INFO:tasks.rados.rados.0.smithi160.stdout:3697: done (4 left) 2024-09-21T18:10:14.720 INFO:tasks.rados.rados.0.smithi160.stdout:3700: done (3 left) 2024-09-21T18:10:14.720 INFO:tasks.rados.rados.0.smithi160.stdout:3703: done (2 left) 2024-09-21T18:10:14.720 INFO:tasks.rados.rados.0.smithi160.stdout:3704: cache_try_flush oid 208 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3705: read oid 243 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3705: expect deleted 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3706: cache_try_flush oid 220 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3707: cache_try_flush oid 104 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3708: cache_flush oid 391 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3709: read oid 492 snap -1 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3709: expect (ObjNum 491 snap 0 seq_num 491) 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3710: write (excl) oid 199 current snap is 0 2024-09-21T18:10:14.721 INFO:tasks.rados.rados.0.smithi160.stdout:3710: seq_num 1546 ranges {783622=754613,2051397=589545,3205241=753826} 2024-09-21T18:10:14.722 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:14.723 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:14.724 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.724 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:14.724 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:14.724 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:14.724 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.737 INFO:tasks.rados.rados.0.smithi160.stdout:3710: writing smithi16030472-199 from 783622 to 1538235 tid 1 2024-09-21T18:10:14.739 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:10:14.748 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:14.749 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:14.749 INFO:tasks.rados.rados.0.smithi160.stdout:3710: writing smithi16030472-199 from 2051397 to 2640942 tid 2 2024-09-21T18:10:14.761 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:14.755+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:14.764 INFO:tasks.rados.rados.0.smithi160.stdout:3710: writing smithi16030472-199 from 3205241 to 3959067 tid 3 2024-09-21T18:10:14.764 INFO:tasks.rados.rados.0.smithi160.stdout:3711: delete oid 466 current snap is 0 2024-09-21T18:10:14.764 INFO:tasks.rados.rados.0.smithi160.stdout:3708: got (0) Success 2024-09-21T18:10:14.764 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 391 v 0 (ObjNum 145 snap 0 seq_num 2436185440) clean dne 2024-09-21T18:10:14.820 INFO:tasks.rados.rados.0.smithi160.stdout:3702: finishing copy_from racing read to smithi16030472-25 2024-09-21T18:10:14.822 INFO:tasks.rados.rados.0.smithi160.stdout:3706: got (0) Success 2024-09-21T18:10:14.822 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 220 v 0 (ObjNum 947 snap 0 seq_num 947) dirty exists 2024-09-21T18:10:14.822 INFO:tasks.rados.rados.0.smithi160.stdout:3707: got (2) No such file or directory 2024-09-21T18:10:14.831 INFO:tasks.rados.rados.0.smithi160.stdout:3709: expect (ObjNum 491 snap 0 seq_num 491) 2024-09-21T18:10:14.831 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 344831 2024-09-21T18:10:14.850 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:14.862 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:14.856+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:14.866 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:14.866 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:14.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:14.866 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:14.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 176.02492611599999, 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:10:14.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.613283822, 2024-09-21T18:10:14.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:10:14.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.875 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.61045295, 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.876 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:10:14.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.879 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:10:14.880 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.60648004199999, 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:10:14.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.60336309100001, 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:10:14.885 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:10:14.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:10:14.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.889 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.60143208700001, 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:14.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.59832532799999, 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:10:14.894 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:10:14.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.59516764400001, 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.902 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.592934593, 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:14.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:10:14.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.909 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.59068292000001, 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.912 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:10:14.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.49810533199999, 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.917 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:10:14.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:10:14.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:14.924 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 168.72143595599999, 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:14.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:10:14.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:10:14.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 162.71192837199999, 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.932 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:14.933 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.934 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:10:14.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.938 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 162.60509839299999, 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.939 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.940 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:10:14.941 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:10:14.942 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.943 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:10:14.944 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:10:14.945 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:14.946 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.271726841, 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:14.947 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.948 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:10:14.949 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.950 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:10:14.951 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.952 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.953 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 150.499735372, 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:10:14.954 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:14.955 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.956 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:10:14.957 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.958 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:10:14.959 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.960 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:10:14.961 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:14.962 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 140.61758691700001, 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.963 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.964 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:10:14.965 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.966 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:10:14.967 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:10:14.968 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.969 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 116.077462469, 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.970 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.971 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.972 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.973 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.974 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 98.767504009000007, 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.975 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.976 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.977 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.978 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:14.979 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 95.523101698000005, 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:14.980 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:10:14.981 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:10:14.982 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:14.983 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:14.985 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:14.985 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:14.985 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:14.985 INFO:tasks.rados.rados.0.smithi160.stdout:3701: got (0) Success 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 327 v 0 (ObjNum 1205 snap 0 seq_num 1205) dirty exists 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3702: finishing copy_from to smithi16030472-25 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 25 v 1042 (ObjNum 1439 snap 0 seq_num 1439) dirty exists 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: finishing write tid 1 to smithi16030472-199 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: finishing write tid 2 to smithi16030472-199 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: finishing write tid 3 to smithi16030472-199 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: finishing write tid 4 to smithi16030472-199 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: finishing write tid 5 to smithi16030472-199 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: oid 199 updating version 0 to 1107 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: oid 199 updating version 1107 to 1109 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: oid 199 version 1109 is already newer than 1108 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: oid 199 updating version 1109 to 1110 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 199 v 1110 (ObjNum 1546 snap 0 seq_num 1546) dirty exists 2024-09-21T18:10:14.986 INFO:tasks.rados.rados.0.smithi160.stdout:3710: left oid 199 (ObjNum 1546 snap 0 seq_num 1546) 2024-09-21T18:10:14.987 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:14.987 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:14.988 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:10:14.988 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:14.981+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:14.988 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:14.988 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:14.989 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:14.989 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:14.990 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:15.041 INFO:tasks.rados.rados.0.smithi160.stdout:3701: done (9 left) 2024-09-21T18:10:15.041 INFO:tasks.rados.rados.0.smithi160.stdout:3702: done (8 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3705: done (7 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3706: done (6 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3707: done (5 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3708: done (4 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3709: done (3 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3710: done (2 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3711: done (1 left) 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3712: write (excl) oid 480 current snap is 0 2024-09-21T18:10:15.042 INFO:tasks.rados.rados.0.smithi160.stdout:3712: seq_num 1548 ranges {417236=505111,1652008=365228} 2024-09-21T18:10:15.048 INFO:tasks.rados.rados.0.smithi160.stdout:3712: writing smithi16030472-480 from 417236 to 922347 tid 1 2024-09-21T18:10:15.056 INFO:tasks.rados.rados.0.smithi160.stdout:3712: writing smithi16030472-480 from 1652008 to 2017236 tid 2 2024-09-21T18:10:15.056 INFO:tasks.rados.rados.0.smithi160.stdout:3713: cache_flush oid 387 snap -1 2024-09-21T18:10:15.056 INFO:tasks.rados.rados.0.smithi160.stdout:3704: got (0) Success 2024-09-21T18:10:15.056 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 208 v 0 (ObjNum 1468 snap 0 seq_num 1468) dirty exists 2024-09-21T18:10:15.057 INFO:tasks.rados.rados.0.smithi160.stdout:3704: done (2 left) 2024-09-21T18:10:15.057 INFO:tasks.rados.rados.0.smithi160.stdout:3714: write oid 260 current snap is 0 2024-09-21T18:10:15.057 INFO:tasks.rados.rados.0.smithi160.stdout:3714: seq_num 1549 ranges {434043=580360,1710490=323553} 2024-09-21T18:10:15.064 INFO:tasks.rados.rados.0.smithi160.stdout:3714: writing smithi16030472-260 from 434043 to 1014403 tid 1 2024-09-21T18:10:15.071 INFO:tasks.rados.rados.0.smithi160.stdout:3714: writing smithi16030472-260 from 1710490 to 2034043 tid 2 2024-09-21T18:10:15.072 INFO:tasks.rados.rados.0.smithi160.stdout:3715: cache_evict oid 127 snap -1 2024-09-21T18:10:15.072 INFO:tasks.rados.rados.0.smithi160.stdout:3716: copy_from oid 312 from oid 23 current snap is 0 2024-09-21T18:10:15.072 INFO:tasks.rados.rados.0.smithi160.stdout:3717: cache_evict oid 24 snap -1 2024-09-21T18:10:15.072 INFO:tasks.rados.rados.0.smithi160.stdout:3718: write oid 393 current snap is 0 2024-09-21T18:10:15.072 INFO:tasks.rados.rados.0.smithi160.stdout:3718: seq_num 1550 ranges {450850=655609,1768972=281878} 2024-09-21T18:10:15.079 INFO:tasks.rados.rados.0.smithi160.stdout:3718: writing smithi16030472-393 from 450850 to 1106459 tid 1 2024-09-21T18:10:15.086 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:15.079+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:3718: writing smithi16030472-393 from 1768972 to 2050850 tid 2 2024-09-21T18:10:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:3719: cache_evict oid 258 snap -1 2024-09-21T18:10:15.086 INFO:tasks.rados.rados.0.smithi160.stdout:3720: delete oid 287 current snap is 0 2024-09-21T18:10:15.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:10:15.121 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.121 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.135 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:10:15.157 INFO:tasks.rados.rados.0.smithi160.stdout:3713: got (0) Success 2024-09-21T18:10:15.157 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 387 v 0 (ObjNum 386 snap 0 seq_num 386) dirty exists 2024-09-21T18:10:15.173 INFO:tasks.rados.rados.0.smithi160.stdout:3715: got (16) Device or resource busy 2024-09-21T18:10:15.173 INFO:tasks.rados.rados.0.smithi160.stdout:3719: got (16) Device or resource busy 2024-09-21T18:10:15.186 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:15.180+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:15.188 INFO:tasks.rados.rados.0.smithi160.stdout:3716: finishing copy_from to smithi16030472-312 2024-09-21T18:10:15.188 INFO:tasks.rados.rados.0.smithi160.stdout:3716: got expected ENOENT (src dne) 2024-09-21T18:10:15.188 INFO:tasks.rados.rados.0.smithi160.stdout:3716: finishing copy_from racing read to smithi16030472-312 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3713: done (8 left) 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3715: done (7 left) 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3716: done (6 left) 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3719: done (5 left) 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3720: done (4 left) 2024-09-21T18:10:15.190 INFO:tasks.rados.rados.0.smithi160.stdout:3721: delete oid 467 current snap is 0 2024-09-21T18:10:15.259 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.259 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.274 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:15.287 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:15.285+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:15.364 INFO:tasks.rados.rados.0.smithi160.stdout:3718: finishing write tid 1 to smithi16030472-393 2024-09-21T18:10:15.388 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:15.380+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:15.440 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.441 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.456 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:15.464 INFO:tasks.rados.rados.0.smithi160.stdout:3718: finishing write tid 2 to smithi16030472-393 2024-09-21T18:10:15.468 INFO:tasks.rados.rados.0.smithi160.stdout:3712: finishing write tid 1 to smithi16030472-480 2024-09-21T18:10:15.468 INFO:tasks.rados.rados.0.smithi160.stdout:3712: finishing write tid 2 to smithi16030472-480 2024-09-21T18:10:15.468 INFO:tasks.rados.rados.0.smithi160.stdout:3718: finishing write tid 3 to smithi16030472-393 2024-09-21T18:10:15.468 INFO:tasks.rados.rados.0.smithi160.stdout:3712: finishing write tid 3 to smithi16030472-480 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3718: finishing write tid 4 to smithi16030472-393 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3718: oid 393 updating version 0 to 1082 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3718: oid 393 updating version 1082 to 1083 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3718: oid 393 updating version 1083 to 1084 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 393 v 1084 (ObjNum 1550 snap 0 seq_num 1550) dirty exists 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3718: left oid 393 (ObjNum 1550 snap 0 seq_num 1550) 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3712: finishing write tid 4 to smithi16030472-480 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3712: oid 480 updating version 0 to 1018 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3712: oid 480 updating version 1018 to 1019 2024-09-21T18:10:15.469 INFO:tasks.rados.rados.0.smithi160.stdout:3712: oid 480 updating version 1019 to 1020 2024-09-21T18:10:15.470 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 480 v 1020 (ObjNum 1548 snap 0 seq_num 1548) dirty exists 2024-09-21T18:10:15.470 INFO:tasks.rados.rados.0.smithi160.stdout:3712: left oid 480 (ObjNum 1548 snap 0 seq_num 1548) 2024-09-21T18:10:15.470 INFO:tasks.rados.rados.0.smithi160.stdout:3714: finishing write tid 1 to smithi16030472-260 2024-09-21T18:10:15.470 INFO:tasks.rados.rados.0.smithi160.stdout:3714: finishing write tid 2 to smithi16030472-260 2024-09-21T18:10:15.490 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:15.489+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:15.570 INFO:tasks.rados.rados.0.smithi160.stdout:3717: got (16) Device or resource busy 2024-09-21T18:10:15.572 INFO:tasks.rados.rados.0.smithi160.stdout:3714: finishing write tid 3 to smithi16030472-260 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:3714: finishing write tid 4 to smithi16030472-260 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:3714: oid 260 updating version 0 to 1022 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:3714: oid 260 version 1022 is already newer than 1021 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:3714: oid 260 updating version 1022 to 1023 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 260 v 1023 (ObjNum 1549 snap 0 seq_num 1549) dirty exists 2024-09-21T18:10:15.573 INFO:tasks.rados.rados.0.smithi160.stdout:3714: left oid 260 (ObjNum 1549 snap 0 seq_num 1549) 2024-09-21T18:10:15.587 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.587 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.590 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:15.583+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:15.601 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3712: done (4 left) 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3714: done (3 left) 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3717: done (2 left) 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3718: done (1 left) 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3721: done (0 left) 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3722: read oid 326 snap -1 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3722: expect deleted 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3723: write oid 31 current snap is 0 2024-09-21T18:10:15.675 INFO:tasks.rados.rados.0.smithi160.stdout:3723: seq_num 1553 ranges {501271=597709,1544418=556853} 2024-09-21T18:10:15.682 INFO:tasks.rados.rados.0.smithi160.stdout:3723: writing smithi16030472-31 from 501271 to 1098980 tid 1 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3723: writing smithi16030472-31 from 1544418 to 2101271 tid 2 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3724: copy_from oid 220 from oid 283 current snap is 0 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3725: read oid 8 snap -1 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3725: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3726: cache_try_flush oid 14 snap -1 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3727: write (excl) oid 496 current snap is 0 2024-09-21T18:10:15.690 INFO:tasks.rados.rados.0.smithi160.stdout:3727: seq_num 1554 ranges {518078=672958,1602900=487868} 2024-09-21T18:10:15.691 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:15.689+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:15.698 INFO:tasks.rados.rados.0.smithi160.stdout:3727: writing smithi16030472-496 from 518078 to 1191036 tid 1 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3727: writing smithi16030472-496 from 1602900 to 2090768 tid 2 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3728: copy_from oid 391 from oid 234 current snap is 0 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: finishing write tid 1 to smithi16030472-31 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3729: cache_evict oid 258 snap -1 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3730: cache_evict oid 319 snap -1 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: finishing write tid 2 to smithi16030472-31 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: finishing write tid 3 to smithi16030472-31 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: finishing write tid 4 to smithi16030472-31 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: oid 31 updating version 0 to 1086 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: oid 31 version 1086 is already newer than 1085 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: oid 31 updating version 1086 to 1087 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 31 v 1087 (ObjNum 1553 snap 0 seq_num 1553) dirty exists 2024-09-21T18:10:15.706 INFO:tasks.rados.rados.0.smithi160.stdout:3723: left oid 31 (ObjNum 1553 snap 0 seq_num 1553) 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3723: done (8 left) 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3731: copy_from oid 58 from oid 303 current snap is 0 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3732: read oid 113 snap -1 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3732: expect (ObjNum 817 snap 0 seq_num 817) 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3733: write (excl) oid 124 current snap is 0 2024-09-21T18:10:15.707 INFO:tasks.rados.rados.0.smithi160.stdout:3733: seq_num 1555 ranges {534885=748207,1945029=189856} 2024-09-21T18:10:15.714 INFO:tasks.rados.rados.0.smithi160.stdout:3733: writing smithi16030472-124 from 534885 to 1283092 tid 1 2024-09-21T18:10:15.721 INFO:tasks.rados.rados.0.smithi160.stdout:3733: writing smithi16030472-124 from 1945029 to 2134885 tid 2 2024-09-21T18:10:15.721 INFO:tasks.rados.rados.0.smithi160.stdout:3734: write oid 343 current snap is 0 2024-09-21T18:10:15.721 INFO:tasks.rados.rados.0.smithi160.stdout:3734: seq_num 1556 ranges {551692=423456,1603511=491537} 2024-09-21T18:10:15.728 INFO:tasks.rados.rados.0.smithi160.stdout:3734: writing smithi16030472-343 from 551692 to 975148 tid 1 2024-09-21T18:10:15.736 INFO:tasks.rados.rados.0.smithi160.stdout:3734: writing smithi16030472-343 from 1603511 to 2095048 tid 2 2024-09-21T18:10:15.736 INFO:tasks.rados.rados.0.smithi160.stdout:3735: write oid 197 current snap is 0 2024-09-21T18:10:15.736 INFO:tasks.rados.rados.0.smithi160.stdout:3735: seq_num 1557 ranges {568499=498705,1661993=506506} 2024-09-21T18:10:15.743 INFO:tasks.rados.rados.0.smithi160.stdout:3735: writing smithi16030472-197 from 568499 to 1067204 tid 1 2024-09-21T18:10:15.751 INFO:tasks.rados.rados.0.smithi160.stdout:3735: writing smithi16030472-197 from 1661993 to 2168499 tid 2 2024-09-21T18:10:15.751 INFO:tasks.rados.rados.0.smithi160.stdout:3736: cache_evict oid 466 snap -1 2024-09-21T18:10:15.751 INFO:tasks.rados.rados.0.smithi160.stdout:3737: copy_from oid 317 from oid 165 current snap is 0 2024-09-21T18:10:15.751 INFO:tasks.rados.rados.0.smithi160.stdout:3738: cache_flush oid 457 snap -1 2024-09-21T18:10:15.751 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:15.770 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.770 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.784 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:15.792 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:15.787+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3722: done (15 left) 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3739: copy_from oid 429 from oid 70 current snap is 0 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3730: got (16) Device or resource busy 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3730: done (15 left) 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3740: read oid 19 snap -1 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout:3740: expect (ObjNum 960 snap 0 seq_num 960) 2024-09-21T18:10:15.808 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:15.815 INFO:tasks.rados.rados.0.smithi160.stdout:3728: finishing copy_from to smithi16030472-391 2024-09-21T18:10:15.815 INFO:tasks.rados.rados.0.smithi160.stdout:3728: got expected ENOENT (src dne) 2024-09-21T18:10:15.817 INFO:tasks.rados.rados.0.smithi160.stdout:3728: finishing copy_from racing read to smithi16030472-391 2024-09-21T18:10:15.817 INFO:tasks.rados.rados.0.smithi160.stdout:3728: done (15 left) 2024-09-21T18:10:15.817 INFO:tasks.rados.rados.0.smithi160.stdout:3741: delete oid 461 current snap is 0 2024-09-21T18:10:15.819 INFO:tasks.rados.rados.0.smithi160.stdout:3734: finishing write tid 1 to smithi16030472-343 2024-09-21T18:10:15.819 INFO:tasks.rados.rados.0.smithi160.stdout:3734: finishing write tid 2 to smithi16030472-343 2024-09-21T18:10:15.893 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:15.889+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:15.920 INFO:tasks.rados.rados.0.smithi160.stdout:3734: finishing write tid 3 to smithi16030472-343 2024-09-21T18:10:15.920 INFO:tasks.rados.rados.0.smithi160.stdout:3732: expect (ObjNum 817 snap 0 seq_num 817) 2024-09-21T18:10:15.920 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:15.931 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:15.931 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:15.945 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:15.994 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:15.987+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:16.043 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3082660 2024-09-21T18:10:16.055 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:16.072 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:10:16.073 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:16.067+0000 7f440980f640 -1 osd.13 123 handle_osd_map injecting map crc failure 2024-09-21T18:10:16.095 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:16.093+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:16.114 INFO:tasks.rados.rados.0.smithi160.stdout:3736: got (16) Device or resource busy 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: finishing write tid 1 to smithi16030472-124 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3734: finishing write tid 4 to smithi16030472-343 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3734: oid 343 updating version 0 to 1091 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3734: oid 343 version 1091 is already newer than 1088 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3734: oid 343 version 1091 is already newer than 1089 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 343 v 1091 (ObjNum 1556 snap 0 seq_num 1556) dirty exists 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3734: left oid 343 (ObjNum 1556 snap 0 seq_num 1556) 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: finishing write tid 2 to smithi16030472-124 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: finishing write tid 3 to smithi16030472-124 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: finishing write tid 4 to smithi16030472-124 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: oid 124 updating version 0 to 1113 2024-09-21T18:10:16.115 INFO:tasks.rados.rados.0.smithi160.stdout:3733: oid 124 version 1113 is already newer than 1111 2024-09-21T18:10:16.116 INFO:tasks.rados.rados.0.smithi160.stdout:3733: oid 124 version 1113 is already newer than 1112 2024-09-21T18:10:16.116 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 124 v 1113 (ObjNum 1555 snap 0 seq_num 1555) dirty exists 2024-09-21T18:10:16.116 INFO:tasks.rados.rados.0.smithi160.stdout:3733: left oid 124 (ObjNum 1555 snap 0 seq_num 1555) 2024-09-21T18:10:16.116 INFO:tasks.rados.rados.0.smithi160.stdout:3725: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:10:16.121 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.121 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.122 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:16.119+0000 7f0b6a212640 -1 osd.9 123 handle_osd_map injecting map crc failure 2024-09-21T18:10:16.136 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:16.157 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 505656 2024-09-21T18:10:16.196 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:16.191+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:16.196 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:16.243 INFO:tasks.rados.rados.0.smithi160.stdout:3726: got (0) Success 2024-09-21T18:10:16.243 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 14 v 0 (ObjNum 899553 snap 0 seq_num 3141171696) clean dne 2024-09-21T18:10:16.243 INFO:tasks.rados.rados.0.smithi160.stdout:3729: got (16) Device or resource busy 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3727: finishing write tid 1 to smithi16030472-496 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3724: finishing copy_from to smithi16030472-220 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3724: got expected ENOENT (src dne) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3731: finishing copy_from to smithi16030472-58 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 58 v 1002 (ObjNum 1423 snap 0 seq_num 1423) dirty exists 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3731: finishing copy_from racing read to smithi16030472-58 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3725: done (15 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3726: done (14 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3729: done (13 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3731: done (12 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3732: done (11 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3733: done (10 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3734: done (9 left) 2024-09-21T18:10:16.244 INFO:tasks.rados.rados.0.smithi160.stdout:3736: done (8 left) 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3741: done (7 left) 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3742: cache_try_flush oid 260 snap -1 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3724: finishing copy_from racing read to smithi16030472-220 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3724: done (7 left) 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3743: cache_flush oid 335 snap -1 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3744: cache_flush oid 39 snap -1 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3745: write (excl) oid 83 current snap is 0 2024-09-21T18:10:16.245 INFO:tasks.rados.rados.0.smithi160.stdout:3745: seq_num 1559 ranges {602113=765556,1778957=423156} 2024-09-21T18:10:16.260 INFO:tasks.rados.rados.0.smithi160.stdout:3745: writing smithi16030472-83 from 602113 to 1367669 tid 1 2024-09-21T18:10:16.268 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.268 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.271 INFO:tasks.rados.rados.0.smithi160.stdout:3745: writing smithi16030472-83 from 1778957 to 2202113 tid 2 2024-09-21T18:10:16.271 INFO:tasks.rados.rados.0.smithi160.stdout:3746: write (excl) oid 60 current snap is 0 2024-09-21T18:10:16.271 INFO:tasks.rados.rados.0.smithi160.stdout:3746: seq_num 1560 ranges {618920=440805,1837439=381481} 2024-09-21T18:10:16.283 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:16.284 INFO:tasks.rados.rados.0.smithi160.stdout:3746: writing smithi16030472-60 from 618920 to 1059725 tid 1 2024-09-21T18:10:16.296 INFO:tasks.rados.rados.0.smithi160.stdout:3746: writing smithi16030472-60 from 1837439 to 2218920 tid 2 2024-09-21T18:10:16.296 INFO:tasks.rados.rados.0.smithi160.stdout:3747: read oid 457 snap -1 2024-09-21T18:10:16.296 INFO:tasks.rados.rados.0.smithi160.stdout:3747: expect (ObjNum 1242 snap 0 seq_num 1242) 2024-09-21T18:10:16.296 INFO:tasks.rados.rados.0.smithi160.stdout:3748: delete oid 115 current snap is 0 2024-09-21T18:10:16.297 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:16.293+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:16.303 INFO:tasks.rados.rados.0.smithi160.stdout:3735: finishing write tid 1 to smithi16030472-197 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: finishing write tid 2 to smithi16030472-197 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: finishing write tid 3 to smithi16030472-197 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: finishing write tid 4 to smithi16030472-197 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: oid 197 updating version 0 to 1073 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: oid 197 updating version 1073 to 1075 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:3735: oid 197 version 1075 is already newer than 1074 2024-09-21T18:10:16.304 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 197 v 1075 (ObjNum 1557 snap 0 seq_num 1557) dirty exists 2024-09-21T18:10:16.305 INFO:tasks.rados.rados.0.smithi160.stdout:3735: left oid 197 (ObjNum 1557 snap 0 seq_num 1557) 2024-09-21T18:10:16.305 INFO:tasks.rados.rados.0.smithi160.stdout:3727: finishing write tid 2 to smithi16030472-496 2024-09-21T18:10:16.305 INFO:tasks.rados.rados.0.smithi160.stdout:3727: finishing write tid 3 to smithi16030472-496 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:3737: finishing copy_from racing read to smithi16030472-317 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:3727: finishing write tid 4 to smithi16030472-496 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:3727: oid 496 updating version 0 to 1025 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:3727: oid 496 updating version 1025 to 1027 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:3727: oid 496 version 1027 is already newer than 1026 2024-09-21T18:10:16.315 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 496 v 1027 (ObjNum 1554 snap 0 seq_num 1554) dirty exists 2024-09-21T18:10:16.316 INFO:tasks.rados.rados.0.smithi160.stdout:3727: left oid 496 (ObjNum 1554 snap 0 seq_num 1554) 2024-09-21T18:10:16.317 INFO:tasks.rados.rados.0.smithi160.stdout:3737: finishing copy_from to smithi16030472-317 2024-09-21T18:10:16.317 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 997 (ObjNum 1509 snap 0 seq_num 1509) dirty exists 2024-09-21T18:10:16.397 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:16.393+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:16.401 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.401 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:10:16.432 INFO:tasks.rados.rados.0.smithi160.stdout:3727: done (12 left) 2024-09-21T18:10:16.432 INFO:tasks.rados.rados.0.smithi160.stdout:3735: done (11 left) 2024-09-21T18:10:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:3737: done (10 left) 2024-09-21T18:10:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:3748: done (9 left) 2024-09-21T18:10:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:3749: cache_try_flush oid 232 snap -1 2024-09-21T18:10:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:3750: write oid 325 current snap is 0 2024-09-21T18:10:16.433 INFO:tasks.rados.rados.0.smithi160.stdout:3750: seq_num 1562 ranges {652534=591303,1954403=298131} 2024-09-21T18:10:16.445 INFO:tasks.rados.rados.0.smithi160.stdout:3750: writing smithi16030472-325 from 652534 to 1243837 tid 1 2024-09-21T18:10:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:3750: writing smithi16030472-325 from 1954403 to 2252534 tid 2 2024-09-21T18:10:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:3751: cache_evict oid 66 snap -1 2024-09-21T18:10:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:3752: write oid 254 current snap is 0 2024-09-21T18:10:16.456 INFO:tasks.rados.rados.0.smithi160.stdout:3749: got (0) Success 2024-09-21T18:10:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 232 v 0 (ObjNum 1021 snap 0 seq_num 1021) dirty exists 2024-09-21T18:10:16.457 INFO:tasks.rados.rados.0.smithi160.stdout:3752: seq_num 1563 ranges {669341=666552,2012885=256456} 2024-09-21T18:10:16.470 INFO:tasks.rados.rados.0.smithi160.stdout:3752: writing smithi16030472-254 from 669341 to 1335893 tid 1 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3752: writing smithi16030472-254 from 2012885 to 2269341 tid 2 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3749: done (12 left) 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3753: copy_from oid 460 from oid 100 current snap is 0 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3754: read oid 387 snap -1 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3754: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3755: read oid 500 snap -1 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3755: expect (ObjNum 1383 snap 0 seq_num 1383) 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3756: write (excl) oid 102 current snap is 0 2024-09-21T18:10:16.480 INFO:tasks.rados.rados.0.smithi160.stdout:3756: seq_num 1564 ranges {686148=741801,1955014=331134} 2024-09-21T18:10:16.493 INFO:tasks.rados.rados.0.smithi160.stdout:3756: writing smithi16030472-102 from 686148 to 1427949 tid 1 2024-09-21T18:10:16.498 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:16.492+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:16.501 INFO:tasks.rados.rados.0.smithi160.stdout:3756: writing smithi16030472-102 from 1955014 to 2286148 tid 2 2024-09-21T18:10:16.501 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:16.553 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.553 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.563 INFO:tasks.rados.rados.0.smithi160.stdout:3743: got (0) Success 2024-09-21T18:10:16.563 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 335 v 0 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:3744: got (0) Success 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 0 (ObjNum 1301 snap 0 seq_num 1301) dirty exists 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:3743: done (15 left) 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:3744: done (14 left) 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:3757: write oid 265 current snap is 0 2024-09-21T18:10:16.564 INFO:tasks.rados.rados.0.smithi160.stdout:3757: seq_num 1565 ranges {702955=417050,1613496=649871} 2024-09-21T18:10:16.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:10:16.572 INFO:tasks.rados.rados.0.smithi160.stdout:3757: writing smithi16030472-265 from 702955 to 1120005 tid 1 2024-09-21T18:10:16.581 INFO:tasks.rados.rados.0.smithi160.stdout:3757: writing smithi16030472-265 from 1613496 to 2263367 tid 2 2024-09-21T18:10:16.581 INFO:tasks.rados.rados.0.smithi160.stdout:3758: copy_from oid 377 from oid 479 current snap is 0 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: finishing write tid 1 to smithi16030472-83 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: finishing write tid 2 to smithi16030472-83 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: finishing write tid 3 to smithi16030472-83 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: finishing write tid 4 to smithi16030472-83 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: oid 83 updating version 0 to 1117 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: oid 83 version 1117 is already newer than 1115 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: oid 83 version 1117 is already newer than 1116 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 83 v 1117 (ObjNum 1559 snap 0 seq_num 1559) dirty exists 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: left oid 83 (ObjNum 1559 snap 0 seq_num 1559) 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3745: done (15 left) 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout:3759: cache_flush oid 437 snap -1 2024-09-21T18:10:16.582 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:16.586 INFO:tasks.rados.rados.0.smithi160.stdout:3754: expect (ObjNum 386 snap 0 seq_num 386) 2024-09-21T18:10:16.599 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:16.595+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:16.663 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:16.695 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.815 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3759: got (2) No such file or directory 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: finishing write tid 1 to smithi16030472-60 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: finishing write tid 2 to smithi16030472-60 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: finishing write tid 3 to smithi16030472-60 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: finishing write tid 4 to smithi16030472-60 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: oid 60 updating version 0 to 1120 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: oid 60 version 1120 is already newer than 1118 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: oid 60 version 1120 is already newer than 1119 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 60 v 1120 (ObjNum 1560 snap 0 seq_num 1560) dirty exists 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3746: left oid 60 (ObjNum 1560 snap 0 seq_num 1560) 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3753: finishing copy_from to smithi16030472-460 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3753: got expected ENOENT (src dne) 2024-09-21T18:10:16.818 INFO:tasks.rados.rados.0.smithi160.stdout:3753: finishing copy_from racing read to smithi16030472-460 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3746: done (15 left) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3753: done (14 left) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3754: done (13 left) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3759: done (12 left) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3760: copy_from oid 147 from oid 292 current snap is 0 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3761: read oid 210 snap -1 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3761: expect (ObjNum 1037 snap 0 seq_num 1037) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3762: read oid 371 snap -1 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3762: expect (ObjNum 1138 snap 0 seq_num 1138) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3763: cache_flush oid 234 snap -1 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:3740: expect (ObjNum 960 snap 0 seq_num 960) 2024-09-21T18:10:16.819 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1229446 2024-09-21T18:10:16.820 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:16.813+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3739: finishing copy_from racing read to smithi16030472-429 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3740: done (15 left) 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3764: read oid 172 snap -1 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3764: expect (ObjNum 1290 snap 0 seq_num 1290) 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3739: finishing copy_from to smithi16030472-429 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 429 v 867 (ObjNum 1257 snap 0 seq_num 1257) dirty exists 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3750: finishing write tid 1 to smithi16030472-325 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3751: got (16) Device or resource busy 2024-09-21T18:10:16.824 INFO:tasks.rados.rados.0.smithi160.stdout:3755: expect (ObjNum 1383 snap 0 seq_num 1383) 2024-09-21T18:10:16.825 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:16.870 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1895497 2024-09-21T18:10:16.903 INFO:tasks.rados.rados.0.smithi160.stdout:3762: expect (ObjNum 1138 snap 0 seq_num 1138) 2024-09-21T18:10:16.917 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:16.911+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:16.956 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:16.956 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:16.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:10:16.987 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:16.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:17.018 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:17.015+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:17.029 INFO:tasks.rados.rados.0.smithi160.stdout:3758: finishing copy_from to smithi16030472-377 2024-09-21T18:10:17.030 INFO:tasks.rados.rados.0.smithi160.stdout:3758: got expected ENOENT (src dne) 2024-09-21T18:10:17.030 INFO:tasks.rados.rados.0.smithi160.stdout:3738: got (0) Success 2024-09-21T18:10:17.030 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 457 v 0 (ObjNum 1242 snap 0 seq_num 1242) dirty exists 2024-09-21T18:10:17.030 INFO:tasks.rados.rados.0.smithi160.stdout:3760: finishing copy_from racing read to smithi16030472-147 2024-09-21T18:10:17.030 INFO:tasks.rados.rados.0.smithi160.stdout:3761: expect (ObjNum 1037 snap 0 seq_num 1037) 2024-09-21T18:10:17.093 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:17.093 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:17.102 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:17.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:10:17.119 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:17.112+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3752: finishing write tid 1 to smithi16030472-254 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: finishing write tid 2 to smithi16030472-325 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: finishing write tid 3 to smithi16030472-325 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: finishing write tid 4 to smithi16030472-325 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: oid 325 updating version 0 to 1077 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: oid 325 updating version 1077 to 1079 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: oid 325 version 1079 is already newer than 1078 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 325 v 1079 (ObjNum 1562 snap 0 seq_num 1562) dirty exists 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3750: left oid 325 (ObjNum 1562 snap 0 seq_num 1562) 2024-09-21T18:10:17.139 INFO:tasks.rados.rados.0.smithi160.stdout:3764: expect (ObjNum 1290 snap 0 seq_num 1290) 2024-09-21T18:10:17.164 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:17.165 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:17.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:17.207 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:17.208 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:17.220 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:17.217+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:17.228 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:17.238 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:17.239 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:17.246 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:17.246 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:17.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:10:17.267 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:17.268 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:17.290 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:17.291 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:17.291 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3758: finishing copy_from racing read to smithi16030472-377 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3756: finishing write tid 1 to smithi16030472-102 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3756: finishing write tid 2 to smithi16030472-102 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3738: done (15 left) 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3739: done (14 left) 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3750: done (13 left) 2024-09-21T18:10:17.292 INFO:tasks.rados.rados.0.smithi160.stdout:3751: done (12 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3755: done (11 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3758: done (10 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3761: done (9 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3762: done (8 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3764: done (7 left) 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3765: cache_evict oid 199 snap -1 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: finishing write tid 3 to smithi16030472-102 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: finishing write tid 4 to smithi16030472-102 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: oid 102 updating version 0 to 1029 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: oid 102 version 1029 is already newer than 1028 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: oid 102 updating version 1029 to 1030 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 102 v 1030 (ObjNum 1564 snap 0 seq_num 1564) dirty exists 2024-09-21T18:10:17.293 INFO:tasks.rados.rados.0.smithi160.stdout:3756: left oid 102 (ObjNum 1564 snap 0 seq_num 1564) 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:3757: finishing write tid 1 to smithi16030472-265 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:3742: got (0) Success 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 260 v 0 (ObjNum 1549 snap 0 seq_num 1549) dirty exists 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:3763: got (0) Success 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 234 v 0 (ObjNum -1154728096 snap 21993 seq_num 304) clean dne 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:3760: finishing copy_from to smithi16030472-147 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 147 v 477 (ObjNum 640 snap 0 seq_num 640) dirty exists 2024-09-21T18:10:17.294 INFO:tasks.rados.rados.0.smithi160.stdout:3747: expect (ObjNum 1242 snap 0 seq_num 1242) 2024-09-21T18:10:17.311 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:17.311 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:17.321 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:17.315+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:17.333 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:17.355 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:17.355 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:17.362 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:17.377 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:17.377 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 184.17708172100001, 2024-09-21T18:10:17.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.395 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.63314417199999, 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:17.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:10:17.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.47222070800001, 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.401 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:10:17.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.47154611299999, 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.405 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:10:17.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:10:17.408 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.471519575, 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:10:17.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:10:17.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:10:17.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.412 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.46168583299999, 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:10:17.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:17.416 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.44927404500001, 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:10:17.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.422 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.440383975, 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.423 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:10:17.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:10:17.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:10:17.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:10:17.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:10:17.428 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.43957195799999, 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.429 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:10:17.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.439463746, 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:10:17.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:10:17.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.33919259999999, 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.440 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.33829634099999, 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:10:17.448 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.33781472800001, 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.449 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.33736230299999, 2024-09-21T18:10:17.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:10:17.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.33731197700001, 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.21157919699999, 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:10:17.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:10:17.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:10:17.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.20730114, 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.467 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:10:17.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:10:17.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:10:17.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:10:17.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.473 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.2069309, 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:17.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:10:17.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:10:17.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:10:17.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:10:17.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 166.20686117, 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 165.61528418, 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:10:17.484 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:17.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:17.490 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:17.491 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:17.491 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:17.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:10:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:17.493 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:17.493 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:17.494 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:17.495 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:17.489+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:3752: finishing write tid 2 to smithi16030472-254 2024-09-21T18:10:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:3752: finishing write tid 3 to smithi16030472-254 2024-09-21T18:10:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:3742: done (7 left) 2024-09-21T18:10:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:3747: done (6 left) 2024-09-21T18:10:17.495 INFO:tasks.rados.rados.0.smithi160.stdout:3756: done (5 left) 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3760: done (4 left) 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3763: done (3 left) 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3766: cache_evict oid 351 snap -1 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3752: finishing write tid 4 to smithi16030472-254 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3752: oid 254 updating version 0 to 1080 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3752: oid 254 updating version 1080 to 1081 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3752: oid 254 version 1081 is already newer than 1048 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 254 v 1081 (ObjNum 1563 snap 0 seq_num 1563) dirty exists 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3752: left oid 254 (ObjNum 1563 snap 0 seq_num 1563) 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3765: got (16) Device or resource busy 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: finishing write tid 2 to smithi16030472-265 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: finishing write tid 3 to smithi16030472-265 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: finishing write tid 4 to smithi16030472-265 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: oid 265 updating version 0 to 1049 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: oid 265 updating version 1049 to 1082 2024-09-21T18:10:17.496 INFO:tasks.rados.rados.0.smithi160.stdout:3757: oid 265 updating version 1082 to 1083 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 265 v 1083 (ObjNum 1565 snap 0 seq_num 1565) dirty exists 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3757: left oid 265 (ObjNum 1565 snap 0 seq_num 1565) 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3752: done (3 left) 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3757: done (2 left) 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3765: done (1 left) 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3767: read oid 91 snap -1 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3767: expect (ObjNum 1491 snap 0 seq_num 1491) 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3768: delete oid 28 current snap is 0 2024-09-21T18:10:17.497 INFO:tasks.rados.rados.0.smithi160.stdout:3766: got (0) Success 2024-09-21T18:10:17.498 INFO:tasks.rados.rados.0.smithi160.stdout:3767: expect (ObjNum 1491 snap 0 seq_num 1491) 2024-09-21T18:10:17.588 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2978945 2024-09-21T18:10:17.592 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:17.584+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:17.666 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:17.666 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:17.666 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:17.666 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:17.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:10:17.698 INFO:tasks.rados.rados.0.smithi160.stdout:3766: done (2 left) 2024-09-21T18:10:17.698 INFO:tasks.rados.rados.0.smithi160.stdout:3767: done (1 left) 2024-09-21T18:10:17.698 INFO:tasks.rados.rados.0.smithi160.stdout:3768: done (0 left) 2024-09-21T18:10:17.698 INFO:tasks.rados.rados.0.smithi160.stdout:3769: copy_from oid 493 from oid 114 current snap is 0 2024-09-21T18:10:17.698 INFO:tasks.rados.rados.0.smithi160.stdout:3770: delete oid 390 current snap is 0 2024-09-21T18:10:17.727 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:17.725+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:17.793 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:17.787+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:17.806 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:17.807 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:17.807 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:17.819 INFO:tasks.rados.rados.0.smithi160.stdout:3769: finishing copy_from racing read to smithi16030472-493 2024-09-21T18:10:17.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:10:17.821 INFO:tasks.rados.rados.0.smithi160.stdout:3769: finishing copy_from to smithi16030472-493 2024-09-21T18:10:17.822 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 493 v 986 (ObjNum 1493 snap 0 seq_num 1493) dirty exists 2024-09-21T18:10:17.894 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:17.891+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:17.902 INFO:tasks.rados.rados.0.smithi160.stdout:3769: done (1 left) 2024-09-21T18:10:17.902 INFO:tasks.rados.rados.0.smithi160.stdout:3770: done (0 left) 2024-09-21T18:10:17.903 INFO:tasks.rados.rados.0.smithi160.stdout:3771: read oid 31 snap -1 2024-09-21T18:10:17.903 INFO:tasks.rados.rados.0.smithi160.stdout:3771: expect (ObjNum 1553 snap 0 seq_num 1553) 2024-09-21T18:10:17.903 INFO:tasks.rados.rados.0.smithi160.stdout:3772: write (excl) oid 77 current snap is 0 2024-09-21T18:10:17.903 INFO:tasks.rados.rados.0.smithi160.stdout:3772: seq_num 1568 ranges {753376=759150,2188942=164434} 2024-09-21T18:10:17.927 INFO:tasks.rados.rados.0.smithi160.stdout:3772: writing smithi16030472-77 from 753376 to 1512526 tid 1 2024-09-21T18:10:17.941 INFO:tasks.rados.rados.0.smithi160.stdout:3772: writing smithi16030472-77 from 2188942 to 2353376 tid 2 2024-09-21T18:10:17.941 INFO:tasks.rados.rados.0.smithi160.stdout:3773: write (excl) oid 401 current snap is 0 2024-09-21T18:10:17.941 INFO:tasks.rados.rados.0.smithi160.stdout:3773: seq_num 1569 ranges {770183=434399,1847424=522759} 2024-09-21T18:10:17.958 INFO:tasks.rados.rados.0.smithi160.stdout:3773: writing smithi16030472-401 from 770183 to 1204582 tid 1 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3773: writing smithi16030472-401 from 1847424 to 2370183 tid 2 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3774: copy_from oid 237 from oid 333 current snap is 0 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3775: cache_try_flush oid 353 snap -1 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3776: cache_try_flush oid 84 snap -1 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3777: cache_try_flush oid 41 snap -1 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3778: delete oid 355 current snap is 0 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:3777: got (0) Success 2024-09-21T18:10:17.972 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 3131692952) clean dne 2024-09-21T18:10:17.991 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:17.991 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:17.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:17.991 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:10:17.992 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:17.995 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:17.993+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:18.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:10:18.096 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:18.088+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:18.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:10:18.128 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:18.128 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:18.129 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:18.129 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:18.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:10:18.212 INFO:tasks.rados.rados.0.smithi160.stdout:3771: expect (ObjNum 1553 snap 0 seq_num 1553) 2024-09-21T18:10:18.212 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:18.231 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:18.229+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:18.298 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:18.293+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:18.308 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:18.308 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:18.308 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:18.308 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:18.309 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:18.324 INFO:tasks.rados.rados.0.smithi160.stdout:3775: got (0) Success 2024-09-21T18:10:18.324 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 353 v 0 (ObjNum 976 snap 0 seq_num 976) dirty exists 2024-09-21T18:10:18.324 INFO:tasks.rados.rados.0.smithi160.stdout:3772: finishing write tid 1 to smithi16030472-77 2024-09-21T18:10:18.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:10:18.399 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:18.397+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.488 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:5 2.2 e27/25 2:5cea7035:::rbd_info:head v 27'1, mlcod=0'0)", 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 212.59089089299999, 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10179619099999999, 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896461+0000", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896462+0000", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1999999999999999e-07 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896476+0000", 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3929e-05 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896506+0000", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0256e-05 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.896526+0000", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9945999999999998e-05 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996805+0000", 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100278484 2024-09-21T18:10:18.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.996839+0000", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4065e-05 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998160+0000", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001320774 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:45.998258+0000", 2024-09-21T18:10:18.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8116999999999996e-05 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.4347.0:7 2.2 e28/25 2:5cea7035:::rbd_info:head v 28'2, mlcod=0'0)", 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 211.505498504, 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001802795, 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:18.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981854+0000", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981855+0000", 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981873+0000", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8597000000000002e-05 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981912+0000", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9181999999999999e-05 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.981937+0000", 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4159e-05 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982102+0000", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016559499999999999 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.982198+0000", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6124000000000003e-05 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983573+0000", 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001375081 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:18.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:06:46.983657+0000", 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3231000000000003e-05 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:18.497 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:18.500 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:18.495+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:18.502 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:10:18.514 INFO:tasks.rados.rados.0.smithi160.stdout:3772: finishing write tid 2 to smithi16030472-77 2024-09-21T18:10:18.600 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:18.595+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:18.614 INFO:tasks.rados.rados.0.smithi160.stdout:3772: finishing write tid 3 to smithi16030472-77 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3772: finishing write tid 4 to smithi16030472-77 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3772: oid 77 updating version 0 to 1095 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3772: oid 77 version 1095 is already newer than 1093 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3772: oid 77 version 1095 is already newer than 1094 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 77 v 1095 (ObjNum 1568 snap 0 seq_num 1568) dirty exists 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3772: left oid 77 (ObjNum 1568 snap 0 seq_num 1568) 2024-09-21T18:10:18.615 INFO:tasks.rados.rados.0.smithi160.stdout:3773: finishing write tid 1 to smithi16030472-401 2024-09-21T18:10:18.616 INFO:tasks.rados.rados.0.smithi160.stdout:3773: finishing write tid 2 to smithi16030472-401 2024-09-21T18:10:18.616 INFO:tasks.rados.rados.0.smithi160.stdout:3773: finishing write tid 3 to smithi16030472-401 2024-09-21T18:10:18.616 INFO:tasks.rados.rados.0.smithi160.stdout:3773: finishing write tid 4 to smithi16030472-401 2024-09-21T18:10:18.616 INFO:tasks.rados.rados.0.smithi160.stdout:3773: oid 401 updating version 0 to 1098 2024-09-21T18:10:18.617 INFO:tasks.rados.rados.0.smithi160.stdout:3773: oid 401 version 1098 is already newer than 1096 2024-09-21T18:10:18.617 INFO:tasks.rados.rados.0.smithi160.stdout:3773: oid 401 version 1098 is already newer than 1097 2024-09-21T18:10:18.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 401 v 1098 (ObjNum 1569 snap 0 seq_num 1569) dirty exists 2024-09-21T18:10:18.617 INFO:tasks.rados.rados.0.smithi160.stdout:3773: left oid 401 (ObjNum 1569 snap 0 seq_num 1569) 2024-09-21T18:10:18.618 INFO:tasks.rados.rados.0.smithi160.stdout:3776: got (0) Success 2024-09-21T18:10:18.618 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 84 v 0 (ObjNum 1217 snap 0 seq_num 1217) dirty exists 2024-09-21T18:10:18.622 INFO:tasks.rados.rados.0.smithi160.stdout:3771: done (7 left) 2024-09-21T18:10:18.622 INFO:tasks.rados.rados.0.smithi160.stdout:3772: done (6 left) 2024-09-21T18:10:18.622 INFO:tasks.rados.rados.0.smithi160.stdout:3773: done (5 left) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3775: done (4 left) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3776: done (3 left) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3777: done (2 left) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3778: done (1 left) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3779: cache_flush oid 235 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3780: read oid 58 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3780: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3781: cache_try_flush oid 484 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3782: cache_try_flush oid 425 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3783: cache_flush oid 26 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3784: read oid 194 snap -1 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3784: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:10:18.623 INFO:tasks.rados.rados.0.smithi160.stdout:3785: copy_from oid 101 from oid 417 current snap is 0 2024-09-21T18:10:18.624 INFO:tasks.rados.rados.0.smithi160.stdout:3786: write (excl) oid 6 current snap is 0 2024-09-21T18:10:18.624 INFO:tasks.rados.rados.0.smithi160.stdout:3786: seq_num 1571 ranges {403797=584897,1564388=660878} 2024-09-21T18:10:18.634 INFO:tasks.rados.rados.0.smithi160.stdout:3786: writing smithi16030472-6 from 403797 to 988694 tid 1 2024-09-21T18:10:18.639 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:1069 3.0 3.2888510c (undecoded) ondisk+write+ignore_overlay+enforce_snapc+known_if_redirected+supports_pool_eio e124)", 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:10:18.623106+0000", 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 0.015873857000000002, 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.015936207000000001, 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623106+0000", 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.640 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623106+0000", 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623110+0000", 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8979999999999998e-06 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.641 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623124+0000", 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3808999999999999e-05 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623152+0000", 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8303999999999999e-05 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:10:18.623168+0000", 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6169000000000001e-05 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:18.642 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.643 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.643 INFO:teuthology.orchestra.run.smithi040.stdout: ], 2024-09-21T18:10:18.643 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 1 2024-09-21T18:10:18.643 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:18.646 INFO:tasks.rados.rados.0.smithi160.stdout:3786: writing smithi16030472-6 from 1564388 to 2225266 tid 2 2024-09-21T18:10:18.646 INFO:tasks.rados.rados.0.smithi160.stdout:3787: cache_flush oid 172 snap -1 2024-09-21T18:10:18.646 INFO:tasks.rados.rados.0.smithi160.stdout:3788: cache_evict oid 459 snap -1 2024-09-21T18:10:18.647 INFO:tasks.rados.rados.0.smithi160.stdout:3789: write (excl) oid 180 current snap is 0 2024-09-21T18:10:18.647 INFO:tasks.rados.rados.0.smithi160.stdout:3789: seq_num 1572 ranges {420604=660146,1506517=404536} 2024-09-21T18:10:18.656 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:10:18.656 INFO:tasks.rados.rados.0.smithi160.stdout:3789: writing smithi16030472-180 from 420604 to 1080750 tid 1 2024-09-21T18:10:18.665 INFO:tasks.rados.rados.0.smithi160.stdout:3789: writing smithi16030472-180 from 1506517 to 1911053 tid 2 2024-09-21T18:10:18.665 INFO:tasks.rados.rados.0.smithi160.stdout:3790: write (excl) oid 317 current snap is 0 2024-09-21T18:10:18.665 INFO:tasks.rados.rados.0.smithi160.stdout:3780: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:18.665 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:18.701 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:18.697+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:18.802 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:18.797+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:18.832 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:18.832 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:18.832 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:18.832 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:18.832 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:18.846 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:10:18.902 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:18.896+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1541 3.0 3:1baed0db:::smithi16030472-343:head [write 3605976~142018] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93325252099999, 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99848867200000002, 2024-09-21T18:10:18.975 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037892+0000", 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.976 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.037893+0000", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7600000000000001e-06 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038093+0000", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019925800000000001 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038105+0000", 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2167000000000001e-05 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.977 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038116+0000", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1163e-05 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.257790+0000", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21967384300000001 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258058+0000", 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00026871099999999998 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.978 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.258460+0000", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00040108 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267904+0000", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094449430000000008 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036213+0000", 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76830861900000003 2024-09-21T18:10:18.979 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036302+0000", 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8858999999999997e-05 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036380+0000", 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8269000000000006e-05 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.980 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1542 3.0 3:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=7b,truncate 3747994] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.933012897, 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.998611841, 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.981 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038131+0000", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038132+0000", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2640000000000001e-06 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038137+0000", 2024-09-21T18:10:18.982 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.8230000000000002e-06 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038148+0000", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.023e-05 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038155+0000", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8960000000000003e-06 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.259909+0000", 2024-09-21T18:10:18.983 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.221753599 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260062+0000", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000153115 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.260264+0000", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000202282 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268357+0000", 2024-09-21T18:10:18.984 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0080926420000000006 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036520+0000", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76816251199999996 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036601+0000", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1606000000000004e-05 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036743+0000", 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000141872 2024-09-21T18:10:18.985 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:384 3.0 3:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93297749000001, 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.99894798100000004, 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.986 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038167+0000", 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2600000000000001e-07 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.987 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038171+0000", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.6669999999999998e-06 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038180+0000", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.2900000000000008e-06 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038190+0000", 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4800000000000007e-06 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.988 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.267995+0000", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22980497699999999 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268123+0000", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000127813 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036814+0000", 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.76869119100000005 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.989 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.036946+0000", 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013180200000000001 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.037115+0000", 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016893499999999999 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:385 3.0 3:1baed0db:::smithi16030472-343:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:18.990 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93294431699999, 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4260516839999999, 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038200+0000", 2024-09-21T18:10:18.991 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038201+0000", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.3300000000000001e-07 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038204+0000", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3469999999999999e-06 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038212+0000", 2024-09-21T18:10:18.992 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2770000000000005e-06 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038220+0000", 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.576e-06 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268202+0000", 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.22998260200000001 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.993 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268335+0000", 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236899999999999 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.463971+0000", 2024-09-21T18:10:18.994 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.195635797 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464107+0000", 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000136358 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464251+0000", 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000144525 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:18.995 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:386 3.0 3:1baed0db:::smithi16030472-343:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93291475699999, 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.431542903, 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.996 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038229+0000", 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038230+0000", 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8599999999999997e-07 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.997 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038234+0000", 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3839999999999999e-06 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038243+0000", 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.2290000000000003e-06 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:18.998 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038250+0000", 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3339999999999996e-06 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268423+0000", 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23017289399999999 2024-09-21T18:10:18.999 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268523+0000", 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000100519 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464385+0000", 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1958614110000001 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.000 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.464514+0000", 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012891500000000001 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.469772+0000", 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.005258431 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1544 3.0 3:2f6cf193:::smithi16030472-344:head [write 564801~646292,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93288510100001, 2024-09-21T18:10:19.001 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099256593, 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038259+0000", 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.002 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038260+0000", 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4e-07 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038701+0000", 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00044100600000000003 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.003 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038717+0000", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.645e-05 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038727+0000", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3430000000000002e-06 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268598+0000", 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.229871663 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.004 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.268811+0000", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021309500000000001 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.269038+0000", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022705499999999999 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386808+0000", 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.117769639 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.005 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137311+0000", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.75050274699999997 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137400+0000", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9559000000000001e-05 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137516+0000", 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000115196 2024-09-21T18:10:19.006 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:388 3.0 3:2f6cf193:::smithi16030472-344:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93239884499999, 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0992454519999999, 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.007 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038745+0000", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038746+0000", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1230000000000001e-06 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.008 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038750+0000", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.8e-06 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038762+0000", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1503e-05 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038769+0000", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6969999999999993e-06 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271815+0000", 2024-09-21T18:10:19.009 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.233045745 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.271959+0000", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014342200000000001 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137677+0000", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86571868200000002 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137836+0000", 2024-09-21T18:10:19.010 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015918200000000001 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.137991+0000", 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00015429800000000001 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:389 3.0 3:2f6cf193:::smithi16030472-344:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.93236525200001, 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.317745183, 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.011 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038779+0000", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038780+0000", 2024-09-21T18:10:19.012 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7700000000000004e-07 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038784+0000", 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.4159999999999997e-06 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.038795+0000", 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0720000000000001e-05 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.013 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138919+0000", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10012399199999999 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272041+0000", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13312209799999999 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272174+0000", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013295900000000001 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.014 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353568+0000", 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0813937309999999 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353825+0000", 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00025782099999999998 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.356524+0000", 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0026986689999999999 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.015 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1549 3.0 3:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.83214756000001, 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.099370379, 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.016 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.138997+0000", 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.017 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139000+0000", 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7060000000000002e-06 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139530+0000", 2024-09-21T18:10:19.018 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00052937600000000002 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139559+0000", 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9606000000000001e-05 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.019 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139574+0000", 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5096000000000001e-05 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.020 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272252+0000", 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13267720799999999 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272594+0000", 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00034217699999999999 2024-09-21T18:10:19.021 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.272819+0000", 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022532499999999999 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386816+0000", 2024-09-21T18:10:19.022 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.11399695999999999 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238196+0000", 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85138034600000001 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238290+0000", 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3843000000000003e-05 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.023 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238367+0000", 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6736000000000005e-05 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1550 3.0 3:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.831548656, 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.098996299, 2024-09-21T18:10:19.024 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139595+0000", 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.025 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.139597+0000", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.683e-06 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140263+0000", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00066587799999999996 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140283+0000", 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9643000000000001e-05 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.026 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140293+0000", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.084e-05 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275247+0000", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13495328400000001 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275382+0000", 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013479900000000001 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.027 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.275570+0000", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018849099999999999 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386826+0000", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.111256098 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238467+0000", 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85164084500000004 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.028 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238540+0000", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2991999999999995e-05 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238592+0000", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.1746000000000002e-05 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1551 3.0 3:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:19.029 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.830834039, 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0985966730000001, 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140310+0000", 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.030 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.140311+0000", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.26e-06 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141041+0000", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072968700000000002 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141058+0000", 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7376000000000001e-05 2024-09-21T18:10:19.031 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.141072+0000", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3884e-05 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277474+0000", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.13640205399999999 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277598+0000", 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012336899999999999 2024-09-21T18:10:19.032 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.277789+0000", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019139499999999999 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386836+0000", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.109046584 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238706+0000", 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.851870562 2024-09-21T18:10:19.033 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238797+0000", 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1225999999999997e-05 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238907+0000", 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109276 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.034 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1552 3.0 3:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.821581307, 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0896968810000001, 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.035 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149563+0000", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149567+0000", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.2760000000000002e-06 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149586+0000", 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8686999999999999e-05 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.036 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149617+0000", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1457000000000001e-05 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149635+0000", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8145999999999999e-05 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.279995+0000", 2024-09-21T18:10:19.037 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.130359591 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280127+0000", 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013236199999999999 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.280319+0000", 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00019220199999999999 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.038 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386849+0000", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.106529626 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.238984+0000", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85213517999999999 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239064+0000", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9992000000000003e-05 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.039 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239260+0000", 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000195362 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:394 3.0 3:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.82131660600001, 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0898034999999999, 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.040 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149827+0000", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.041 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149830+0000", 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.1430000000000001e-06 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149848+0000", 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7887e-05 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149874+0000", 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6622999999999999e-05 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.042 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149887+0000", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2563e-05 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380099+0000", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23021206999999999 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380240+0000", 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014140499999999999 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.043 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239327+0000", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85908694200000002 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239457+0000", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013033200000000001 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239631+0000", 2024-09-21T18:10:19.044 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000173535 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:395 3.0 3:05986648:::smithi16030472-345:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.821221917, 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.096881634, 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.045 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149922+0000", 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.046 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149924+0000", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.68e-06 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149930+0000", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.1670000000000004e-06 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149942+0000", 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1905999999999999e-05 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.047 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149951+0000", 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380301+0000", 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.23034991799999999 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.048 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380398+0000", 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7288999999999996e-05 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246570+0000", 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.86617164199999996 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.049 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246694+0000", 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012468900000000001 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246804+0000", 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000109363 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.050 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:396 3.0 3:05986648:::smithi16030472-345:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.82118062699999, 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.203476765, 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.051 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149963+0000", 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149964+0000", 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.5300000000000002e-07 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149974+0000", 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4339999999999998e-06 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.052 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149985+0000", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1306e-05 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.149993+0000", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.1200000000000002e-06 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380702+0000", 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.230708938 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.053 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380809+0000", 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000107116 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347015+0000", 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.966206012 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.054 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.347152+0000", 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00013657799999999999 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.353440+0000", 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0062883080000000003 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.055 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1564 3.0 3:03b3ee07:::smithi16030472-348:head [write 632029~547288,stat] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.80991072399999, 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.078667029, 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.056 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161233+0000", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161238+0000", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1729999999999997e-06 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161967+0000", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00072942999999999999 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.057 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.161998+0000", 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.0939999999999999e-05 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.162019+0000", 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.143e-05 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.380856+0000", 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21883624700000001 2024-09-21T18:10:19.058 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381065+0000", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00020931500000000001 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.381294+0000", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022882100000000001 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386858+0000", 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0055643280000000003 2024-09-21T18:10:19.059 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239725+0000", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85286647999999998 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239802+0000", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.7843000000000007e-05 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239900+0000", 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.8022000000000004e-05 2024-09-21T18:10:19.060 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(client.4876.0:1565 3.0 3:03b3ee07:::smithi16030472-348:head [write 1639134~749453] snapc 0=[] ondisk+write+ignore_cache+ignore_overlay+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.800794491, 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0698436069999999, 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.061 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170350+0000", 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.170353+0000", 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5930000000000002e-06 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.062 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171339+0000", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00098613499999999996 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171371+0000", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1476000000000002e-05 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171387+0000", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5979999999999999e-05 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.063 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383424+0000", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.212036741 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383549+0000", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000125928 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for subops from 9", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.383738+0000", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00018882600000000001 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "op_commit", 2024-09-21T18:10:19.064 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387510+0000", 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0037716360000000001 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.239977+0000", 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85246694000000001 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240051+0000", 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3763000000000005e-05 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.065 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240193+0000", 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00014258900000000001 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:399 3.0 3:03b3ee07:::smithi16030472-348:snapdir [list-snaps] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.799731124, 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.06909447, 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.066 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171413+0000", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.067 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171415+0000", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7269999999999999e-06 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171420+0000", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.7200000000000003e-06 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171432+0000", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1409999999999999e-05 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171441+0000", 2024-09-21T18:10:19.068 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9560000000000003e-06 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386665+0000", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215224149 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386788+0000", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012277300000000001 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240261+0000", 2024-09-21T18:10:19.069 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85347283100000004 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240373+0000", 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112135 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.240507+0000", 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000134769 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.070 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:400 3.0 3:03b3ee07:::smithi16030472-348:head [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.799690814, 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0750442840000001, 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.071 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171453+0000", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171454+0000", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3600000000000002e-07 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171460+0000", 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.5230000000000003e-06 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.072 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171472+0000", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2473999999999999e-05 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.171481+0000", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4400000000000005e-06 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.386921+0000", 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.215440094 2024-09-21T18:10:19.073 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.387051+0000", 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000129868 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241242+0000", 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.85419095899999997 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.074 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.241446+0000", 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000204005 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.246498+0000", 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0050519850000000002 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_op(osd.4.21:416 3.0 3:385a7674:::smithi16030472-358:head [copy-get max 8388608 in=29b] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:19.075 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 167.037052972, 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.84581704899999999, 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "started", 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.076 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934091+0000", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934092+0000", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4300000000000002e-07 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934097+0000", 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.0429999999999998e-06 2024-09-21T18:10:19.077 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934108+0000", 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1416999999999999e-05 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.934116+0000", 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.6869999999999994e-06 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.078 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475693+0000", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.54157666800000004 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.475909+0000", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00021642800000000001 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777183+0000", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30127366300000002 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.079 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.777313+0000", 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00012977600000000001 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:32.779908+0000", 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0025955240000000001 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.080 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:3784: expect (ObjNum 326 snap 0 seq_num 326) 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3276898 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 338477 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:3785: finishing copy_from to smithi16030472-101 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:3785: got expected ENOENT (src dne) 2024-09-21T18:10:19.083 INFO:tasks.rados.rados.0.smithi160.stdout:3785: finishing copy_from racing read to smithi16030472-101 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3774: finishing copy_from racing read to smithi16030472-237 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3774: finishing copy_from to smithi16030472-237 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 237 v 960 (ObjNum 1371 snap 0 seq_num 1371) dirty exists 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3789: finishing write tid 1 to smithi16030472-180 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3786: finishing write tid 1 to smithi16030472-6 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3781: got (0) Success 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 484 v 0 (ObjNum 1526 snap 0 seq_num 1526) dirty exists 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3779: got (0) Success 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 235 v 0 (ObjNum 849 snap 0 seq_num 849) dirty exists 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3788: got (16) Device or resource busy 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3786: finishing write tid 2 to smithi16030472-6 2024-09-21T18:10:19.084 INFO:tasks.rados.rados.0.smithi160.stdout:3790: seq_num 1573 ranges {437411=735395,1964999=472412} 2024-09-21T18:10:19.085 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:10:19.085 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:19.077+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:19.090 INFO:tasks.rados.rados.0.smithi160.stdout:3790: writing smithi16030472-317 from 437411 to 1172806 tid 1 2024-09-21T18:10:19.100 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:10:19.109 INFO:tasks.rados.rados.0.smithi160.stdout:3790: writing smithi16030472-317 from 1964999 to 2437411 tid 2 2024-09-21T18:10:19.109 INFO:tasks.rados.rados.0.smithi160.stdout:3774: done (12 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3779: done (11 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3780: done (10 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3781: done (9 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3784: done (8 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3785: done (7 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3788: done (6 left) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3791: read oid 407 snap -1 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3791: expect (ObjNum 814 snap 0 seq_num 814) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3792: read oid 69 snap -1 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3792: expect (ObjNum 1517 snap 0 seq_num 1517) 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3793: write (excl) oid 209 current snap is 0 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3786: finishing write tid 3 to smithi16030472-6 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3786: finishing write tid 4 to smithi16030472-6 2024-09-21T18:10:19.110 INFO:tasks.rados.rados.0.smithi160.stdout:3786: oid 6 updating version 0 to 1103 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3786: oid 6 version 1103 is already newer than 1101 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3786: oid 6 version 1103 is already newer than 1102 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 6 v 1103 (ObjNum 1571 snap 0 seq_num 1571) dirty exists 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3786: left oid 6 (ObjNum 1571 snap 0 seq_num 1571) 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: finishing write tid 2 to smithi16030472-180 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: finishing write tid 3 to smithi16030472-180 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: finishing write tid 4 to smithi16030472-180 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: oid 180 updating version 0 to 1121 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: oid 180 updating version 1121 to 1122 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: oid 180 updating version 1122 to 1123 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 1123 (ObjNum 1572 snap 0 seq_num 1572) dirty exists 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3789: left oid 180 (ObjNum 1572 snap 0 seq_num 1572) 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3783: got (0) Success 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 26 v 0 (ObjNum 1504 snap 0 seq_num 1504) dirty exists 2024-09-21T18:10:19.111 INFO:tasks.rados.rados.0.smithi160.stdout:3782: got (0) Success 2024-09-21T18:10:19.112 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 425 v 0 (ObjNum 1418 snap 0 seq_num 1418) dirty exists 2024-09-21T18:10:19.112 INFO:tasks.rados.rados.0.smithi160.stdout:3793: seq_num 1574 ranges {454218=526997,1739834=408205} 2024-09-21T18:10:19.124 INFO:tasks.rados.rados.0.smithi160.stdout:3793: writing smithi16030472-209 from 454218 to 981215 tid 1 2024-09-21T18:10:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:3793: writing smithi16030472-209 from 1739834 to 2148039 tid 2 2024-09-21T18:10:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:3782: done (8 left) 2024-09-21T18:10:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:3783: done (7 left) 2024-09-21T18:10:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:3786: done (6 left) 2024-09-21T18:10:19.140 INFO:tasks.rados.rados.0.smithi160.stdout:3789: done (5 left) 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:3794: read oid 98 snap -1 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:3794: expect deleted 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:3795: write (excl) oid 106 current snap is 0 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:3787: got (0) Success 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 172 v 0 (ObjNum 1290 snap 0 seq_num 1290) dirty exists 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:3791: expect (ObjNum 814 snap 0 seq_num 814) 2024-09-21T18:10:19.141 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:19.184 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:19.181+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:19.220 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:19.220 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:19.220 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:19.220 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:19.235 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:10:19.255 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 763634 2024-09-21T18:10:19.285 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:19.279+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:19.297 INFO:tasks.rados.rados.0.smithi160.stdout:3792: expect (ObjNum 1517 snap 0 seq_num 1517) 2024-09-21T18:10:19.360 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:19.360 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:19.360 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:19.360 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:19.360 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:19.374 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:10:19.387 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:19.385+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:19.444 INFO:tasks.rados.rados.0.smithi160.stdout:3790: finishing write tid 1 to smithi16030472-317 2024-09-21T18:10:19.444 INFO:tasks.rados.rados.0.smithi160.stdout:3793: finishing write tid 1 to smithi16030472-209 2024-09-21T18:10:19.444 INFO:tasks.rados.rados.0.smithi160.stdout:3793: finishing write tid 2 to smithi16030472-209 2024-09-21T18:10:19.444 INFO:tasks.rados.rados.0.smithi160.stdout:3793: finishing write tid 3 to smithi16030472-209 2024-09-21T18:10:19.444 INFO:tasks.rados.rados.0.smithi160.stdout:3790: finishing write tid 2 to smithi16030472-317 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3793: finishing write tid 4 to smithi16030472-209 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3793: oid 209 updating version 0 to 1105 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3793: oid 209 updating version 1105 to 1106 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3793: oid 209 updating version 1106 to 1107 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 209 v 1107 (ObjNum 1574 snap 0 seq_num 1574) dirty exists 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3793: left oid 209 (ObjNum 1574 snap 0 seq_num 1574) 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3790: finishing write tid 3 to smithi16030472-317 2024-09-21T18:10:19.445 INFO:tasks.rados.rados.0.smithi160.stdout:3795: seq_num 1575 ranges {471025=602246,1798316=668216} 2024-09-21T18:10:19.461 INFO:tasks.rados.rados.0.smithi160.stdout:3795: writing smithi16030472-106 from 471025 to 1073271 tid 1 2024-09-21T18:10:19.484 INFO:tasks.rados.rados.0.smithi160.stdout:3795: writing smithi16030472-106 from 1798316 to 2466532 tid 2 2024-09-21T18:10:19.484 INFO:tasks.rados.rados.0.smithi160.stdout:3787: done (6 left) 2024-09-21T18:10:19.485 INFO:tasks.rados.rados.0.smithi160.stdout:3791: done (5 left) 2024-09-21T18:10:19.485 INFO:tasks.rados.rados.0.smithi160.stdout:3792: done (4 left) 2024-09-21T18:10:19.485 INFO:tasks.rados.rados.0.smithi160.stdout:3793: done (3 left) 2024-09-21T18:10:19.485 INFO:tasks.rados.rados.0.smithi160.stdout:3794: done (2 left) 2024-09-21T18:10:19.485 INFO:tasks.rados.rados.0.smithi160.stdout:3796: delete oid 157 current snap is 0 2024-09-21T18:10:19.487 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:19.483+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:19.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:3790: finishing write tid 4 to smithi16030472-317 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:3790: oid 317 updating version 0 to 1032 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:3790: oid 317 version 1032 is already newer than 1031 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:3790: oid 317 updating version 1032 to 1033 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 1033 (ObjNum 1573 snap 0 seq_num 1573) dirty exists 2024-09-21T18:10:19.504 INFO:tasks.rados.rados.0.smithi160.stdout:3790: left oid 317 (ObjNum 1573 snap 0 seq_num 1573) 2024-09-21T18:10:19.549 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:19.549 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [ 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:22 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'12, mlcod=25'9)", 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 215.247872839, 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30324315200000002, 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.550 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299052+0000", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299055+0000", 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7199999999999998e-06 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.551 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299148+0000", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.3317000000000001e-05 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299172+0000", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.4015999999999998e-05 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.299186+0000", 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.3696000000000001e-05 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.552 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399469+0000", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10028379799999999 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.399509+0000", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.9922000000000002e-05 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602204+0000", 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20269451599999999 2024-09-21T18:10:19.553 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602295+0000", 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1167000000000003e-05 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:23 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'13, mlcod=25'9)", 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 215.24665877999999, 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30213793, 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.554 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300266+0000", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.555 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300269+0000", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6970000000000002e-06 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300386+0000", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011777000000000001 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300413+0000", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6942999999999999e-05 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.556 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.300429+0000", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5389000000000001e-05 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500333+0000", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19990381600000001 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.500365+0000", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.2762999999999998e-05 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.557 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602304+0000", 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.101938602 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602404+0000", 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.9950000000000004e-05 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:24 1.0 e25/23 1:30306672:devicehealth::main.db.0000000000000000:head v 25'14, mlcod=25'9)", 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 215.24530786, 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30114570299999999, 2024-09-21T18:10:19.558 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301617+0000", 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.559 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301620+0000", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6390000000000001e-06 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301710+0000", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.0210999999999994e-05 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.301737+0000", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7005999999999999e-05 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.560 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.401832+0000", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100095193 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601189+0000", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.19935719499999999 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.601227+0000", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7945999999999999e-05 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602650+0000", 2024-09-21T18:10:19.561 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0014225469999999999 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:44.602763+0000", 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000112966 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4263.0:116 1.0 e28/23 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 28'86, mlcod=28'80)", 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 208.95772722300001, 2024-09-21T18:10:19.562 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30302275299999998, 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589198+0000", 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.563 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589201+0000", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4000000000000001e-06 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589216+0000", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5243e-05 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589236+0000", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.0293e-05 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.589251+0000", 2024-09-21T18:10:19.564 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.4049999999999999e-05 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689502+0000", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100251389 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.689535+0000", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3312000000000002e-05 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892103+0000", 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20256780299999999 2024-09-21T18:10:19.565 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:06:50.892220+0000", 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00011726300000000001 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1479 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'219, mlcod=50'209)", 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.55848798299999, 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31054149399999997, 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.566 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988437+0000", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.567 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988438+0000", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.403e-06 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988805+0000", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00036673000000000001 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:30.988821+0000", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.562e-05 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.088966+0000", 2024-09-21T18:10:19.568 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100145306 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089478+0000", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00051220200000000001 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089516+0000", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7932999999999997e-05 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298756+0000", 2024-09-21T18:10:19.569 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.209240389 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.298978+0000", 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00022191100000000001 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1480 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'220, mlcod=50'210)", 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45791859100001, 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31319552099999998, 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.570 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089006+0000", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.571 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089008+0000", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2970000000000002e-06 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089313+0000", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00030408699999999998 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089335+0000", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2172999999999999e-05 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089347+0000", 2024-09-21T18:10:19.572 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1783000000000001e-05 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191595+0000", 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.102248617 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.191637+0000", 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1556e-05 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.573 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402100+0000", 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.210463552 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402202+0000", 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000101456 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1481 3.2 e51/31 3:6a5c42b4:::smithi16030472-331:head v 51'221, mlcod=50'210)", 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:19.574 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.457560524, 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31291549600000002, 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089364+0000", 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.575 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089365+0000", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.17e-06 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089373+0000", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3000000000000004e-06 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089384+0000", 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1123e-05 2024-09-21T18:10:19.576 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089391+0000", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.2130000000000002e-06 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.192974+0000", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.10358305499999999 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.193008+0000", 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4343e-05 2024-09-21T18:10:19.577 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402211+0000", 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20920282300000001 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402280+0000", 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.8468999999999998e-05 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1498 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'222, mlcod=50'210)", 2024-09-21T18:10:19.578 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45752369100001, 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31295912300000001, 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089401+0000", 2024-09-21T18:10:19.579 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089402+0000", 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8299999999999996e-07 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089800+0000", 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00039861699999999998 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.580 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089835+0000", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4209999999999999e-05 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089844+0000", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7960000000000007e-06 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293518+0000", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20367395099999999 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.581 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.293554+0000", 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5989999999999999e-05 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402288+0000", 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.108733963 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402360+0000", 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.1812999999999998e-05 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.582 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1499 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'223, mlcod=50'210)", 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45706395900001, 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31712664899999998, 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.583 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089861+0000", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.089862+0000", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1370000000000001e-06 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090260+0000", 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000397613 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.584 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090272+0000", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.2317000000000001e-05 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090280+0000", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.9929999999999995e-06 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295622+0000", 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.205341673 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.585 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.295682+0000", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.0426e-05 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406672+0000", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110989826 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.406987+0000", 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00031566400000000001 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.586 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1500 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'224, mlcod=50'210)", 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45663243199999, 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32063271399999999, 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:19.587 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090292+0000", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.090293+0000", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.9700000000000005e-07 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091184+0000", 2024-09-21T18:10:19.588 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00089114800000000003 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091202+0000", 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7767000000000001e-05 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091211+0000", 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.1500000000000005e-06 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.589 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399178+0000", 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30796689700000002 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.399214+0000", 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5769000000000003e-05 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410861+0000", 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.01164713 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.590 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410925+0000", 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.3955999999999995e-05 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1501 3.2 e51/31 3:70fc244d:::smithi16030472-335:head v 51'225, mlcod=50'210)", 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45569581699999, 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31974105200000003, 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.591 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091229+0000", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091230+0000", 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0729999999999999e-06 2024-09-21T18:10:19.592 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091240+0000", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0030000000000001e-05 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091251+0000", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0726e-05 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091257+0000", 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7270000000000001e-06 2024-09-21T18:10:19.593 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401467+0000", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31020984400000001 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.401526+0000", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.8423e-05 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410933+0000", 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0094069389999999996 2024-09-21T18:10:19.594 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410970+0000", 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.7289999999999997e-05 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1533 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'226, mlcod=50'210)", 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.45565713400001, 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31973923900000001, 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.595 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.596 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.091268+0000", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.9999999999999997e-07 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092483+0000", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0012143519999999999 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092508+0000", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.548e-05 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.597 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092524+0000", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.6337e-05 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402446+0000", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.30992114300000001 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.402493+0000", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.7352000000000003e-05 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.598 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.410975+0000", 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0084820120000000006 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.411007+0000", 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.1962999999999999e-05 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1534 3.2 e51/31 3:7b415eca:::smithi16030472-342:head v 51'227, mlcod=50'210)", 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.454370482, 2024-09-21T18:10:19.599 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.32111261000000002, 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092554+0000", 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.600 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.092556+0000", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9630000000000001e-06 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093147+0000", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00059045099999999997 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093169+0000", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.2560000000000001e-05 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.601 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.093188+0000", 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.8329999999999999e-05 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404542+0000", 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31135426300000002 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.404584+0000", 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1810000000000001e-05 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.602 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413609+0000", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0090248529999999993 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413667+0000", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 5.838e-05 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1559 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'228, mlcod=50'210)", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:19.603 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.35222601800001, 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21900686899999999, 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194699+0000", 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.604 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.194702+0000", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.546e-06 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195356+0000", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00065332499999999996 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195382+0000", 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.6849e-05 2024-09-21T18:10:19.605 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195397+0000", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.434e-05 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407103+0000", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21170646000000001 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.407198+0000", 2024-09-21T18:10:19.606 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.4806e-05 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413672+0000", 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0064741729999999997 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413706+0000", 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3370000000000001e-05 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.607 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:1560 3.2 e51/31 3:51bfab75:::smithi16030472-347:head v 51'229, mlcod=50'210)", 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 168.35150222499999, 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21831868500000001, 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.608 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195423+0000", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.195424+0000", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.513e-06 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196045+0000", 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00062145500000000003 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.609 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196063+0000", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.7249999999999999e-05 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.196072+0000", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 9.7000000000000003e-06 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409146+0000", 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21307404799999999 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.610 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.409180+0000", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.3197000000000001e-05 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413712+0000", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0045320760000000003 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:31.413741+0000", 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.9445999999999999e-05 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.611 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:2182 3.2 e53/31 3:47b73fcf:::smithi16030472-484:head v 53'313, mlcod=53'311)", 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 160.22579198599999, 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.25991207100000002, 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.612 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321133+0000", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321136+0000", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.976e-06 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321602+0000", 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00046601999999999999 2024-09-21T18:10:19.613 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321629+0000", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7432000000000001e-05 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.321643+0000", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.417e-05 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421916+0000", 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100272444 2024-09-21T18:10:19.614 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.421951+0000", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.4922999999999998e-05 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.580934+0000", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.15898279700000001 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:07:39.581045+0000", 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000111309 2024-09-21T18:10:19.615 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:3329 4.3 e69/33 4:d9b5818f:::smithi16030472-117:head v 69'160, mlcod=68'156)", 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 135.585997499, 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21610710699999999, 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.616 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960927+0000", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.960930+0000", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.7539999999999998e-06 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.617 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966806+0000", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0058756779999999996 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966835+0000", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.889e-05 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.966850+0000", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.5523e-05 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.618 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967019+0000", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00016870200000000001 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:03.967061+0000", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.1950000000000003e-05 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.176950+0000", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20988915999999999 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.619 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:08:04.177034+0000", 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.4450000000000006e-05 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8700 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'653, mlcod=116'650)", 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.675057107000001, 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.31160568599999999, 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.620 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871868+0000", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.871871+0000", 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 3.5370000000000002e-06 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.621 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.872549+0000", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00067808400000000002 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.872577+0000", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 2.8140000000000002e-05 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.972708+0000", 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100130698 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.622 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.073005+0000", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.100297214 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.073072+0000", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7106000000000004e-05 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.183367+0000", 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.110295064 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.623 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.183473+0000", 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000105843 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8702 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'655, mlcod=116'651)", 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.573445011, 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21321606600000001, 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.624 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973480+0000", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.973481+0000", 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0669999999999999e-06 2024-09-21T18:10:19.625 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974481+0000", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.00099984100000000001 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974500+0000", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.9717000000000001e-05 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974511+0000", 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.0723000000000001e-05 2024-09-21T18:10:19.626 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.179699+0000", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20518751700000001 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.179742+0000", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 4.3769000000000001e-05 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.186579+0000", 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0068362400000000004 2024-09-21T18:10:19.627 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.186696+0000", 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.000117192 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "description": "osd_repop(client.4876.0:8703 4.3 e116/33 4:d903dcdd:::smithi16030472-36:head v 116'656, mlcod=116'651)", 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "initiated_at": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "age": 25.572396875999999, 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.21259102999999999, 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "continuous": false, 2024-09-21T18:10:19.628 INFO:teuthology.orchestra.run.smithi040.stdout: "type_data": { 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "events": [ 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "initiated", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "header_read", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974528+0000", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "throttled", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974529+0000", 2024-09-21T18:10:19.629 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 1.1030000000000001e-06 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "all_read", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974536+0000", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.5129999999999999e-06 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "dispatched", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974544+0000", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.8199999999999997e-06 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "queued_for_pg", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:53.974551+0000", 2024-09-21T18:10:19.630 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 6.7039999999999997e-06 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "reached_pg", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.182573+0000", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.20802183899999999 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "started", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.182647+0000", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 7.3903e-05 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "commit_sent", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.187031+0000", 2024-09-21T18:10:19.631 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 0.0043840160000000001 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: }, 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: { 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: "event": "done", 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: "time": "2024-09-21T18:09:54.187119+0000", 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 8.8132000000000003e-05 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: } 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout: ] 2024-09-21T18:10:19.632 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:19.634 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:10:19.634 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:19.632+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:19.654 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:19.675 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:19.675 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:19.699 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:19.699 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:19.720 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:19.720 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:19.734 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:19.732+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:19.737 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:19.738 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:19.757 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:19.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:19.771 INFO:tasks.rados.rados.0.smithi160.stdout:3795: finishing write tid 1 to smithi16030472-106 2024-09-21T18:10:19.779 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:19.779 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:19.779 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:19.800 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:19.801 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:19.807 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:19.807 INFO:teuthology.orchestra.run.smithi040.stdout: "num_ops": 0 2024-09-21T18:10:19.807 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:19.821 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:10:19.822 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:19.835 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:19.833+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:19.846 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:19.846 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:19.869 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:19.889 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:19.889 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:19.890 INFO:tasks.rados.rados.0.smithi160.stdout:3790: done (2 left) 2024-09-21T18:10:19.890 INFO:tasks.rados.rados.0.smithi160.stdout:3796: done (1 left) 2024-09-21T18:10:19.890 INFO:tasks.rados.rados.0.smithi160.stdout:3797: write (excl) oid 180 current snap is 0 2024-09-21T18:10:19.890 INFO:tasks.rados.rados.0.smithi160.stdout:3797: seq_num 1577 ranges {504639=752744,1798927=671885} 2024-09-21T18:10:19.910 INFO:tasks.rados.rados.0.smithi160.stdout:3797: writing smithi16030472-180 from 504639 to 1257383 tid 1 2024-09-21T18:10:19.911 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:19.930 INFO:tasks.rados.rados.0.smithi160.stdout:3797: writing smithi16030472-180 from 1798927 to 2470812 tid 2 2024-09-21T18:10:19.930 INFO:tasks.rados.rados.0.smithi160.stdout:3798: copy_from oid 404 from oid 421 current snap is 0 2024-09-21T18:10:19.930 INFO:tasks.rados.rados.0.smithi160.stdout:3799: cache_try_flush oid 264 snap -1 2024-09-21T18:10:19.930 INFO:tasks.rados.rados.0.smithi160.stdout:3800: read oid 136 snap -1 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3800: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3801: copy_from oid 365 from oid 152 current snap is 0 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3802: read oid 466 snap -1 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3802: expect deleted 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3803: cache_flush oid 221 snap -1 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3804: write (excl) oid 74 current snap is 0 2024-09-21T18:10:19.931 INFO:tasks.rados.rados.0.smithi160.stdout:3804: seq_num 1578 ranges {521446=427993,1457409=415543,2301012=220434} 2024-09-21T18:10:19.934 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:19.935 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:19.933+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:19.946 INFO:tasks.rados.rados.0.smithi160.stdout:3804: writing smithi16030472-74 from 521446 to 949439 tid 1 2024-09-21T18:10:19.946 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:19.946 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:19.956 INFO:tasks.rados.rados.0.smithi160.stdout:3804: writing smithi16030472-74 from 1457409 to 1872952 tid 2 2024-09-21T18:10:19.960 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:10:19.963 INFO:tasks.rados.rados.0.smithi160.stdout:3804: writing smithi16030472-74 from 2301012 to 2521446 tid 3 2024-09-21T18:10:19.963 INFO:tasks.rados.rados.0.smithi160.stdout:3805: cache_try_flush oid 417 snap -1 2024-09-21T18:10:19.963 INFO:tasks.rados.rados.0.smithi160.stdout:3806: read oid 460 snap -1 2024-09-21T18:10:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:3806: expect (ObjNum 1481 snap 0 seq_num 1481) 2024-09-21T18:10:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:3807: read oid 306 snap -1 2024-09-21T18:10:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:3807: expect (ObjNum 976 snap 0 seq_num 976) 2024-09-21T18:10:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:3808: cache_flush oid 325 snap -1 2024-09-21T18:10:19.964 INFO:tasks.rados.rados.0.smithi160.stdout:3800: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:10:20.036 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:20.033+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:20.058 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:20.058 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:20.064 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:20.073 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2226373 2024-09-21T18:10:20.089 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.089 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.105 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:10:20.134 INFO:tasks.rados.rados.0.smithi160.stdout:3807: expect (ObjNum 976 snap 0 seq_num 976) 2024-09-21T18:10:20.137 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:20.133+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:20.193 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:20.225 INFO:tasks.rados.rados.0.smithi160.stdout:3803: got (0) Success 2024-09-21T18:10:20.225 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 221 v 0 (ObjNum 1192 snap 0 seq_num 1192) dirty exists 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: finishing write tid 2 to smithi16030472-106 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: finishing write tid 3 to smithi16030472-106 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: finishing write tid 4 to smithi16030472-106 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: oid 106 updating version 0 to 1096 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: oid 106 updating version 1096 to 1126 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: oid 106 updating version 1126 to 1127 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 106 v 1127 (ObjNum 1575 snap 0 seq_num 1575) dirty exists 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3795: left oid 106 (ObjNum 1575 snap 0 seq_num 1575) 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3797: finishing write tid 1 to smithi16030472-180 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3808: got (0) Success 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 325 v 0 (ObjNum 1562 snap 0 seq_num 1562) dirty exists 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3797: finishing write tid 2 to smithi16030472-180 2024-09-21T18:10:20.226 INFO:tasks.rados.rados.0.smithi160.stdout:3797: finishing write tid 3 to smithi16030472-180 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: finishing write tid 4 to smithi16030472-180 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: oid 180 updating version 0 to 1130 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: oid 180 version 1130 is already newer than 1128 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: oid 180 version 1130 is already newer than 1129 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 1130 (ObjNum 1577 snap 0 seq_num 1577) dirty exists 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: left oid 180 (ObjNum 1577 snap 0 seq_num 1577) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3798: finishing copy_from racing read to smithi16030472-404 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3795: done (12 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3797: done (11 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3800: done (10 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3803: done (9 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3807: done (8 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3808: done (7 left) 2024-09-21T18:10:20.227 INFO:tasks.rados.rados.0.smithi160.stdout:3809: write (excl) oid 340 current snap is 0 2024-09-21T18:10:20.228 INFO:tasks.rados.rados.0.smithi160.stdout:3809: seq_num 1579 ranges {538253=503242,1515891=675554} 2024-09-21T18:10:20.228 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.228 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.237 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:20.237+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:20.238 INFO:tasks.rados.rados.0.smithi160.stdout:3809: writing smithi16030472-340 from 538253 to 1041495 tid 1 2024-09-21T18:10:20.240 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:10:20.252 INFO:tasks.rados.rados.0.smithi160.stdout:3809: writing smithi16030472-340 from 1515891 to 2191445 tid 2 2024-09-21T18:10:20.252 INFO:tasks.rados.rados.0.smithi160.stdout:3810: read oid 145 snap -1 2024-09-21T18:10:20.252 INFO:tasks.rados.rados.0.smithi160.stdout:3810: expect deleted 2024-09-21T18:10:20.252 INFO:tasks.rados.rados.0.smithi160.stdout:3811: write (excl) oid 286 current snap is 0 2024-09-21T18:10:20.252 INFO:tasks.rados.rados.0.smithi160.stdout:3811: seq_num 1580 ranges {555060=578491,1858020=419212} 2024-09-21T18:10:20.266 INFO:tasks.rados.rados.0.smithi160.stdout:3811: writing smithi16030472-286 from 555060 to 1133551 tid 1 2024-09-21T18:10:20.279 INFO:tasks.rados.rados.0.smithi160.stdout:3811: writing smithi16030472-286 from 1858020 to 2277232 tid 2 2024-09-21T18:10:20.279 INFO:tasks.rados.rados.0.smithi160.stdout:3812: write oid 381 current snap is 0 2024-09-21T18:10:20.279 INFO:tasks.rados.rados.0.smithi160.stdout:3812: seq_num 1581 ranges {571867=653740,1916502=655365} 2024-09-21T18:10:20.290 INFO:tasks.rados.rados.0.smithi160.stdout:3812: writing smithi16030472-381 from 571867 to 1225607 tid 1 2024-09-21T18:10:20.302 INFO:tasks.rados.rados.0.smithi160.stdout:3812: writing smithi16030472-381 from 1916502 to 2571867 tid 2 2024-09-21T18:10:20.302 INFO:tasks.rados.rados.0.smithi160.stdout:3813: copy_from oid 16 from oid 207 current snap is 0 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout:3814: cache_flush oid 288 snap -1 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout:3815: read oid 109 snap -1 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout:3815: expect (ObjNum 88 snap 0 seq_num 88) 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout:3816: cache_try_flush oid 424 snap -1 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout:3817: copy_from oid 263 from oid 191 current snap is 0 2024-09-21T18:10:20.303 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout:3798: finishing copy_from to smithi16030472-404 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 404 v 764 (ObjNum 1119 snap 0 seq_num 1119) dirty exists 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout:3798: done (15 left) 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout:3818: read oid 95 snap -1 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout:3818: expect (ObjNum 1530 snap 0 seq_num 1530) 2024-09-21T18:10:20.317 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:20.339 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:20.336+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:20.355 INFO:tasks.rados.rados.0.smithi160.stdout:3809: finishing write tid 1 to smithi16030472-340 2024-09-21T18:10:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.367 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.381 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:10:20.439 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:20.437+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:20.456 INFO:tasks.rados.rados.0.smithi160.stdout:3810: done (15 left) 2024-09-21T18:10:20.456 INFO:tasks.rados.rados.0.smithi160.stdout:3819: read oid 119 snap -1 2024-09-21T18:10:20.456 INFO:tasks.rados.rados.0.smithi160.stdout:3819: expect (ObjNum 534 snap 0 seq_num 534) 2024-09-21T18:10:20.456 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:20.462 INFO:tasks.rados.rados.0.smithi160.stdout:3814: got (0) Success 2024-09-21T18:10:20.463 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 288 v 0 (ObjNum 872 snap 0 seq_num 872) dirty exists 2024-09-21T18:10:20.463 INFO:tasks.rados.rados.0.smithi160.stdout:3814: done (15 left) 2024-09-21T18:10:20.463 INFO:tasks.rados.rados.0.smithi160.stdout:3820: cache_flush oid 80 snap -1 2024-09-21T18:10:20.463 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:20.474 INFO:tasks.rados.rados.0.smithi160.stdout:3815: expect (ObjNum 88 snap 0 seq_num 88) 2024-09-21T18:10:20.474 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:20.511 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.511 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.526 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:10:20.540 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:20.537+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:20.610 INFO:tasks.rados.rados.0.smithi160.stdout:3804: finishing write tid 1 to smithi16030472-74 2024-09-21T18:10:20.610 INFO:tasks.rados.rados.0.smithi160.stdout:3806: expect (ObjNum 1481 snap 0 seq_num 1481) 2024-09-21T18:10:20.641 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:20.637+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:20.655 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.655 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.673 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:10:20.699 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:20.742 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:20.741+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3804: finishing write tid 2 to smithi16030472-74 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3804: finishing write tid 3 to smithi16030472-74 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3802: done (15 left) 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3806: done (14 left) 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3815: done (13 left) 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3821: delete oid 412 current snap is 0 2024-09-21T18:10:20.744 INFO:tasks.rados.rados.0.smithi160.stdout:3804: finishing write tid 4 to smithi16030472-74 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: finishing write tid 5 to smithi16030472-74 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: oid 74 updating version 0 to 1133 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: oid 74 updating version 1133 to 1135 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: oid 74 version 1135 is already newer than 1132 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: oid 74 version 1135 is already newer than 1134 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 74 v 1135 (ObjNum 1578 snap 0 seq_num 1578) dirty exists 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3804: left oid 74 (ObjNum 1578 snap 0 seq_num 1578) 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3816: got (0) Success 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3809: finishing write tid 2 to smithi16030472-340 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3811: finishing write tid 1 to smithi16030472-286 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3811: finishing write tid 2 to smithi16030472-286 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3811: finishing write tid 3 to smithi16030472-286 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:3819: expect (ObjNum 534 snap 0 seq_num 534) 2024-09-21T18:10:20.745 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:20.794 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1015693 2024-09-21T18:10:20.796 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.796 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.810 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:10:20.854 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:20.853+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:20.856 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2874422 2024-09-21T18:10:20.857 INFO:tasks.rados.rados.0.smithi160.stdout:3820: got (0) Success 2024-09-21T18:10:20.857 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 80 v 0 (ObjNum 1490 snap 0 seq_num 1490) dirty exists 2024-09-21T18:10:20.857 INFO:tasks.rados.rados.0.smithi160.stdout:3809: finishing write tid 3 to smithi16030472-340 2024-09-21T18:10:20.857 INFO:tasks.rados.rados.0.smithi160.stdout:3811: finishing write tid 4 to smithi16030472-286 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3811: oid 286 updating version 0 to 1036 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3811: oid 286 version 1036 is already newer than 1034 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3811: oid 286 version 1036 is already newer than 1035 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 286 v 1036 (ObjNum 1580 snap 0 seq_num 1580) dirty exists 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3811: left oid 286 (ObjNum 1580 snap 0 seq_num 1580) 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3799: got (0) Success 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 264 v 0 (ObjNum 1519 snap 0 seq_num 1519) dirty exists 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3805: got (0) Success 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 417 v 0 (ObjNum 129 snap 0 seq_num 3126265664) clean dne 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3813: finishing copy_from to smithi16030472-16 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 16 v 949 (ObjNum 1362 snap 0 seq_num 1362) dirty exists 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3801: finishing copy_from racing read to smithi16030472-365 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3813: finishing copy_from racing read to smithi16030472-16 2024-09-21T18:10:20.858 INFO:tasks.rados.rados.0.smithi160.stdout:3817: finishing copy_from to smithi16030472-263 2024-09-21T18:10:20.859 INFO:tasks.rados.rados.0.smithi160.stdout:3817: got expected ENOENT (src dne) 2024-09-21T18:10:20.859 INFO:tasks.rados.rados.0.smithi160.stdout:3817: finishing copy_from racing read to smithi16030472-263 2024-09-21T18:10:20.859 INFO:tasks.rados.rados.0.smithi160.stdout:3818: expect (ObjNum 1530 snap 0 seq_num 1530) 2024-09-21T18:10:20.943 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:20.941+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:20.972 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:20.972 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:20.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3809: finishing write tid 4 to smithi16030472-340 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3809: oid 340 updating version 0 to 1111 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3809: oid 340 version 1111 is already newer than 1109 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3809: oid 340 version 1111 is already newer than 1110 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 340 v 1111 (ObjNum 1579 snap 0 seq_num 1579) dirty exists 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3809: left oid 340 (ObjNum 1579 snap 0 seq_num 1579) 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3812: finishing write tid 1 to smithi16030472-381 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3812: finishing write tid 2 to smithi16030472-381 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3812: finishing write tid 3 to smithi16030472-381 2024-09-21T18:10:21.023 INFO:tasks.rados.rados.0.smithi160.stdout:3812: finishing write tid 4 to smithi16030472-381 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3812: oid 381 updating version 0 to 1114 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3812: oid 381 version 1114 is already newer than 1112 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3812: oid 381 version 1114 is already newer than 1113 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 381 v 1114 (ObjNum 1581 snap 0 seq_num 1581) dirty exists 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3812: left oid 381 (ObjNum 1581 snap 0 seq_num 1581) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3801: finishing copy_from to smithi16030472-365 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 365 v 35 (ObjNum 60 snap 0 seq_num 60) dirty exists 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3799: done (13 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3801: done (12 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3804: done (11 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3805: done (10 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3809: done (9 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3811: done (8 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3812: done (7 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3813: done (6 left) 2024-09-21T18:10:21.024 INFO:tasks.rados.rados.0.smithi160.stdout:3816: done (5 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3817: done (4 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3818: done (3 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3819: done (2 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3820: done (1 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3821: done (0 left) 2024-09-21T18:10:21.025 INFO:tasks.rados.rados.0.smithi160.stdout:3822: delete oid 232 current snap is 0 2024-09-21T18:10:21.043 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:21.039+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:21.111 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.111 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.125 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:10:21.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:10:21.127 INFO:tasks.rados.rados.0.smithi160.stdout:3822: done (0 left) 2024-09-21T18:10:21.127 INFO:tasks.rados.rados.0.smithi160.stdout:3823: cache_flush oid 445 snap -1 2024-09-21T18:10:21.127 INFO:tasks.rados.rados.0.smithi160.stdout:3824: delete oid 443 current snap is 0 2024-09-21T18:10:21.132 INFO:tasks.rados.rados.0.smithi160.stdout:3824: done (1 left) 2024-09-21T18:10:21.132 INFO:tasks.rados.rados.0.smithi160.stdout:3825: delete oid 458 current snap is 0 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3825: done (1 left) 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3826: read oid 230 snap -1 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3826: expect deleted 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3827: read oid 136 snap -1 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3827: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3828: cache_flush oid 448 snap -1 2024-09-21T18:10:21.137 INFO:tasks.rados.rados.0.smithi160.stdout:3829: delete oid 215 current snap is 0 2024-09-21T18:10:21.144 INFO:tasks.rados.rados.0.smithi160.stdout:3826: done (4 left) 2024-09-21T18:10:21.144 INFO:tasks.rados.rados.0.smithi160.stdout:3829: done (3 left) 2024-09-21T18:10:21.144 INFO:tasks.rados.rados.0.smithi160.stdout:3830: delete oid 62 current snap is 0 2024-09-21T18:10:21.146 INFO:tasks.rados.rados.0.smithi160.stdout:3827: expect (ObjNum 525 snap 0 seq_num 525) 2024-09-21T18:10:21.147 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 356660 2024-09-21T18:10:21.174 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:21.173+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:21.215 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2779462 2024-09-21T18:10:21.245 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:21.241+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:21.249 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.249 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.258 INFO:tasks.rados.rados.0.smithi160.stdout:3823: got (0) Success 2024-09-21T18:10:21.258 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 445 v 0 (ObjNum 950 snap 0 seq_num 950) dirty exists 2024-09-21T18:10:21.258 INFO:tasks.rados.rados.0.smithi160.stdout:3823: done (3 left) 2024-09-21T18:10:21.259 INFO:tasks.rados.rados.0.smithi160.stdout:3827: done (2 left) 2024-09-21T18:10:21.259 INFO:tasks.rados.rados.0.smithi160.stdout:3830: done (1 left) 2024-09-21T18:10:21.259 INFO:tasks.rados.rados.0.smithi160.stdout:3831: write oid 54 current snap is 0 2024-09-21T18:10:21.259 INFO:tasks.rados.rados.0.smithi160.stdout:3831: seq_num 1588 ranges {689516=496836,1809523=433888} 2024-09-21T18:10:21.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:10:21.277 INFO:tasks.rados.rados.0.smithi160.stdout:3831: writing smithi16030472-54 from 689516 to 1186352 tid 1 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3831: writing smithi16030472-54 from 1809523 to 2243411 tid 2 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3832: read oid 12 snap -1 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3832: expect deleted 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3833: read oid 53 snap -1 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3833: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:10:21.295 INFO:tasks.rados.rados.0.smithi160.stdout:3834: write oid 442 current snap is 0 2024-09-21T18:10:21.296 INFO:tasks.rados.rados.0.smithi160.stdout:3834: seq_num 1589 ranges {706323=688438,1984358=577546} 2024-09-21T18:10:21.315 INFO:tasks.rados.rados.0.smithi160.stdout:3834: writing smithi16030472-442 from 706323 to 1394761 tid 1 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3834: writing smithi16030472-442 from 1984358 to 2561904 tid 2 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3835: read oid 349 snap -1 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3835: expect deleted 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3836: cache_evict oid 291 snap -1 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3828: got (0) Success 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 0 (ObjNum 1525 snap 0 seq_num 1525) dirty exists 2024-09-21T18:10:21.328 INFO:tasks.rados.rados.0.smithi160.stdout:3828: done (6 left) 2024-09-21T18:10:21.329 INFO:tasks.rados.rados.0.smithi160.stdout:3837: write (excl) oid 247 current snap is 0 2024-09-21T18:10:21.329 INFO:tasks.rados.rados.0.smithi160.stdout:3837: seq_num 1590 ranges {723130=763687,2042840=437557} 2024-09-21T18:10:21.343 INFO:tasks.rados.rados.0.smithi160.stdout:3837: writing smithi16030472-247 from 723130 to 1486817 tid 1 2024-09-21T18:10:21.345 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:21.343+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3837: writing smithi16030472-247 from 2042840 to 2480397 tid 2 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3838: read oid 137 snap -1 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3838: expect (ObjNum 1368 snap 0 seq_num 1368) 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3839: cache_flush oid 98 snap -1 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3840: read oid 31 snap -1 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3840: expect (ObjNum 1553 snap 0 seq_num 1553) 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3841: read oid 94 snap -1 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3841: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:10:21.353 INFO:tasks.rados.rados.0.smithi160.stdout:3842: read oid 21 snap -1 2024-09-21T18:10:21.354 INFO:tasks.rados.rados.0.smithi160.stdout:3842: expect (ObjNum 1474 snap 0 seq_num 1474) 2024-09-21T18:10:21.354 INFO:tasks.rados.rados.0.smithi160.stdout:3843: copy_from oid 420 from oid 269 current snap is 0 2024-09-21T18:10:21.354 INFO:tasks.rados.rados.0.smithi160.stdout:3844: cache_evict oid 154 snap -1 2024-09-21T18:10:21.354 INFO:tasks.rados.rados.0.smithi160.stdout:3845: write oid 475 current snap is 0 2024-09-21T18:10:21.354 INFO:tasks.rados.rados.0.smithi160.stdout:3845: seq_num 1591 ranges {739937=438936,1701322=581215} 2024-09-21T18:10:21.364 INFO:tasks.rados.rados.0.smithi160.stdout:3845: writing smithi16030472-475 from 739937 to 1178873 tid 1 2024-09-21T18:10:21.374 INFO:tasks.rados.rados.0.smithi160.stdout:3845: writing smithi16030472-475 from 1701322 to 2282537 tid 2 2024-09-21T18:10:21.374 INFO:tasks.rados.rados.0.smithi160.stdout:3846: delete oid 355 current snap is 0 2024-09-21T18:10:21.433 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.433 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.446 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:21.445+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:21.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:10:21.454 INFO:tasks.rados.rados.0.smithi160.stdout:3837: finishing write tid 1 to smithi16030472-247 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: finishing write tid 2 to smithi16030472-247 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: finishing write tid 3 to smithi16030472-247 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: finishing write tid 4 to smithi16030472-247 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: oid 247 updating version 0 to 1116 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: oid 247 version 1116 is already newer than 1115 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: oid 247 updating version 1116 to 1117 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 247 v 1117 (ObjNum 1590 snap 0 seq_num 1590) dirty exists 2024-09-21T18:10:21.461 INFO:tasks.rados.rados.0.smithi160.stdout:3837: left oid 247 (ObjNum 1590 snap 0 seq_num 1590) 2024-09-21T18:10:21.462 INFO:tasks.rados.rados.0.smithi160.stdout:3840: expect (ObjNum 1553 snap 0 seq_num 1553) 2024-09-21T18:10:21.462 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:21.544 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:21.545 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:21.545+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:21.616 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.616 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3843: finishing copy_from to smithi16030472-420 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3843: got expected ENOENT (src dne) 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3843: finishing copy_from racing read to smithi16030472-420 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3836: got (16) Device or resource busy 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3839: got (0) Success 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 98 v 0 (ObjNum 97217 snap 0 seq_num 2436187232) clean dne 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3831: finishing write tid 1 to smithi16030472-54 2024-09-21T18:10:21.617 INFO:tasks.rados.rados.0.smithi160.stdout:3831: finishing write tid 2 to smithi16030472-54 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3836: done (15 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3837: done (14 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3839: done (13 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3840: done (12 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3843: done (11 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3846: done (10 left) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3847: cache_evict oid 436 snap -1 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: finishing write tid 3 to smithi16030472-54 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: finishing write tid 4 to smithi16030472-54 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: oid 54 updating version 0 to 1144 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: oid 54 version 1144 is already newer than 1143 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: oid 54 updating version 1144 to 1145 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 54 v 1145 (ObjNum 1588 snap 0 seq_num 1588) dirty exists 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3831: left oid 54 (ObjNum 1588 snap 0 seq_num 1588) 2024-09-21T18:10:21.618 INFO:tasks.rados.rados.0.smithi160.stdout:3844: got (0) Success 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3831: done (10 left) 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3844: done (9 left) 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3848: cache_flush oid 104 snap -1 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3849: read oid 138 snap -1 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3849: expect (ObjNum 1016 snap 0 seq_num 1016) 2024-09-21T18:10:21.619 INFO:tasks.rados.rados.0.smithi160.stdout:3850: delete oid 149 current snap is 0 2024-09-21T18:10:21.630 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:10:21.647 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:21.643+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:21.703 INFO:tasks.rados.rados.0.smithi160.stdout:3833: expect (ObjNum 1115 snap 0 seq_num 1115) 2024-09-21T18:10:21.748 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:21.745+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:21.794 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.794 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.802 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2511821 2024-09-21T18:10:21.808 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:10:21.809 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:21.849 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:21.844+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:21.857 INFO:tasks.rados.rados.0.smithi160.stdout:3838: expect (ObjNum 1368 snap 0 seq_num 1368) 2024-09-21T18:10:21.902 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:21.949 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:21.948+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:21.973 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:21.974 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:21.988 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:10:21.997 INFO:tasks.rados.rados.0.smithi160.stdout:3841: expect (ObjNum 845 snap 0 seq_num 845) 2024-09-21T18:10:21.999 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 462648 2024-09-21T18:10:22.050 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:22.043+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:22.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:22.071 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 407203 2024-09-21T18:10:22.105 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:22.105 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:22.106 INFO:tasks.rados.rados.0.smithi160.stdout:3842: expect (ObjNum 1474 snap 0 seq_num 1474) 2024-09-21T18:10:22.112 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:10:22.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:10:22.120 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1258012 2024-09-21T18:10:22.150 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:22.148+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:22.224 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:22.225 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:22.231 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.243 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:22.244 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:22.245 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:22.245 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:22.251 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:22.247+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:22.258 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:10:22.266 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:22.266 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:22.287 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:22.287 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:22.299 INFO:tasks.rados.rados.0.smithi160.stdout:3849: expect (ObjNum 1016 snap 0 seq_num 1016) 2024-09-21T18:10:22.299 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 39670 2024-09-21T18:10:22.305 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:22.327 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:22.340 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.348 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:22.348 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:22.352 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:22.347+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:22.367 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:22.381 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:22.381 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:22.383 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.390 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:22.396 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:10:22.412 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:22.426 INFO:tasks.rados.rados.0.smithi160.stdout:3847: got (0) Success 2024-09-21T18:10:22.426 INFO:tasks.rados.rados.0.smithi160.stdout:3832: done (11 left) 2024-09-21T18:10:22.426 INFO:tasks.rados.rados.0.smithi160.stdout:3833: done (10 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3838: done (9 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3841: done (8 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3842: done (7 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3847: done (6 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3849: done (5 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3850: done (4 left) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3851: read oid 27 snap -1 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3851: expect (ObjNum 1186 snap 0 seq_num 1186) 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3852: cache_evict oid 137 snap -1 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: finishing write tid 1 to smithi16030472-442 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: finishing write tid 2 to smithi16030472-442 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: finishing write tid 3 to smithi16030472-442 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: finishing write tid 4 to smithi16030472-442 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: oid 442 updating version 0 to 1041 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: oid 442 version 1041 is already newer than 1040 2024-09-21T18:10:22.427 INFO:tasks.rados.rados.0.smithi160.stdout:3834: oid 442 version 1041 is already newer than 1039 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 1041 (ObjNum 1589 snap 0 seq_num 1589) dirty exists 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:3834: left oid 442 (ObjNum 1589 snap 0 seq_num 1589) 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:3845: finishing write tid 1 to smithi16030472-475 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:3834: done (5 left) 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:3853: write (excl) oid 419 current snap is 0 2024-09-21T18:10:22.428 INFO:tasks.rados.rados.0.smithi160.stdout:3853: seq_num 1594 ranges {790358=664683,2160415=444895} 2024-09-21T18:10:22.434 INFO:tasks.rados.rados.0.smithi160.stdout:3853: writing smithi16030472-419 from 790358 to 1455041 tid 1 2024-09-21T18:10:22.437 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:22.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3853: writing smithi16030472-419 from 2160415 to 2605310 tid 2 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3854: read oid 203 snap -1 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3854: expect (ObjNum 950 snap 0 seq_num 950) 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3855: cache_flush oid 491 snap -1 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3835: done (7 left) 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3856: read oid 96 snap -1 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3856: expect (ObjNum 1465 snap 0 seq_num 1465) 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3857: cache_flush oid 10 snap -1 2024-09-21T18:10:22.441 INFO:tasks.rados.rados.0.smithi160.stdout:3848: got (2) No such file or directory 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: finishing write tid 2 to smithi16030472-475 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3848: done (8 left) 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3858: copy_from oid 71 from oid 463 current snap is 0 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: finishing write tid 3 to smithi16030472-475 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: finishing write tid 4 to smithi16030472-475 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: oid 475 updating version 0 to 1044 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: oid 475 version 1044 is already newer than 1042 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: oid 475 version 1044 is already newer than 1043 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 475 v 1044 (ObjNum 1591 snap 0 seq_num 1591) dirty exists 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: left oid 475 (ObjNum 1591 snap 0 seq_num 1591) 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3845: done (8 left) 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3859: read oid 449 snap -1 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3859: expect (ObjNum 568 snap 0 seq_num 568) 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3860: delete oid 255 current snap is 0 2024-09-21T18:10:22.442 INFO:tasks.rados.rados.0.smithi160.stdout:3852: got (16) Device or resource busy 2024-09-21T18:10:22.453 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:22.449+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:22.457 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:22.458 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:22.481 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:22.481 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:22.504 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:22.504 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:22.521 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:22.521 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:22.521 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:10:22.521 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:22.528 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:22.528 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:22.529 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:22.525+0000 7ff40b999640 -1 osd.4 125 handle_osd_map injecting map crc failure 2024-09-21T18:10:22.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:22.532 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:22.536 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:10:22.537 INFO:tasks.rados.rados.0.smithi160.stdout:3851: expect (ObjNum 1186 snap 0 seq_num 1186) 2024-09-21T18:10:22.537 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.554 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:22.553+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:22.628 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2412622 2024-09-21T18:10:22.651 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:22.651 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:22.651 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:22.651 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:22.651 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:22.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:22.651+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:22.665 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.672 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:22.735 INFO:tasks.rados.rados.0.smithi160.stdout:3856: expect (ObjNum 1465 snap 0 seq_num 1465) 2024-09-21T18:10:22.735 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 200000 2024-09-21T18:10:22.756 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:22.753+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:22.760 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.791 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:22.791 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:22.792 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [] 2024-09-21T18:10:22.792 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:22.805 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: finishing write tid 1 to smithi16030472-419 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: finishing write tid 2 to smithi16030472-419 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: finishing write tid 3 to smithi16030472-419 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: finishing write tid 4 to smithi16030472-419 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: oid 419 updating version 0 to 1045 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: oid 419 updating version 1045 to 1047 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:3853: oid 419 version 1047 is already newer than 1046 2024-09-21T18:10:22.814 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 419 v 1047 (ObjNum 1594 snap 0 seq_num 1594) dirty exists 2024-09-21T18:10:22.815 INFO:tasks.rados.rados.0.smithi160.stdout:3853: left oid 419 (ObjNum 1594 snap 0 seq_num 1594) 2024-09-21T18:10:22.815 INFO:tasks.rados.rados.0.smithi160.stdout:3854: expect (ObjNum 950 snap 0 seq_num 950) 2024-09-21T18:10:22.815 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.858 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:22.853+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:22.886 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:22.919 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:22.919 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:10:22.919 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:22.932 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:22.960 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:22.957+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:23.027 INFO:tasks.rados.rados.0.smithi160.stdout:3855: got (0) Success 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 491 v 0 (ObjNum 935 snap 0 seq_num 935) dirty exists 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3857: got (0) Success 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 10 v 0 (ObjNum 412 snap 0 seq_num 412) dirty exists 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3851: done (9 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3852: done (8 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3853: done (7 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3854: done (6 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3855: done (5 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3856: done (4 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3857: done (3 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3860: done (2 left) 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3861: read oid 192 snap -1 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3861: expect deleted 2024-09-21T18:10:23.028 INFO:tasks.rados.rados.0.smithi160.stdout:3862: copy_from oid 189 from oid 109 current snap is 0 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3858: finishing copy_from to smithi16030472-71 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3858: got expected ENOENT (src dne) 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3863: read oid 490 snap -1 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3863: expect (ObjNum 1426 snap 0 seq_num 1426) 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3864: cache_flush oid 444 snap -1 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3865: cache_try_flush oid 442 snap -1 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3866: read oid 307 snap -1 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3866: expect (ObjNum 1502 snap 0 seq_num 1502) 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3867: cache_try_flush oid 187 snap -1 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3858: finishing copy_from racing read to smithi16030472-71 2024-09-21T18:10:23.029 INFO:tasks.rados.rados.0.smithi160.stdout:3859: expect (ObjNum 568 snap 0 seq_num 568) 2024-09-21T18:10:23.036 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:23.033+0000 7f8c3a0ca640 -1 osd.12 125 handle_osd_map injecting map crc failure 2024-09-21T18:10:23.061 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:23.055+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:23.079 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.097 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:23.097 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:23.097 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:23.097 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:23.097 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:23.111 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:23.132 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.161 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:23.157+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:23.186 INFO:tasks.rados.rados.0.smithi160.stdout:3864: got (0) Success 2024-09-21T18:10:23.186 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 444 v 0 (ObjNum 129 snap 0 seq_num 3140234080) clean dne 2024-09-21T18:10:23.186 INFO:tasks.rados.rados.0.smithi160.stdout:3863: expect (ObjNum 1426 snap 0 seq_num 1426) 2024-09-21T18:10:23.218 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1804030 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.236 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:139 4.3 4:cd35ea38:::smithi16030472-35:head [setxattr _header (56) in=63b,truncate 2571438] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:23.237 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:23.237 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 190.08677735800001, 2024-09-21T18:10:23.237 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3326640089999999, 2024-09-21T18:10:23.237 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.237 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 139 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.238 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.240 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.240 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.241 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144966+0000", 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144967+0000", 2024-09-21T18:10:23.242 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.703e-06 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144974+0000", 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8070000000000002e-06 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.243 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144986+0000", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1479e-05 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.144998+0000", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1956999999999999e-05 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245336+0000", 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100338737 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.244 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245374+0000", 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7595999999999997e-05 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228425+0000", 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98305140499999999 2024-09-21T18:10:23.245 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228580+0000", 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015490999999999999 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.228824+0000", 2024-09-21T18:10:23.246 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00024349300000000001 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233659+0000", 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0048354629999999999 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.247 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477465+0000", 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24380559299999999 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477537+0000", 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2557000000000003e-05 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.248 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477630+0000", 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2309e-05 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:140 4.3 4:cd35ea38:::smithi16030472-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 190.08673104900001, 2024-09-21T18:10:23.249 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3329282390000001, 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 140 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.250 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145012+0000", 2024-09-21T18:10:23.251 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145013+0000", 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9000000000000005e-07 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.252 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145017+0000", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7500000000000001e-06 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145026+0000", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2429999999999994e-06 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.145034+0000", 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.6629999999999993e-06 2024-09-21T18:10:23.253 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245442+0000", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10040795 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.245523+0000", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1015999999999998e-05 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233502+0000", 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98797960100000004 2024-09-21T18:10:23.254 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233596+0000", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3336000000000002e-05 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477684+0000", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.24408848899999999 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477811+0000", 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000126682 2024-09-21T18:10:23.255 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477940+0000", 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000129519 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:146 4.3 4:f074207e:::smithi16030472-37:head [write 1923587~633336 in=633336b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 190.084635136, 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3310062620000001, 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.256 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 146 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.257 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147108+0000", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147109+0000", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1060000000000001e-06 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147735+0000", 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00062637900000000004 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.258 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147750+0000", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4763000000000001e-05 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147759+0000", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3000000000000002e-06 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246600+0000", 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098841132999999998 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.259 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246671+0000", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.1359000000000006e-05 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233703+0000", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98703235600000006 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233788+0000", 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4118999999999999e-05 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.260 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.233915+0000", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012724000000000001 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237953+0000", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0040377579999999998 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.477995+0000", 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.240042123 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.261 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478061+0000", 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6828999999999993e-05 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.478114+0000", 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2797000000000003e-05 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.262 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:147 4.3 4:f074207e:::smithi16030472-37:head [setxattr _header (56) in=63b,truncate 2605052] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 190.08397202099999, 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4282742719999999, 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 147 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.263 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:23.264 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.264 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.264 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147771+0000", 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.265 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147772+0000", 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2100000000000001e-06 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147777+0000", 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.023e-06 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147788+0000", 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0223e-05 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.266 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147798+0000", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0856000000000001e-05 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246731+0000", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.098933035000000003 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.246776+0000", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4329000000000002e-05 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.267 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235589+0000", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.98881306499999999 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235681+0000", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.1704000000000002e-05 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.235802+0000", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012181 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.268 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237958+0000", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021560630000000002 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575850+0000", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33789151000000001 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.575939+0000", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8838999999999994e-05 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.269 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576045+0000", 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000106605 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:148 4.3 4:f074207e:::smithi16030472-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e43)", 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 190.08393552199999, 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4285937500000001, 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.270 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 148 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.271 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147808+0000", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7700000000000003e-07 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147812+0000", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.343e-06 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147822+0000", 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0536999999999999e-05 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.272 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.147829+0000", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.126e-06 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346918+0000", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.19908835899999999 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.346972+0000", 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.4499000000000001e-05 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.273 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236045+0000", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.88907262799999998 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236132+0000", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6811999999999995e-05 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576109+0000", 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.33997696700000002 2024-09-21T18:10:23.274 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576262+0000", 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015331399999999999 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.576401+0000", 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000139288 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.275 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:170 4.3 4:e85d446d:::smithi16030472-43:head [write 1758126~644343 in=644343b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 189.95029735, 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.396162956, 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 170 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.276 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281446+0000", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.281448+0000", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8700000000000001e-06 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.277 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282102+0000", 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000654299 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282121+0000", 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9162000000000001e-05 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282140+0000", 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8783999999999998e-05 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.278 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448781+0000", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16664120800000001 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448825+0000", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.4113000000000003e-05 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236175+0000", 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78734934899999998 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.279 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236255+0000", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0587999999999995e-05 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.236374+0000", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000119277 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237988+0000", 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001613345 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.280 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677445+0000", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43945761300000002 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677536+0000", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0729000000000005e-05 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677609+0000", 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2619000000000004e-05 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.281 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:171 4.3 4:e85d446d:::smithi16030472-43:head [setxattr _header (56) in=63b,truncate 2705894] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 189.94958555900001, 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3956427149999999, 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 171 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.282 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282158+0000", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282159+0000", 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.327e-06 2024-09-21T18:10:23.283 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282164+0000", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.3059999999999997e-06 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282173+0000", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4230000000000005e-06 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282182+0000", 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.5000000000000005e-06 2024-09-21T18:10:23.284 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448858+0000", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16667584899999999 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.448898+0000", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0293000000000002e-05 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237532+0000", 2024-09-21T18:10:23.285 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.78863408999999995 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237621+0000", 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.8311000000000002e-05 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 5", 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.237739+0000", 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00011824699999999999 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.286 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.240640+0000", 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0029011940000000002 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677708+0000", 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43706790000000001 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677750+0000", 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2175000000000002e-05 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.287 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677800+0000", 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.0099999999999998e-05 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:172 4.3 4:e85d446d:::smithi16030472-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 189.94954799600001, 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3960833369999999, 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.288 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 172 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282195+0000", 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.289 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282196+0000", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.7000000000000003e-07 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282200+0000", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6500000000000002e-06 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282211+0000", 2024-09-21T18:10:23.290 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.119e-05 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.282221+0000", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0118999999999999e-05 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449291+0000", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.16707040000000001 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:13.449407+0000", 2024-09-21T18:10:23.291 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000115388 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326026+0000", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.87661911599999998 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.326124+0000", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7807999999999996e-05 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.677964+0000", 2024-09-21T18:10:23.292 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.35183998799999999 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678133+0000", 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016927700000000001 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:14.678278+0000", 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014553100000000001 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.293 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2012 4.3 4:d5e652b8:::smithi16030472-449:head [write 1749580~611149 in=611149b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.15216199700001, 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3826805879999999, 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2012 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.294 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079581+0000", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079582+0000", 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.8899999999999999e-07 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.295 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079759+0000", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000176799 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079765+0000", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9120000000000001e-06 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079770+0000", 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.3229999999999999e-06 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.296 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.139983+0000", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060213005999999999 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462258+0000", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322274752 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462262+0000", 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2069999999999997e-06 2024-09-21T18:10:23.297 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2013 4.3 4:d5e652b8:::smithi16030472-449:head [write 2853383~452450 in=452450b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.15196745099999, 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382525979, 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.298 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2013 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.299 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.079776+0000", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8800000000000003e-07 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080045+0000", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000269039 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080052+0000", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6170000000000003e-06 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.300 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080056+0000", 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1790000000000001e-06 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140368+0000", 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060311647000000003 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462299+0000", 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.321931827 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.301 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462302+0000", 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.182e-06 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2014 4.3 4:d5e652b8:::smithi16030472-449:head [setxattr _header (57) in=64b,truncate 3529536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.15168106300001, 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382268034, 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.302 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2014 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080062+0000", 2024-09-21T18:10:23.303 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080063+0000", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.5300000000000004e-07 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080066+0000", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.534e-06 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080070+0000", 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.873e-06 2024-09-21T18:10:23.304 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080074+0000", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.7069999999999998e-06 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.140776+0000", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.060702514999999999 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462329+0000", 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3215525299999999 2024-09-21T18:10:23.305 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462330+0000", 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.322e-06 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2015 4.3 4:d5e652b8:::smithi16030472-449:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.15166498299999, 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.382277634, 2024-09-21T18:10:23.306 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2015 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.307 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080078+0000", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080079+0000", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9799999999999999e-07 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080081+0000", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.978e-06 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.308 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080085+0000", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6750000000000001e-06 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.080089+0000", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.8659999999999999e-06 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.141091+0000", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.061001970000000003 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462356+0000", 2024-09-21T18:10:23.309 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3212647470000001 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2021 4.3 4:fed6f1ad:::smithi16030472-451:head [write 763150~726877 in=726877b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.122737731, 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3533763890000001, 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.310 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2021 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109005+0000", 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.311 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109010+0000", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5709999999999999e-06 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109234+0000", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000224043 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109258+0000", 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3629999999999999e-05 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.312 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.109270+0000", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2327000000000001e-05 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241411+0000", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13214074000000001 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462380+0000", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220969462 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.313 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462382+0000", 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6160000000000001e-06 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2022 4.3 4:fed6f1ad:::smithi16030472-451:head [write 1982897~614818 in=614818b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.11651865600001, 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.347185544, 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.314 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2022 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115224+0000", 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.315 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115228+0000", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.2399999999999999e-06 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115581+0000", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00035299700000000002 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115600+0000", 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8975e-05 2024-09-21T18:10:23.316 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.115610+0000", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0011e-05 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241672+0000", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12606255 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462409+0000", 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220736464 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.317 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462410+0000", 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3069999999999999e-06 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2023 4.3 4:fed6f1ad:::smithi16030472-451:head [write 3024582~474994 in=474994b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.111417061, 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3421081990000001, 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.318 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2023 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.319 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120326+0000", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120329+0000", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.3459999999999998e-06 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120612+0000", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028260200000000002 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.320 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120632+0000", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9658000000000001e-05 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120642+0000", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0356999999999999e-05 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.241983+0000", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121341123 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.321 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462433+0000", 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220449772 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462434+0000", 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341e-06 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2024 4.3 4:fed6f1ad:::smithi16030472-451:head [setxattr _header (57) in=64b,truncate 3563150] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.11108311300001, 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3417996409999999, 2024-09-21T18:10:23.322 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2024 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.323 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120660+0000", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120661+0000", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.68e-07 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120665+0000", 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.0470000000000004e-06 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.324 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120670+0000", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2560000000000002e-06 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120675+0000", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5129999999999998e-06 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242140+0000", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.121465121 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.325 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462458+0000", 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220318837 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462460+0000", 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2025 4.3 4:fed6f1ad:::smithi16030472-451:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.11106263400001, 2024-09-21T18:10:23.326 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.341803866, 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2025 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.327 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120680+0000", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120681+0000", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-07 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120683+0000", 2024-09-21T18:10:23.328 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.266e-06 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120688+0000", 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.2079999999999999e-06 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.120693+0000", 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7260000000000002e-06 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.329 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.242325+0000", 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12163207700000001 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462484+0000", 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.220159166 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2036 4.3 4:e9fae96f:::smithi16030472-454:head [write 413571~552624 in=552624b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:23.330 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.024734107, 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.255503627, 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2036 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:23.331 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207009+0000", 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207011+0000", 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.979e-06 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.332 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207189+0000", 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017843000000000001 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207206+0000", 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6467e-05 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.207215+0000", 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2380000000000003e-06 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.333 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.341752+0000", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.13453654000000001 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462511+0000", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1207597739999999 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462513+0000", 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1990000000000001e-06 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.334 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2037 4.3 4:e9fae96f:::smithi16030472-454:head [write 1641990~762145 in=762145b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.01627392099999, 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2470707910000001, 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.335 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2037 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215469+0000", 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.336 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215471+0000", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0710000000000002e-06 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215907+0000", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00043553100000000001 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215928+0000", 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.0822000000000002e-05 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.337 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.215939+0000", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1243000000000001e-05 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342151+0000", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.12621222500000001 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462539+0000", 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1203877950000001 2024-09-21T18:10:23.338 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462540+0000", 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1039999999999999e-06 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2038 4.3 4:e9fae96f:::smithi16030472-454:head [write 2990498~425163 in=425163b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 166.01034621599999, 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2411672460000001, 2024-09-21T18:10:23.339 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2038 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.340 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221397+0000", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221399+0000", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.1210000000000001e-06 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221660+0000", 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00026131899999999998 2024-09-21T18:10:23.341 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221678+0000", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.747e-05 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.221687+0000", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3200000000000006e-06 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.342408+0000", 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.120720987 2024-09-21T18:10:23.342 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462563+0000", 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.120154917 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.462564+0000", 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1119999999999999e-06 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.343 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:23.345 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:23.345 INFO:tasks.rados.rados.0.smithi160.stdout:3862: finishing copy_from racing read to smithi16030472-189 2024-09-21T18:10:23.346 INFO:tasks.rados.rados.0.smithi160.stdout:3866: expect (ObjNum 1502 snap 0 seq_num 1502) 2024-09-21T18:10:23.346 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.346 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:23.341+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:23.355 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 591087 2024-09-21T18:10:23.382 INFO:tasks.rados.rados.0.smithi160.stdout:3865: got (0) Success 2024-09-21T18:10:23.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 442 v 0 (ObjNum 1589 snap 0 seq_num 1589) dirty exists 2024-09-21T18:10:23.382 INFO:tasks.rados.rados.0.smithi160.stdout:3862: finishing copy_from to smithi16030472-189 2024-09-21T18:10:23.382 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 189 v 25 (ObjNum 88 snap 0 seq_num 88) dirty exists 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3867: got (0) Success 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 187 v 0 (ObjNum 1443 snap 0 seq_num 1443) dirty exists 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3858: done (8 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3859: done (7 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3861: done (6 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3862: done (5 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3863: done (4 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3864: done (3 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3865: done (2 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3866: done (1 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3867: done (0 left) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3868: read oid 427 snap -1 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3868: expect (ObjNum 1533 snap 0 seq_num 1533) 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3869: copy_from oid 344 from oid 256 current snap is 0 2024-09-21T18:10:23.383 INFO:tasks.rados.rados.0.smithi160.stdout:3870: read oid 167 snap -1 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3870: expect deleted 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3871: cache_try_flush oid 168 snap -1 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3872: read oid 83 snap -1 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3872: expect (ObjNum 1559 snap 0 seq_num 1559) 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3873: cache_try_flush oid 232 snap -1 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3874: cache_flush oid 414 snap -1 2024-09-21T18:10:23.384 INFO:tasks.rados.rados.0.smithi160.stdout:3875: delete oid 114 current snap is 0 2024-09-21T18:10:23.389 INFO:tasks.rados.rados.0.smithi160.stdout:3872: expect (ObjNum 1559 snap 0 seq_num 1559) 2024-09-21T18:10:23.424 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.446 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:23.444+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:23.464 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.473 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:23.473 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:23.473 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:10:23.473 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:23.487 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:23.505 INFO:tasks.rados.rados.0.smithi160.stdout:3868: expect (ObjNum 1533 snap 0 seq_num 1533) 2024-09-21T18:10:23.549 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:23.548+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:23.613 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.629 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:23.629 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:10:23.630 INFO:teuthology.orchestra.run.smithi160.stdout: "complaint_time": 30, 2024-09-21T18:10:23.630 INFO:teuthology.orchestra.run.smithi160.stdout: "num_blocked_ops": 0 2024-09-21T18:10:23.630 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:23.642 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:23.668 INFO:tasks.rados.rados.0.smithi160.stdout:3874: got (0) Success 2024-09-21T18:10:23.668 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 414 v 0 (ObjNum 1053 snap 0 seq_num 1053) dirty exists 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3869: finishing copy_from racing read to smithi16030472-344 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3868: done (7 left) 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3870: done (6 left) 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3872: done (5 left) 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3874: done (4 left) 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3875: done (3 left) 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3876: delete oid 143 current snap is 0 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3869: finishing copy_from to smithi16030472-344 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 344 v 1009 (ObjNum 1482 snap 0 seq_num 1482) dirty exists 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:3871: got (0) Success 2024-09-21T18:10:23.669 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 168 v 0 (ObjNum 1113 snap 0 seq_num 1113) dirty exists 2024-09-21T18:10:23.670 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:23.664+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:23.672 INFO:tasks.rados.rados.0.smithi160.stdout:3869: done (3 left) 2024-09-21T18:10:23.672 INFO:tasks.rados.rados.0.smithi160.stdout:3871: done (2 left) 2024-09-21T18:10:23.673 INFO:tasks.rados.rados.0.smithi160.stdout:3876: done (1 left) 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3877: read oid 419 snap -1 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3877: expect (ObjNum 1594 snap 0 seq_num 1594) 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3878: cache_try_flush oid 265 snap -1 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3879: cache_try_flush oid 462 snap -1 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3880: read oid 58 snap -1 2024-09-21T18:10:23.674 INFO:tasks.rados.rados.0.smithi160.stdout:3880: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:23.675 INFO:tasks.rados.rados.0.smithi160.stdout:3881: delete oid 21 current snap is 0 2024-09-21T18:10:23.678 INFO:tasks.rados.rados.0.smithi160.stdout:3877: expect (ObjNum 1594 snap 0 seq_num 1594) 2024-09-21T18:10:23.678 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.734 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.747 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:23.744+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:23.789 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "size": 20, 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 600, 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [ 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:552 4.2 4:412eff35:::smithi16030472-123:head [setxattr _header (57) in=7b,truncate 2050454] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.792 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.11604510800001, 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843309019999999, 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 552 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:23.793 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672325+0000", 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672327+0000", 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.34e-06 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.794 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672341+0000", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4025999999999999e-05 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672359+0000", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7663e-05 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672374+0000", 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5150000000000001e-05 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.795 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672846+0000", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047216899999999998 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856649+0000", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.183802918 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856656+0000", 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6359999999999997e-06 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.796 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:553 4.2 4:412eff35:::smithi16030472-123:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.11597474600001, 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1843235329999999, 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 553 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.797 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672395+0000", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672397+0000", 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6300000000000001e-06 2024-09-21T18:10:23.798 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672404+0000", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.793e-06 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672427+0000", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3312999999999999e-05 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.672446+0000", 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9096000000000001e-05 2024-09-21T18:10:23.799 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684741+0000", 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.012294591000000001 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856719+0000", 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.17197811 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.800 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:554 4.2 4:7a50b852:::smithi16030472-124:head [write 467261~717275,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.10417331100001, 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1726082790000001, 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 554 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.801 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684197+0000", 2024-09-21T18:10:23.802 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684201+0000", 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.6629999999999999e-06 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684542+0000", 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00034140800000000002 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.803 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684573+0000", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.0630000000000003e-05 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.684588+0000", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5882000000000001e-05 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692591+0000", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0080021850000000002 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.804 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856803+0000", 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1642119049999999 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856805+0000", 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.6060000000000001e-06 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:555 4.2 4:7a50b852:::smithi16030472-124:head [write 1847991~219270] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.805 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.09630692799999, 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1648269849999999, 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 555 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.806 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692063+0000", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692067+0000", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.5130000000000001e-06 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.807 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692373+0000", 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00030637200000000001 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692403+0000", 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.9893e-05 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692419+0000", 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6243e-05 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.808 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692988+0000", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00056903100000000001 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856886+0000", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.163897703 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.856890+0000", 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2300000000000002e-06 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.809 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:556 4.2 4:7a50b852:::smithi16030472-124:head [setxattr _header (57) in=7b,truncate 2067261] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.09592128400001, 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.164567492, 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 556 2024-09-21T18:10:23.810 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692449+0000", 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.811 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692451+0000", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.7859999999999999e-06 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692457+0000", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2269999999999998e-06 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692471+0000", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3936e-05 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692480+0000", 2024-09-21T18:10:23.812 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.9800000000000004e-06 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.772962+0000", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.080482625000000002 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857014+0000", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0840511779999999 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.857016+0000", 2024-09-21T18:10:23.813 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7599999999999998e-06 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:557 4.2 4:7a50b852:::smithi16030472-124:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 185.09587583499999, 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.27278596, 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.814 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 557 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692494+0000", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.815 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692495+0000", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0359999999999999e-06 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692499+0000", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9999999999999998e-06 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692512+0000", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2354000000000001e-05 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.816 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.692520+0000", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2660000000000001e-06 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793328+0000", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100808472 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:18.793412+0000", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3712000000000001e-05 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862201+0000", 2024-09-21T18:10:23.817 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.068789051 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.862433+0000", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023203300000000001 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964726+0000", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102292308 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.964984+0000", 2024-09-21T18:10:23.818 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000258769 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.965280+0000", 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00029595899999999999 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:628 4.2 4:639708c1:::smithi16030472-142:head [setxattr _header (57) in=7b,truncate 2369787] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 184.04797791799999, 2024-09-21T18:10:23.819 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135316376, 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 628 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.820 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740392+0000", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740394+0000", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.353e-06 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740400+0000", 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.0939999999999996e-06 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.821 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740412+0000", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2716e-05 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740421+0000", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5029999999999997e-06 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740810+0000", 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00038863800000000002 2024-09-21T18:10:23.822 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875705+0000", 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1348951060000001 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875709+0000", 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9659999999999998e-06 2024-09-21T18:10:23.823 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:629 4.2 4:639708c1:::smithi16030472-142:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e47)", 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:23.824 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 184.047870276, 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.135234222, 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "reached pg", 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.825 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 629 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.826 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740500+0000", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740501+0000", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.28e-06 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740506+0000", 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.8860000000000003e-06 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.827 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740520+0000", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3797e-05 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.740530+0000", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.042e-05 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:19.840665+0000", 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.10013428200000001 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.828 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:20.875734+0000", 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0350695569999999 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1054 4.2 4:5b2f8cd5:::smithi16030472-234:head [setxattr _header (57) in=7b,truncate 3916031] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 178.47208766099999, 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2956394019999999, 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.829 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1054 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.830 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316283+0000", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316284+0000", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9570000000000001e-06 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316291+0000", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8170000000000001e-06 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.831 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316303+0000", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1619e-05 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316312+0000", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.0629999999999995e-06 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316545+0000", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00023341599999999999 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.832 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316702+0000", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015612499999999999 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409090+0000", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.092388873 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409247+0000", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015673799999999999 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:23.833 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409443+0000", 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000195501 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.512114+0000", 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.102671756 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611742+0000", 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.099627785999999996 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.834 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611820+0000", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.7774000000000004e-05 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.611922+0000", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000101977 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1055 4.2 4:5b2f8cd5:::smithi16030472-234:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:23.835 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 178.472035726, 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.296674111, 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1055 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.836 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316334+0000", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316336+0000", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0699999999999999e-06 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316340+0000", 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.5569999999999997e-06 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.837 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316350+0000", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9299999999999998e-06 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316358+0000", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0590000000000003e-06 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316831+0000", 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00047332899999999998 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.838 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:25.316917+0000", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.5827000000000003e-05 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.409975+0000", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0930574449999999 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.410113+0000", 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138503 2024-09-21T18:10:23.839 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612775+0000", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20266200300000001 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.612883+0000", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00010734399999999999 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613009+0000", 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00012604399999999999 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.840 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1115 4.2 4:5b8f0a5a:::smithi16030472-248:head [setxattr _header (57) in=7b,truncate 2151329] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 177.70533133800001, 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1873148819999999, 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1115 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.841 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083039+0000", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083041+0000", 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.2699999999999999e-06 2024-09-21T18:10:23.842 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083052+0000", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0376e-05 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.083071+0000", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.9576e-05 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.183243+0000", 2024-09-21T18:10:23.843 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100171593 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240036+0000", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.056793675000000002 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.240104+0000", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.7850999999999998e-05 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.157806+0000", 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.91770170600000001 2024-09-21T18:10:23.844 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158057+0000", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025140900000000002 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.158344+0000", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028629099999999999 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.159947+0000", 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.001603606 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.845 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270054+0000", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11010668799999999 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270189+0000", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000134986 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270354+0000", 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016485499999999999 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.846 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1122 4.2 4:52980938:::smithi16030472-250:head [write 1587781~597162] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 177.47622808, 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160225917, 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1122 2024-09-21T18:10:23.847 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312142+0000", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312145+0000", 2024-09-21T18:10:23.848 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.337e-06 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312494+0000", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000348389 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312521+0000", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.7602000000000001e-05 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312537+0000", 2024-09-21T18:10:23.849 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.5187e-05 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613526+0000", 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30098965300000002 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.850 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613574+0000", 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.7420000000000003e-05 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266074+0000", 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65250034300000004 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.851 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266362+0000", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028809099999999998 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.266738+0000", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00037607899999999997 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270731+0000", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0039924569999999996 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.852 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472232+0000", 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20150130999999999 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472310+0000", 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.8370999999999999e-05 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472368+0000", 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.7677999999999999e-05 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.853 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1123 4.2 4:52980938:::smithi16030472-250:head [setxattr _header (57) in=7b,truncate 2184943] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 177.47580009000001, 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.160047966, 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1123 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.854 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312570+0000", 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.855 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312572+0000", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.854e-06 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312579+0000", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.109e-06 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312592+0000", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.3127e-05 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.856 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312601+0000", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.6959999999999994e-06 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613614+0000", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30101319599999998 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613656+0000", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1943000000000001e-05 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.857 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.268980+0000", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65532396599999998 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269154+0000", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017407800000000001 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.269351+0000", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00019732800000000001 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.858 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271484+0000", 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0021330300000000002 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472470+0000", 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.20098528700000001 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472538+0000", 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.8021999999999993e-05 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.859 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472618+0000", 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.0329999999999998e-05 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1124 4.2 4:52980938:::smithi16030472-250:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e49)", 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 177.475753521, 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1604176980000001, 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.860 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1124 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312617+0000", 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.861 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312618+0000", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.0300000000000001e-06 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312622+0000", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.2830000000000003e-06 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312634+0000", 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.221e-05 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.862 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.312642+0000", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.2789999999999998e-06 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613712+0000", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.30106994300000001 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:26.613754+0000", 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.1838000000000002e-05 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.863 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.270813+0000", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.65705896500000005 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.271026+0000", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000213167 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472730+0000", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.201703732 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.864 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.472894+0000", 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016410200000000001 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:27.473034+0000", 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000140149 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:1537 4.2 4:7b415eca:::smithi16030472-342:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 172.97508446099999, 2024-09-21T18:10:23.865 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1311484219999999, 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 1537 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.866 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813286+0000", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813287+0000", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.9300000000000006e-07 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.867 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813291+0000", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 3.9670000000000003e-06 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813300+0000", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.4730000000000006e-06 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.813308+0000", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.1729999999999995e-06 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.868 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850536+0000", 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.037227784 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:30.850672+0000", 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013626299999999999 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.841929+0000", 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.99125689100000003 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.869 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.842210+0000", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00028076899999999999 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944168+0000", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.101957567 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944298+0000", 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013039 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.870 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:31.944434+0000", 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00013615200000000001 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2074 4.2 4:6954f9c8:::smithi16030472-461:head [setxattr _header (57) in=64b,truncate 3731220] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 165.967552665, 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2610923810000001, 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.871 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2074 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820818+0000", 2024-09-21T18:10:23.872 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820819+0000", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.7999999999999993e-07 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820823+0000", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 4.6530000000000002e-06 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820830+0000", 2024-09-21T18:10:23.873 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2080000000000003e-06 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820837+0000", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.2750000000000001e-06 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821506+0000", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00066919600000000005 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.821578+0000", 2024-09-21T18:10:23.874 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.2078999999999999e-05 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995124+0000", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.173546049 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995334+0000", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000210235 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.995820+0000", 2024-09-21T18:10:23.875 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00048570599999999997 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "op_commit", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.998030+0000", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.0022098769999999998 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081741+0000", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.083711664000000005 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081826+0000", 2024-09-21T18:10:23.876 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4484e-05 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.081910+0000", 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.3974999999999995e-05 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2075 4.2 4:6954f9c8:::smithi16030472-461:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 165.96747834499999, 2024-09-21T18:10:23.877 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.261432178, 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2075 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.878 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820892+0000", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820893+0000", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 7.9400000000000004e-07 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820895+0000", 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.5239999999999999e-06 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.879 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820902+0000", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.9670000000000004e-06 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.820909+0000", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6930000000000001e-06 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824152+0000", 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.003243119 2024-09-21T18:10:23.880 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:37.824236+0000", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 8.4128000000000005e-05 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996474+0000", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1722375439999999 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.996613+0000", 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000138913 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.881 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082004+0000", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.085391247000000003 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082160+0000", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00015623300000000001 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.082324+0000", 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00016401600000000001 2024-09-21T18:10:23.882 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2097 4.2 4:574ca510:::smithi16030472-466:head [write 615255~488318,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 165.56567509999999, 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.178681662, 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.883 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2097 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222695+0000", 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.884 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222698+0000", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.683e-06 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222950+0000", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00025203200000000002 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.222974+0000", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 2.3751999999999999e-05 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.885 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.323141+0000", 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.100166958 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.756717+0000", 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.43357621400000002 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "commit_sent", 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401367+0000", 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.64465021300000003 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.886 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.401377+0000", 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.8099999999999992e-06 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:2111 4.2 4:743666c8:::smithi16030472-468:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e53)", 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 165.463325212, 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.2106768299999999, 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.887 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 2111 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325045+0000", 2024-09-21T18:10:23.888 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325046+0000", 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4929999999999999e-06 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325053+0000", 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.6939999999999998e-06 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.889 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325070+0000", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.6986e-05 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.325088+0000", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.8206000000000002e-05 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958219+0000", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.63313078899999997 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:38.958284+0000", 2024-09-21T18:10:23.890 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 6.4474999999999995e-05 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420065+0000", 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.46178176399999998 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.420210+0000", 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00014471699999999999 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.891 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535387+0000", 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.11517717399999999 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535551+0000", 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000163921 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:07:39.535722+0000", 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.00017061100000000001 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.892 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "description": "osd_op(client.4876.0:6670 4.2 4:574ca510:::smithi16030472-466:head [cache-flush] snapc 0=[] ondisk+read+ignore_cache+known_if_redirected+supports_pool_eio e95)", 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "initiated_at": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "age": 73.150696814, 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.4286385400000001, 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "continuous": false, 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "type_data": { 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "flag_point": "started", 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "client_info": { 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "client": "client.4876", 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "tid": 6670 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.893 INFO:teuthology.orchestra.run.smithi160.stdout: "events": [ 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "initiated", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "header_read", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637673+0000", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "throttled", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637674+0000", 2024-09-21T18:10:23.894 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.3699999999999999e-07 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "all_read", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637680+0000", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 5.6629999999999998e-06 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "dispatched", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637691+0000", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.1097000000000001e-05 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "queued_for_pg", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:10.637700+0000", 2024-09-21T18:10:23.895 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 9.2490000000000007e-06 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054141+0000", 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.41644096200000003 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:11.054299+0000", 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000157572 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "reached_pg", 2024-09-21T18:10:23.896 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.065938+0000", 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 1.011639553 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "started", 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066192+0000", 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000253222 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: }, 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: { 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "event": "done", 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "time": "2024-09-21T18:09:12.066312+0000", 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: "duration": 0.000120285 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.897 INFO:teuthology.orchestra.run.smithi160.stdout: } 2024-09-21T18:10:23.898 INFO:teuthology.orchestra.run.smithi160.stdout: ] 2024-09-21T18:10:23.898 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:10:23.899 INFO:tasks.rados.rados.0.smithi160.stdout:3880: expect (ObjNum 1423 snap 0 seq_num 1423) 2024-09-21T18:10:23.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:10:23.900 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:23.895+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:23.924 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.000 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:23.995+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:24.016 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.036 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.036 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:24.036 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-09-21T18:10:24.036 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:10:24.101 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:24.097+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3873: got (0) Success 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 232 v 0 (ObjNum 129 snap 0 seq_num 3138276016) clean dne 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3879: got (0) Success 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 462 v 0 (ObjNum 49 snap 0 seq_num 3111024432) clean dne 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3878: got (0) Success 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 265 v 0 (ObjNum 1565 snap 0 seq_num 1565) dirty exists 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3873: done (5 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3877: done (4 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3878: done (3 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3879: done (2 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3880: done (1 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3881: done (0 left) 2024-09-21T18:10:24.110 INFO:tasks.rados.rados.0.smithi160.stdout:3882: read oid 340 snap -1 2024-09-21T18:10:24.111 INFO:tasks.rados.rados.0.smithi160.stdout:3882: expect (ObjNum 1579 snap 0 seq_num 1579) 2024-09-21T18:10:24.111 INFO:tasks.rados.rados.0.smithi160.stdout:3883: copy_from oid 198 from oid 113 current snap is 0 2024-09-21T18:10:24.111 INFO:tasks.rados.rados.0.smithi160.stdout:3884: write (excl) oid 144 current snap is 0 2024-09-21T18:10:24.111 INFO:tasks.rados.rados.0.smithi160.stdout:3884: seq_num 1599 ranges {474393=757281,1652825=595891} 2024-09-21T18:10:24.130 INFO:tasks.rados.rados.0.smithi160.stdout:3884: writing smithi16030472-144 from 474393 to 1231674 tid 1 2024-09-21T18:10:24.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T18:10:24.147 INFO:tasks.rados.rados.0.smithi160.stdout:3884: writing smithi16030472-144 from 1652825 to 2248716 tid 2 2024-09-21T18:10:24.148 INFO:tasks.rados.rados.0.smithi160.stdout:3885: read oid 172 snap -1 2024-09-21T18:10:24.148 INFO:tasks.rados.rados.0.smithi160.stdout:3885: expect (ObjNum 1290 snap 0 seq_num 1290) 2024-09-21T18:10:24.148 INFO:tasks.rados.rados.0.smithi160.stdout:3886: cache_evict oid 472 snap -1 2024-09-21T18:10:24.148 INFO:tasks.rados.rados.0.smithi160.stdout:3887: write (excl) oid 424 current snap is 0 2024-09-21T18:10:24.148 INFO:tasks.rados.rados.0.smithi160.stdout:3887: seq_num 1600 ranges {491200=432530,1594954=739549} 2024-09-21T18:10:24.161 INFO:tasks.rados.rados.0.smithi160.stdout:3887: writing smithi16030472-424 from 491200 to 923730 tid 1 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3887: writing smithi16030472-424 from 1594954 to 2334503 tid 2 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3888: copy_from oid 277 from oid 470 current snap is 0 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3886: got (16) Device or resource busy 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3887: finishing write tid 1 to smithi16030472-424 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3886: done (6 left) 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3889: read oid 222 snap -1 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3889: expect (ObjNum 988 snap 0 seq_num 988) 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3890: read oid 248 snap -1 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3890: expect (ObjNum 1075 snap 0 seq_num 1075) 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3891: write (excl) oid 208 current snap is 0 2024-09-21T18:10:24.179 INFO:tasks.rados.rados.0.smithi160.stdout:3891: seq_num 1601 ranges {508007=507779,1653436=599560} 2024-09-21T18:10:24.180 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.180 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:24.180 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:24.180 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:24.180 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.190 INFO:tasks.rados.rados.0.smithi160.stdout:3891: writing smithi16030472-208 from 508007 to 1015786 tid 1 2024-09-21T18:10:24.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:10:24.201 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:24.201+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:3891: writing smithi16030472-208 from 1653436 to 2252996 tid 2 2024-09-21T18:10:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:3892: read oid 153 snap -1 2024-09-21T18:10:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:3892: expect deleted 2024-09-21T18:10:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:3893: copy_from oid 313 from oid 253 current snap is 0 2024-09-21T18:10:24.205 INFO:tasks.rados.rados.0.smithi160.stdout:3890: expect (ObjNum 1075 snap 0 seq_num 1075) 2024-09-21T18:10:24.252 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1899402 2024-09-21T18:10:24.303 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:24.300+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:24.315 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.346 INFO:tasks.rados.rados.0.smithi160.stdout:3891: finishing write tid 1 to smithi16030472-208 2024-09-21T18:10:24.347 INFO:tasks.rados.rados.0.smithi160.stdout:3890: done (10 left) 2024-09-21T18:10:24.347 INFO:tasks.rados.rados.0.smithi160.stdout:3894: cache_flush oid 408 snap -1 2024-09-21T18:10:24.347 INFO:tasks.rados.rados.0.smithi160.stdout:3882: expect (ObjNum 1579 snap 0 seq_num 1579) 2024-09-21T18:10:24.347 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.365 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.365 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:24.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:24.365 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [] 2024-09-21T18:10:24.365 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:10:24.403 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:24.399+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: finishing write tid 2 to smithi16030472-424 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: finishing write tid 3 to smithi16030472-424 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: finishing write tid 4 to smithi16030472-424 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: oid 424 updating version 0 to 1151 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: oid 424 version 1151 is already newer than 1098 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: oid 424 version 1151 is already newer than 1150 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 424 v 1151 (ObjNum 1600 snap 0 seq_num 1600) dirty exists 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3887: left oid 424 (ObjNum 1600 snap 0 seq_num 1600) 2024-09-21T18:10:24.456 INFO:tasks.rados.rados.0.smithi160.stdout:3883: finishing copy_from racing read to smithi16030472-198 2024-09-21T18:10:24.457 INFO:tasks.rados.rados.0.smithi160.stdout:3885: expect (ObjNum 1290 snap 0 seq_num 1290) 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:10110 4.0 4:029c904c:::smithi16030472-198:head [copy-from ver 496 in=56b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:24.109799+0000", 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.38815209699999997, 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38818288899999998, 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 10110 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109799+0000", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109799+0000", 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109800+0000", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.44e-07 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109807+0000", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6059999999999997e-06 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109813+0000", 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1330000000000004e-06 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.109817+0000", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.191e-06 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.315206+0000", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20538967499999999 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.315327+0000", 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000120678 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.426698+0000", 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.111370998 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.433180+0000", 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0064819040000000001 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:10112 4.0 4:29dccdbd:::smithi16030472-144:head [write 474393~757281 in=757281b,stat out=16b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T18:10:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:24.130415+0000", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.367536158, 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.36762605300000001, 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 10112 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.130415+0000", 2024-09-21T18:10:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.130415+0000", 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.130418+0000", 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3519999999999998e-06 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.131890+0000", 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001471738 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.131938+0000", 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7639999999999998e-05 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.131958+0000", 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0406000000000001e-05 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.315610+0000", 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18365126600000001 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.315723+0000", 2024-09-21T18:10:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113424 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.315889+0000", 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016602699999999999 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.320730+0000", 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048413650000000003 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:10113 4.0 4:29dccdbd:::smithi16030472-144:head [write 1652825~595891 in=595891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:24.147280+0000", 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.35067127799999998, 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35080908199999999, 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 10113 2024-09-21T18:10:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.147280+0000", 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.147280+0000", 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.147286+0000", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.536e-06 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148024+0000", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073812400000000005 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148063+0000", 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9459000000000003e-05 2024-09-21T18:10:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148084+0000", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0380000000000001e-05 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.317938+0000", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.169854532 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.318008+0000", 2024-09-21T18:10:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9638e-05 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.318093+0000", 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5541e-05 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.320738+0000", 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026447250000000001 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:10114 4.0 4:29dccdbd:::smithi16030472-144:head [setxattr _header (57) in=64b,truncate 2874393] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:24.148123+0000", 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.34982802200000002, 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35002254999999999, 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-09-21T18:10:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 10114 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148123+0000", 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148123+0000", 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148125+0000", 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.84e-06 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148133+0000", 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4869999999999996e-06 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148145+0000", 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2499e-05 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148154+0000", 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4999999999999999e-06 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.319678+0000", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.17152439799999999 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.319796+0000", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000118206 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.319969+0000", 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017244800000000001 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.321032+0000", 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001063381 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:10115 4.0 4:29dccdbd:::smithi16030472-144:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e126)", 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:10:24.148168+0000", 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.349783173, 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.350025064, 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for rw locks", 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 10115 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148168+0000", 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148168+0000", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148169+0000", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.133e-06 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148173+0000", 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0369999999999996e-06 2024-09-21T18:10:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148185+0000", 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1362999999999999e-05 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.148193+0000", 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0139999999999998e-06 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.320759+0000", 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.17256664699999999 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:10:24.320816+0000", 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6215999999999998e-05 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-09-21T18:10:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 5 2024-09-21T18:10:24.524 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.525 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1755566 2024-09-21T18:10:24.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:10:24.526 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:24.521+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:24.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:24.551 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.605 INFO:tasks.rados.rados.0.smithi160.stdout:3889: expect (ObjNum 988 snap 0 seq_num 988) 2024-09-21T18:10:24.608 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 572110 2024-09-21T18:10:24.626 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:24.625+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:24.653 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.653 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-09-21T18:10:24.654 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-09-21T18:10:24.654 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-09-21T18:10:24.654 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:10:24.669 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:24.703 INFO:tasks.rados.rados.0.smithi160.stdout:3893: finishing copy_from to smithi16030472-313 2024-09-21T18:10:24.703 INFO:tasks.rados.rados.0.smithi160.stdout:3893: got expected ENOENT (src dne) 2024-09-21T18:10:24.703 INFO:tasks.rados.rados.0.smithi160.stdout:3893: finishing copy_from racing read to smithi16030472-313 2024-09-21T18:10:24.703 INFO:tasks.rados.rados.0.smithi160.stdout:3891: finishing write tid 2 to smithi16030472-208 2024-09-21T18:10:24.703 INFO:tasks.rados.rados.0.smithi160.stdout:3891: finishing write tid 3 to smithi16030472-208 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3891: finishing write tid 4 to smithi16030472-208 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3891: oid 208 updating version 0 to 1052 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3891: oid 208 version 1052 is already newer than 1051 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3891: oid 208 updating version 1052 to 1053 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 208 v 1053 (ObjNum 1601 snap 0 seq_num 1601) dirty exists 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3891: left oid 208 (ObjNum 1601 snap 0 seq_num 1601) 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3894: got (0) Success 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 0 (ObjNum 819 snap 0 seq_num 819) dirty exists 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: finishing write tid 1 to smithi16030472-144 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: finishing write tid 2 to smithi16030472-144 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: finishing write tid 3 to smithi16030472-144 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: finishing write tid 4 to smithi16030472-144 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: oid 144 updating version 0 to 1125 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: oid 144 version 1125 is already newer than 1124 2024-09-21T18:10:24.704 INFO:tasks.rados.rados.0.smithi160.stdout:3884: oid 144 version 1125 is already newer than 1123 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 144 v 1125 (ObjNum 1599 snap 0 seq_num 1599) dirty exists 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3884: left oid 144 (ObjNum 1599 snap 0 seq_num 1599) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3888: finishing copy_from to smithi16030472-277 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3888: got expected ENOENT (src dne) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3883: finishing copy_from to smithi16030472-198 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 198 v 496 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3888: finishing copy_from racing read to smithi16030472-277 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3882: done (10 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3883: done (9 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3884: done (8 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3885: done (7 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3887: done (6 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3888: done (5 left) 2024-09-21T18:10:24.705 INFO:tasks.rados.rados.0.smithi160.stdout:3889: done (4 left) 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3891: done (3 left) 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3892: done (2 left) 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3893: done (1 left) 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3894: done (0 left) 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3895: cache_try_flush oid 338 snap -1 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3896: cache_evict oid 186 snap -1 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3897: write oid 426 current snap is 0 2024-09-21T18:10:24.706 INFO:tasks.rados.rados.0.smithi160.stdout:3897: seq_num 1602 ranges {524814=583028,1711918=743218} 2024-09-21T18:10:24.721 INFO:tasks.rados.rados.0.smithi160.stdout:3897: writing smithi16030472-426 from 524814 to 1107842 tid 1 2024-09-21T18:10:24.722 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:24.722 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:24.727 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:24.723+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:24.743 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:24.743 INFO:tasks.rados.rados.0.smithi160.stdout:3897: writing smithi16030472-426 from 1711918 to 2455136 tid 2 2024-09-21T18:10:24.743 INFO:tasks.rados.rados.0.smithi160.stdout:3898: delete oid 108 current snap is 0 2024-09-21T18:10:24.744 INFO:tasks.rados.rados.0.smithi160.stdout:3896: got (0) Success 2024-09-21T18:10:24.763 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:24.763 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:24.783 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:24.784 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:159 4.0 4:1192aa23:::smithi16030472-40:head [setxattr _header (56) in=7b,truncate 2655473] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e44)", 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:24.792 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 191.57981943799999, 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175331831, 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 159 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209798+0000", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209799+0000", 2024-09-21T18:10:24.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.096e-06 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209804+0000", 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4510000000000002e-06 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.209809+0000", 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.309945+0000", 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100135848 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310084+0000", 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013899499999999999 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.310170+0000", 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6139999999999999e-05 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733688+0000", 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42351819800000001 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.733841+0000", 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015224499999999999 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.734025+0000", 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018426 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:13.742597+0000", 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085724570000000003 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385037+0000", 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64244015700000001 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385081+0000", 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3180999999999999e-05 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:14.385130+0000", 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9205000000000001e-05 2024-09-21T18:10:24.799 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1532 4.0 4:2aa3e9e6:::smithi16030472-341:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 174.035881889, 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2433444090000001, 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.800 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1532 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:24.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753736+0000", 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2799999999999995e-07 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753739+0000", 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0939999999999999e-06 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753746+0000", 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8979999999999999e-06 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.753752+0000", 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6620000000000002e-06 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935534+0000", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18178228099999999 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.935589+0000", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4926999999999998e-05 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.595981+0000", 2024-09-21T18:10:24.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.66039214700000004 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.596246+0000", 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026437500000000002 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996705+0000", 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400459648 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.996879+0000", 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017414199999999999 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.997080+0000", 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000200507 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1541 4.0 4:1baed0db:::smithi16030472-343:head [write 3605976~142018 in=142018b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.87495842499999, 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1980853970000001, 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.807 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1541 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914659+0000", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.914661+0000", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4540000000000001e-06 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915294+0000", 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00063220099999999999 2024-09-21T18:10:24.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915310+0000", 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6486000000000001e-05 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915319+0000", 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5580000000000001e-06 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936471+0000", 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021152741999999999 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112735+0000", 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.176263823 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112744+0000", 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1330000000000005e-06 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1542 4.0 4:1baed0db:::smithi16030472-343:head [setxattr _header (57) in=64b,truncate 3747994] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.87428383, 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974697089999999, 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.812 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1542 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915334+0000", 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915335+0000", 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915339+0000", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4800000000000003e-06 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915348+0000", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.473e-06 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915352+0000", 2024-09-21T18:10:24.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5229999999999997e-06 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936667+0000", 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021315447000000001 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112799+0000", 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1761313170000001 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112803+0000", 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5090000000000004e-06 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1543 4.0 4:1baed0db:::smithi16030472-343:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.87425729200001, 2024-09-21T18:10:24.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1974842830000001, 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1543 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915360+0000", 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915361+0000", 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700000000000003e-07 2024-09-21T18:10:24.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915363+0000", 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2570000000000002e-06 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915370+0000", 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0489999999999998e-06 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.915374+0000", 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1559999999999997e-06 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.936862+0000", 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.021487707000000002 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.112844+0000", 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175982547 2024-09-21T18:10:24.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1544 4.0 4:2f6cf193:::smithi16030472-344:head [write 564801~646292 in=646292b,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.86442355, 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.213267965, 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.822 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1544 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925194+0000", 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.925198+0000", 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7000000000000002e-06 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926032+0000", 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083487999999999998 2024-09-21T18:10:24.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926064+0000", 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1198999999999997e-05 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.926080+0000", 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6019e-05 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938860+0000", 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012780563999999999 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138456+0000", 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1995954440000001 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.138462+0000", 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1589999999999998e-06 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1545 4.0 4:2f6cf193:::smithi16030472-344:head [write 1921559~742115 in=742115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.85201176199999, 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.63388347, 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1545 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937606+0000", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.937609+0000", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7340000000000002e-06 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938709+0000", 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010995930000000001 2024-09-21T18:10:24.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938735+0000", 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6492999999999999e-05 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.938751+0000", 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5024999999999999e-05 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947435+0000", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086846760000000005 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947531+0000", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5439000000000006e-05 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364459+0000", 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.416928301 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364559+0000", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9899e-05 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.364696+0000", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000136969 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367507+0000", 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028109989999999998 2024-09-21T18:10:24.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571364+0000", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20385711500000001 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571455+0000", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0792999999999996e-05 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571489+0000", 2024-09-21T18:10:24.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4434000000000003e-05 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1546 4.0 4:2f6cf193:::smithi16030472-344:head [write 3202910~561891 in=561891b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.84312169200001, 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7252863409999999, 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1546 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946496+0000", 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.946499+0000", 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.303e-06 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947231+0000", 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073158100000000003 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947260+0000", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9128999999999998e-05 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947274+0000", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4442e-05 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947579+0000", 2024-09-21T18:10:24.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030483600000000001 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947636+0000", 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7144e-05 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366350+0000", 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4187133750000001 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366408+0000", 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8548999999999998e-05 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.366484+0000", 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5823999999999998e-05 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368090+0000", 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001606033 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671637+0000", 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30354753600000001 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671721+0000", 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3123999999999999e-05 2024-09-21T18:10:24.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671782+0000", 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1464999999999995e-05 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1547 4.0 4:2f6cf193:::smithi16030472-344:head [setxattr _header (57) in=7b,truncate 3764801] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:24.841 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.842309675, 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7246675169999999, 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1547 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947308+0000", 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947310+0000", 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.854e-06 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947319+0000", 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7049999999999994e-06 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947340+0000", 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0329000000000001e-05 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947355+0000", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5577e-05 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035544+0000", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.088188995000000006 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.035647+0000", 2024-09-21T18:10:24.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102525 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367682+0000", 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.332035547 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367737+0000", 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4889000000000003e-05 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367815+0000", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8116999999999998e-05 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.571324+0000", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20350843399999999 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671852+0000", 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100527934 2024-09-21T18:10:24.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671909+0000", 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7602000000000002e-05 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.671975+0000", 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6008999999999998e-05 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1548 4.0 4:2f6cf193:::smithi16030472-344:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.84220146300001, 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7248883259999999, 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1548 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.849 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947416+0000", 2024-09-21T18:10:24.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947418+0000", 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1229999999999998e-06 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947426+0000", 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:30.947448+0000", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2345999999999999e-05 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047607+0000", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100158845 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047769+0000", 2024-09-21T18:10:24.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016145499999999999 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047904+0000", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013541399999999999 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368037+0000", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.320132635 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.368084+0000", 2024-09-21T18:10:24.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7392000000000003e-05 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672097+0000", 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.304012532 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672212+0000", 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115136 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.672304+0000", 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2591999999999997e-05 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1549 4.0 4:05986648:::smithi16030472-345:head [write 581608~721541,stat] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.741930317, 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1917094269999999, 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.855 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1549 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:24.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047687+0000", 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.047692+0000", 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0529999999999997e-06 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048528+0000", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083604999999999997 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048544+0000", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5838e-05 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048553+0000", 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-09-21T18:10:24.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048624+0000", 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0554000000000002e-05 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239391+0000", 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1907674150000001 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239397+0000", 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4190000000000001e-06 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1550 4.0 4:05986648:::smithi16030472-345:head [write 1863688~485773] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.741034058, 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190848101, 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.860 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1550 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048583+0000", 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.048585+0000", 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-06 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049029+0000", 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044460000000000002 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049043+0000", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3689000000000001e-05 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049051+0000", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.498e-06 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049086+0000", 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5697000000000001e-05 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1903443650000001 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239431+0000", 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1551 4.0 4:05986648:::smithi16030472-345:head [write 3113980~604043] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.74055244499999, 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1906467649999999, 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.865 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1551 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049065+0000", 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049066+0000", 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049486+0000", 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041977799999999999 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049498+0000", 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1831000000000001e-05 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049505+0000", 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4630000000000004e-06 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049566+0000", 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0575999999999997e-05 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239710+0000", 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.19014442 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239712+0000", 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.522e-06 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1552 4.0 4:05986648:::smithi16030472-345:head [setxattr _header (57) in=7b,truncate 3781608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.74010002, 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1904454280000001, 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.870 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1552 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049517+0000", 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049518+0000", 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8299999999999995e-07 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049524+0000", 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5720000000000002e-06 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049532+0000", 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6009999999999997e-06 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049539+0000", 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4719999999999998e-06 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.149898+0000", 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100359359 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239961+0000", 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.090062949 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.239963+0000", 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1553 4.0 4:05986648:::smithi16030472-345:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.74004969399999, 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190943756, 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1553 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.875 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049568+0000", 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049569+0000", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0440000000000001e-06 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049574+0000", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7360000000000001e-06 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049582+0000", 2024-09-21T18:10:24.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9740000000000008e-06 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.049592+0000", 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1290000000000002e-06 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.150269+0000", 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.10067783399999999 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.240511+0000", 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0902420390000001 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1566 4.0 4:03b3ee07:::smithi16030472-348:head [write 2796249~437859] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.614316914, 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1921979629999999, 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.879 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1566 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175301+0000", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175304+0000", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4259999999999999e-06 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175668+0000", 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036432000000000001 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175698+0000", 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0045000000000001e-05 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175714+0000", 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5330000000000001e-05 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175847+0000", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133604 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.175954+0000", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106518 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255123+0000", 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0791692420000001 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255230+0000", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010719 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.255382+0000", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015215000000000001 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.359435+0000", 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.104052353 2024-09-21T18:10:24.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367415+0000", 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079804720000000006 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367462+0000", 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6634000000000001e-05 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367498+0000", 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6678999999999999e-05 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1567 4.0 4:03b3ee07:::smithi16030472-348:head [write 3653524~178505] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.61003885700001, 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.18799302, 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.886 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1567 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:24.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179579+0000", 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179582+0000", 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.703e-06 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179861+0000", 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027907700000000003 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179892+0000", 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1109000000000003e-05 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179911+0000", 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8303000000000001e-05 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180056+0000", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014548299999999999 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180168+0000", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011127499999999999 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.356967+0000", 2024-09-21T18:10:24.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1767997800000001 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357151+0000", 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183303 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.357398+0000", 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024738400000000002 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367359+0000", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099615160000000001 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367529+0000", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016910199999999999 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367552+0000", 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3611000000000001e-05 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367572+0000", 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9374000000000001e-05 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1568 4.0 4:03b3ee07:::smithi16030472-348:head [setxattr _header (57) in=7b,truncate 3832029] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:24.893 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.60966861700001, 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.187705918, 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1568 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179949+0000", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179951+0000", 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2029999999999999e-06 2024-09-21T18:10:24.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179961+0000", 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0332999999999999e-05 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179980+0000", 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8479e-05 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.179995+0000", 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5553000000000002e-05 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180270+0000", 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027413399999999999 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180360+0000", 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0512000000000003e-05 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358804+0000", 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.178444112 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358869+0000", 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5111999999999996e-05 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 10", 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.358954+0000", 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4395999999999996e-05 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367363+0000", 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084091900000000004 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-09-21T18:10:24.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367615+0000", 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025188899999999999 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367637+0000", 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.23e-05 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.367655+0000", 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7705000000000001e-05 2024-09-21T18:10:24.900 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1569 4.0 4:03b3ee07:::smithi16030472-348:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.609598887, 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3884621639999999, 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:24.901 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1569 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180019+0000", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180020+0000", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7400000000000001e-06 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180028+0000", 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4329999999999998e-06 2024-09-21T18:10:24.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180045+0000", 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6816e-05 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180060+0000", 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5722e-05 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180404+0000", 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000343524 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.180468+0000", 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4090999999999993e-05 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568313+0000", 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3878451759999999 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568394+0000", 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0668999999999996e-05 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.568481+0000", 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6992999999999999e-05 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.4876.0:1618 4.0 4:385a7674:::smithi16030472-358:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e51)", 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.01802189700001, 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4361519540000001, 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "continuous": false, 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.4876", 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "v1:172.21.15.160:0/2675454836", 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 1618 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.907 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771596+0000", 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771597+0000", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5230000000000001e-06 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771603+0000", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.896e-06 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771620+0000", 2024-09-21T18:10:24.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7326e-05 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.771630+0000", 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8530000000000004e-06 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871677+0000", 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.100046994 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for blocked object", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:31.871752+0000", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4474999999999994e-05 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.885775+0000", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0140232950000001 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for rw locks", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:32.886033+0000", 2024-09-21T18:10:24.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025854900000000001 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207552+0000", 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32151863000000003 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207634+0000", 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2385000000000005e-05 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-09-21T18:10:24.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-09-21T18:07:33.207747+0000", 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113028 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-09-21T18:10:24.913 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:10:24.914 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:24.915 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:24.915 INFO:tasks.rados.rados.0.smithi160.stdout:3895: got (0) Success 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 338 v 0 (ObjNum 81 snap 0 seq_num 1) clean dne 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: finishing write tid 1 to smithi16030472-426 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: finishing write tid 2 to smithi16030472-426 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: finishing write tid 3 to smithi16030472-426 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: finishing write tid 4 to smithi16030472-426 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: oid 426 updating version 0 to 1086 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: oid 426 version 1086 is already newer than 1085 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: oid 426 updating version 1086 to 1087 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 426 v 1087 (ObjNum 1602 snap 0 seq_num 1602) dirty exists 2024-09-21T18:10:24.916 INFO:tasks.rados.rados.0.smithi160.stdout:3897: left oid 426 (ObjNum 1602 snap 0 seq_num 1602) 2024-09-21T18:10:24.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:10:24.917 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:24.918 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:24.918 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:24.919 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:24.919 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:24.919 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:24.920 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:24.920 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:24.921 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:24.911+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:24.924 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:24.924 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:24.942 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:24.942 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:3895: done (3 left) 2024-09-21T18:10:24.948 INFO:tasks.rados.rados.0.smithi160.stdout:3896: done (2 left) 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3897: done (1 left) 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3898: done (0 left) 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3899: read oid 311 snap -1 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3899: expect (ObjNum 1291 snap 0 seq_num 1291) 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3900: read oid 115 snap -1 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3900: expect deleted 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3901: write (excl) oid 448 current snap is 0 2024-09-21T18:10:24.949 INFO:tasks.rados.rados.0.smithi160.stdout:3901: seq_num 1604 ranges {558428=733526,1712529=746887,2948598=9830} 2024-09-21T18:10:24.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:24.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:24.969 INFO:tasks.rados.rados.0.smithi160.stdout:3901: writing smithi16030472-448 from 558428 to 1291954 tid 1 2024-09-21T18:10:24.986 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:24.988 INFO:tasks.rados.rados.0.smithi160.stdout:3901: writing smithi16030472-448 from 1712529 to 2459416 tid 2 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3901: writing smithi16030472-448 from 2948598 to 2958428 tid 3 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3902: cache_evict oid 179 snap -1 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3903: cache_try_flush oid 42 snap -1 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3904: read oid 302 snap -1 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3904: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3905: write oid 471 current snap is 0 2024-09-21T18:10:24.997 INFO:tasks.rados.rados.0.smithi160.stdout:3905: seq_num 1605 ranges {575235=525128,1887364=606898} 2024-09-21T18:10:25.009 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:25.010 INFO:tasks.rados.rados.0.smithi160.stdout:3905: writing smithi16030472-471 from 575235 to 1100363 tid 1 2024-09-21T18:10:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:25.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:25.016 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:25.011+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3905: writing smithi16030472-471 from 1887364 to 2494262 tid 2 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3906: read oid 133 snap -1 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3906: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3907: cache_try_flush oid 5 snap -1 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3902: got (16) Device or resource busy 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3902: done (8 left) 2024-09-21T18:10:25.025 INFO:tasks.rados.rados.0.smithi160.stdout:3908: write (excl) oid 495 current snap is 0 2024-09-21T18:10:25.026 INFO:tasks.rados.rados.0.smithi160.stdout:3908: seq_num 1606 ranges {592042=600377,1945846=750556} 2024-09-21T18:10:25.035 INFO:tasks.rados.rados.0.smithi160.stdout:3908: writing smithi16030472-495 from 592042 to 1192419 tid 1 2024-09-21T18:10:25.049 INFO:tasks.rados.rados.0.smithi160.stdout:3908: writing smithi16030472-495 from 1945846 to 2696402 tid 2 2024-09-21T18:10:25.049 INFO:tasks.rados.rados.0.smithi160.stdout:3909: cache_try_flush oid 216 snap -1 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3910: read oid 337 snap -1 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3910: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3911: cache_try_flush oid 433 snap -1 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3912: cache_evict oid 236 snap -1 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3913: cache_flush oid 224 snap -1 2024-09-21T18:10:25.050 INFO:tasks.rados.rados.0.smithi160.stdout:3914: delete oid 81 current snap is 0 2024-09-21T18:10:25.059 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.059 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:10:25.116 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:25.112+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:25.124 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-09-21T18:10:25.150 INFO:tasks.rados.rados.0.smithi160.stdout:3903: got (0) Success 2024-09-21T18:10:25.150 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 42 v 0 (ObjNum 1423 snap 0 seq_num 1423) dirty exists 2024-09-21T18:10:25.167 INFO:tasks.rados.rados.0.smithi160.stdout:3910: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:10:25.168 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:25.197 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.197 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:10:25.217 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:25.217+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:25.306 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3043122 2024-09-21T18:10:25.318 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:25.313+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:25.325 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:25.339 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.339 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:10:25.388 INFO:tasks.rados.rados.0.smithi160.stdout:3905: finishing write tid 1 to smithi16030472-471 2024-09-21T18:10:25.419 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:25.415+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:25.455 INFO:tasks.rados.rados.0.smithi160.stdout:3905: finishing write tid 2 to smithi16030472-471 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: finishing write tid 3 to smithi16030472-471 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: finishing write tid 4 to smithi16030472-471 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: oid 471 updating version 0 to 1062 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: oid 471 updating version 1062 to 1131 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: oid 471 version 1131 is already newer than 1130 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 471 v 1131 (ObjNum 1605 snap 0 seq_num 1605) dirty exists 2024-09-21T18:10:25.458 INFO:tasks.rados.rados.0.smithi160.stdout:3905: left oid 471 (ObjNum 1605 snap 0 seq_num 1605) 2024-09-21T18:10:25.520 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:25.515+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:25.525 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.525 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:10:25.558 INFO:tasks.rados.rados.0.smithi160.stdout:3899: expect (ObjNum 1291 snap 0 seq_num 1291) 2024-09-21T18:10:25.558 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:25.621 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:25.617+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:25.651 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:25.669 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.669 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:10:25.722 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:25.715+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:25.774 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3348046 2024-09-21T18:10:25.784 INFO:tasks.rados.rados.0.smithi160.stdout:3904: expect (ObjNum 592 snap 0 seq_num 592) 2024-09-21T18:10:25.785 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:25.823 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:25.819+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:25.855 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:25.855 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:25.870 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:10:25.920 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2234415 2024-09-21T18:10:25.924 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:25.920+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: finishing write tid 1 to smithi16030472-448 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: finishing write tid 2 to smithi16030472-448 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: finishing write tid 3 to smithi16030472-448 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: finishing write tid 4 to smithi16030472-448 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: finishing write tid 5 to smithi16030472-448 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: oid 448 updating version 0 to 1090 2024-09-21T18:10:25.956 INFO:tasks.rados.rados.0.smithi160.stdout:3901: oid 448 updating version 1090 to 1091 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:3901: oid 448 version 1091 is already newer than 1089 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:3901: oid 448 version 1091 is already newer than 1088 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 448 v 1091 (ObjNum 1604 snap 0 seq_num 1604) dirty exists 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:3901: left oid 448 (ObjNum 1604 snap 0 seq_num 1604) 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:3906: expect (ObjNum 1047 snap 0 seq_num 1047) 2024-09-21T18:10:25.957 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:26.003 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.003 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.017 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:10:26.019 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1453275 2024-09-21T18:10:26.025 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:26.021+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:26.053 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3909: got (0) Success 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 216 v 0 (ObjNum 1461 snap 0 seq_num 1461) dirty exists 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: finishing write tid 1 to smithi16030472-495 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: finishing write tid 2 to smithi16030472-495 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: finishing write tid 3 to smithi16030472-495 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: finishing write tid 4 to smithi16030472-495 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: oid 495 updating version 0 to 1093 2024-09-21T18:10:26.105 INFO:tasks.rados.rados.0.smithi160.stdout:3908: oid 495 version 1093 is already newer than 1092 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3908: oid 495 updating version 1093 to 1094 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 495 v 1094 (ObjNum 1606 snap 0 seq_num 1606) dirty exists 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3908: left oid 495 (ObjNum 1606 snap 0 seq_num 1606) 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3912: got (0) Success 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3913: got (0) Success 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 224 v 0 (ObjNum 1499 snap 0 seq_num 1499) dirty exists 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3911: got (0) Success 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 0 (ObjNum 1261 snap 0 seq_num 1261) dirty exists 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3907: got (0) Success 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 5 v 0 (ObjNum 1307 snap 0 seq_num 1307) dirty exists 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3899: done (14 left) 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3900: done (13 left) 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3901: done (12 left) 2024-09-21T18:10:26.106 INFO:tasks.rados.rados.0.smithi160.stdout:3903: done (11 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3904: done (10 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3905: done (9 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3906: done (8 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3907: done (7 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3908: done (6 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3909: done (5 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3910: done (4 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3911: done (3 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3912: done (2 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3913: done (1 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3914: done (0 left) 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3915: write (excl) oid 25 current snap is 0 2024-09-21T18:10:26.107 INFO:tasks.rados.rados.0.smithi160.stdout:3915: seq_num 1608 ranges {625656=750875,2062810=754225} 2024-09-21T18:10:26.126 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:26.119+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:26.127 INFO:tasks.rados.rados.0.smithi160.stdout:3915: writing smithi16030472-25 from 625656 to 1376531 tid 1 2024-09-21T18:10:26.143 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.144 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3915: writing smithi16030472-25 from 2062810 to 2817035 tid 2 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3916: cache_flush oid 76 snap -1 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3917: cache_flush oid 255 snap -1 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3918: copy_from oid 58 from oid 309 current snap is 0 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3919: write (excl) oid 322 current snap is 0 2024-09-21T18:10:26.151 INFO:tasks.rados.rados.0.smithi160.stdout:3919: seq_num 1609 ranges {642463=426124,1604939=497883,2685713=356750} 2024-09-21T18:10:26.159 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:10:26.165 INFO:tasks.rados.rados.0.smithi160.stdout:3919: writing smithi16030472-322 from 642463 to 1068587 tid 1 2024-09-21T18:10:26.172 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:26.169+0000 7f728550b640 -1 osd.14 127 handle_osd_map injecting map crc failure 2024-09-21T18:10:26.176 INFO:tasks.rados.rados.0.smithi160.stdout:3919: writing smithi16030472-322 from 1604939 to 2102822 tid 2 2024-09-21T18:10:26.185 INFO:tasks.rados.rados.0.smithi160.stdout:3919: writing smithi16030472-322 from 2685713 to 3042463 tid 3 2024-09-21T18:10:26.185 INFO:tasks.rados.rados.0.smithi160.stdout:3920: delete oid 132 current snap is 0 2024-09-21T18:10:26.185 INFO:tasks.rados.rados.0.smithi160.stdout:3917: got (0) Success 2024-09-21T18:10:26.185 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 255 v 0 (ObjNum 81 snap 0 seq_num 0) clean dne 2024-09-21T18:10:26.226 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:26.225+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:26.321 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.321 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.327 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:26.325+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:26.335 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:10:26.337 INFO:tasks.rados.rados.0.smithi160.stdout:3915: finishing write tid 1 to smithi16030472-25 2024-09-21T18:10:26.348 INFO:tasks.rados.rados.0.smithi160.stdout:3915: finishing write tid 2 to smithi16030472-25 2024-09-21T18:10:26.348 INFO:tasks.rados.rados.0.smithi160.stdout:3915: finishing write tid 3 to smithi16030472-25 2024-09-21T18:10:26.428 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:26.423+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:26.445 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:26.439+0000 7f4976023640 -1 osd.5 126 handle_osd_map injecting map crc failure 2024-09-21T18:10:26.447 INFO:tasks.rados.rados.0.smithi160.stdout:3918: finishing copy_from to smithi16030472-58 2024-09-21T18:10:26.448 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 58 v 1019 (ObjNum 1440 snap 0 seq_num 1440) dirty exists 2024-09-21T18:10:26.456 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:26.451+0000 7f440780b640 -1 osd.13 127 handle_osd_map injecting map crc failure 2024-09-21T18:10:26.466 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.466 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.481 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:10:26.528 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:26.523+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:3915: finishing write tid 4 to smithi16030472-25 2024-09-21T18:10:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:3915: oid 25 updating version 0 to 1097 2024-09-21T18:10:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:3915: oid 25 version 1097 is already newer than 1095 2024-09-21T18:10:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:3915: oid 25 version 1097 is already newer than 1096 2024-09-21T18:10:26.544 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 25 v 1097 (ObjNum 1608 snap 0 seq_num 1608) dirty exists 2024-09-21T18:10:26.545 INFO:tasks.rados.rados.0.smithi160.stdout:3915: left oid 25 (ObjNum 1608 snap 0 seq_num 1608) 2024-09-21T18:10:26.545 INFO:tasks.rados.rados.0.smithi160.stdout:3918: finishing copy_from racing read to smithi16030472-58 2024-09-21T18:10:26.555 INFO:tasks.rados.rados.0.smithi160.stdout:3919: finishing write tid 1 to smithi16030472-322 2024-09-21T18:10:26.555 INFO:tasks.rados.rados.0.smithi160.stdout:3919: finishing write tid 2 to smithi16030472-322 2024-09-21T18:10:26.555 INFO:tasks.rados.rados.0.smithi160.stdout:3919: finishing write tid 3 to smithi16030472-322 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: finishing write tid 4 to smithi16030472-322 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: finishing write tid 5 to smithi16030472-322 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: oid 322 updating version 0 to 1056 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: oid 322 updating version 1056 to 1057 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: oid 322 version 1057 is already newer than 1055 2024-09-21T18:10:26.556 INFO:tasks.rados.rados.0.smithi160.stdout:3919: oid 322 version 1057 is already newer than 1054 2024-09-21T18:10:26.557 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 322 v 1057 (ObjNum 1609 snap 0 seq_num 1609) dirty exists 2024-09-21T18:10:26.557 INFO:tasks.rados.rados.0.smithi160.stdout:3919: left oid 322 (ObjNum 1609 snap 0 seq_num 1609) 2024-09-21T18:10:26.629 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:26.629+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:26.650 INFO:tasks.rados.rados.0.smithi160.stdout:3915: done (5 left) 2024-09-21T18:10:26.651 INFO:tasks.rados.rados.0.smithi160.stdout:3917: done (4 left) 2024-09-21T18:10:26.651 INFO:tasks.rados.rados.0.smithi160.stdout:3918: done (3 left) 2024-09-21T18:10:26.651 INFO:tasks.rados.rados.0.smithi160.stdout:3919: done (2 left) 2024-09-21T18:10:26.651 INFO:tasks.rados.rados.0.smithi160.stdout:3920: done (1 left) 2024-09-21T18:10:26.651 INFO:tasks.rados.rados.0.smithi160.stdout:3921: delete oid 221 current snap is 0 2024-09-21T18:10:26.652 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.652 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.654 INFO:tasks.rados.rados.0.smithi160.stdout:3921: done (1 left) 2024-09-21T18:10:26.654 INFO:tasks.rados.rados.0.smithi160.stdout:3922: delete oid 29 current snap is 0 2024-09-21T18:10:26.667 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:10:26.730 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:26.729+0000 7f728eae0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 14 (PID: 26347) UID: 0 2024-09-21T18:10:26.757 INFO:tasks.rados.rados.0.smithi160.stdout:3916: got (0) Success 2024-09-21T18:10:26.757 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 0 (ObjNum 1496 snap 0 seq_num 1496) dirty exists 2024-09-21T18:10:26.792 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.792 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.807 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:10:26.831 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:26.829+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:26.877 INFO:tasks.rados.rados.0.smithi160.stdout:3916: done (1 left) 2024-09-21T18:10:26.877 INFO:tasks.rados.rados.0.smithi160.stdout:3922: done (0 left) 2024-09-21T18:10:26.877 INFO:tasks.rados.rados.0.smithi160.stdout:3923: delete oid 428 current snap is 0 2024-09-21T18:10:26.932 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:26.929+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:26.934 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:26.934 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:26.949 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:10:26.968 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:26.964+0000 7f94d9542640 -1 osd.15 126 handle_osd_map injecting map crc failure 2024-09-21T18:10:27.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T18:10:27.033 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:27.027+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:27.077 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:27.077 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:27.082 INFO:tasks.rados.rados.0.smithi160.stdout:3923: done (0 left) 2024-09-21T18:10:27.082 INFO:tasks.rados.rados.0.smithi160.stdout:3924: read oid 205 snap -1 2024-09-21T18:10:27.082 INFO:tasks.rados.rados.0.smithi160.stdout:3924: expect (ObjNum 204 snap 0 seq_num 204) 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3925: read oid 307 snap -1 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3925: expect (ObjNum 1502 snap 0 seq_num 1502) 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3926: cache_flush oid 281 snap -1 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3927: cache_evict oid 173 snap -1 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3928: write oid 282 current snap is 0 2024-09-21T18:10:27.083 INFO:tasks.rados.rados.0.smithi160.stdout:3928: seq_num 1614 ranges {726498=518722,1897349=765232} 2024-09-21T18:10:27.092 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:10:27.102 INFO:tasks.rados.rados.0.smithi160.stdout:3928: writing smithi16030472-282 from 726498 to 1245220 tid 1 2024-09-21T18:10:27.125 INFO:tasks.rados.rados.0.smithi160.stdout:3928: writing smithi16030472-282 from 1897349 to 2662581 tid 2 2024-09-21T18:10:27.126 INFO:tasks.rados.rados.0.smithi160.stdout:3929: copy_from oid 266 from oid 301 current snap is 0 2024-09-21T18:10:27.126 INFO:tasks.rados.rados.0.smithi160.stdout:3930: write (excl) oid 291 current snap is 0 2024-09-21T18:10:27.126 INFO:tasks.rados.rados.0.smithi160.stdout:3930: seq_num 1615 ranges {743305=593971,1955831=508890} 2024-09-21T18:10:27.135 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:27.133+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:27.143 INFO:tasks.rados.rados.0.smithi160.stdout:3930: writing smithi16030472-291 from 743305 to 1337276 tid 1 2024-09-21T18:10:27.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T18:10:27.156 INFO:tasks.rados.rados.0.smithi160.stdout:3930: writing smithi16030472-291 from 1955831 to 2464721 tid 2 2024-09-21T18:10:27.156 INFO:tasks.rados.rados.0.smithi160.stdout:3931: cache_try_flush oid 459 snap -1 2024-09-21T18:10:27.156 INFO:tasks.rados.rados.0.smithi160.stdout:3932: cache_evict oid 462 snap -1 2024-09-21T18:10:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:3933: cache_flush oid 115 snap -1 2024-09-21T18:10:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:3934: cache_evict oid 278 snap -1 2024-09-21T18:10:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:3935: cache_flush oid 479 snap -1 2024-09-21T18:10:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:3936: write oid 251 current snap is 0 2024-09-21T18:10:27.157 INFO:tasks.rados.rados.0.smithi160.stdout:3936: seq_num 1616 ranges {760112=669220,2014313=768901} 2024-09-21T18:10:27.168 INFO:tasks.rados.rados.0.smithi160.stdout:3936: writing smithi16030472-251 from 760112 to 1429332 tid 1 2024-09-21T18:10:27.175 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:27.175 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout:3936: writing smithi16030472-251 from 2014313 to 2783214 tid 2 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout:3937: read oid 30 snap -1 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout:3937: expect (ObjNum 1262 snap 0 seq_num 1262) 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout:3938: cache_try_flush oid 317 snap -1 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout:3939: cache_flush oid 486 snap -1 2024-09-21T18:10:27.179 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:27.195 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:27.217 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:27.218 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:27.236 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:27.232+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:27.238 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:27.239 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:27.263 INFO:tasks.rados.rados.0.smithi160.stdout:3930: finishing write tid 1 to smithi16030472-291 2024-09-21T18:10:27.264 INFO:tasks.rados.rados.0.smithi160.stdout:3930: finishing write tid 2 to smithi16030472-291 2024-09-21T18:10:27.267 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:27.267 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:27.282 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:10:27.290 INFO:tasks.rados.rados.0.smithi160.stdout:3924: expect (ObjNum 204 snap 0 seq_num 204) 2024-09-21T18:10:27.335 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:27.331+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:27.344 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:27.373 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:27.407 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:27.410 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:27.412 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:27.413 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:27.413 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:27.428 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:10:27.437 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:27.433+0000 7ff418f76640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 26362) UID: 0 2024-09-21T18:10:27.442 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:27.443 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:27.443 INFO:tasks.rados.rados.0.smithi160.stdout:3925: expect (ObjNum 1502 snap 0 seq_num 1502) 2024-09-21T18:10:27.444 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:27.477 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:27.477 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:27.497 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:27.498 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:27.503 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1555651 2024-09-21T18:10:27.516 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3926: got (0) Success 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 281 v 0 (ObjNum 974 snap 0 seq_num 974) dirty exists 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: finishing write tid 1 to smithi16030472-282 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: finishing write tid 2 to smithi16030472-282 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: finishing write tid 3 to smithi16030472-282 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: finishing write tid 4 to smithi16030472-282 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: oid 282 updating version 0 to 1133 2024-09-21T18:10:27.530 INFO:tasks.rados.rados.0.smithi160.stdout:3928: oid 282 updating version 1133 to 1135 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3928: oid 282 version 1135 is already newer than 1134 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 282 v 1135 (ObjNum 1614 snap 0 seq_num 1614) dirty exists 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3928: left oid 282 (ObjNum 1614 snap 0 seq_num 1614) 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3931: got (0) Success 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 459 v 0 (ObjNum 1304 snap 0 seq_num 1304) dirty exists 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3934: got (16) Device or resource busy 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3929: finishing copy_from racing read to smithi16030472-266 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3932: got (0) Success 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3929: finishing copy_from to smithi16030472-266 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 266 v 1022 (ObjNum 1498 snap 0 seq_num 1498) dirty exists 2024-09-21T18:10:27.531 INFO:tasks.rados.rados.0.smithi160.stdout:3935: got (0) Success 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 479 v 0 (ObjNum 129 snap 0 seq_num 3147964240) clean dne 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3924: done (15 left) 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3925: done (14 left) 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3926: done (13 left) 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3928: done (12 left) 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3929: done (11 left) 2024-09-21T18:10:27.532 INFO:tasks.rados.rados.0.smithi160.stdout:3931: done (10 left) 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3932: done (9 left) 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3934: done (8 left) 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3935: done (7 left) 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3940: cache_flush oid 34 snap -1 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: finishing write tid 3 to smithi16030472-291 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: finishing write tid 4 to smithi16030472-291 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: oid 291 updating version 0 to 1155 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: oid 291 version 1155 is already newer than 1153 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: oid 291 version 1155 is already newer than 1154 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 291 v 1155 (ObjNum 1615 snap 0 seq_num 1615) dirty exists 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3930: left oid 291 (ObjNum 1615 snap 0 seq_num 1615) 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3933: got (0) Success 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 115 v 0 (ObjNum 129 snap 0 seq_num 3126289544) clean dne 2024-09-21T18:10:27.533 INFO:tasks.rados.rados.0.smithi160.stdout:3937: expect (ObjNum 1262 snap 0 seq_num 1262) 2024-09-21T18:10:27.534 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:27.534 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:27.537 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:27.531+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:27.552 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:27.552 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:27.574 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:27.574 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:27.596 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:27.596 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:27.597 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:27.597 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [], 2024-09-21T18:10:27.597 INFO:teuthology.orchestra.run.smithi040.stdout: "complaint_time": 30, 2024-09-21T18:10:27.597 INFO:teuthology.orchestra.run.smithi040.stdout: "num_blocked_ops": 0 2024-09-21T18:10:27.597 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:27.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:27.603 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 3109788 2024-09-21T18:10:27.612 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:3939: got (0) Success 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 486 v 0 (ObjNum 485 snap 0 seq_num 485) dirty exists 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:3927: got (0) Success 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:3936: finishing write tid 1 to smithi16030472-251 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:3936: finishing write tid 2 to smithi16030472-251 2024-09-21T18:10:27.643 INFO:tasks.rados.rados.0.smithi160.stdout:3936: finishing write tid 3 to smithi16030472-251 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: finishing write tid 4 to smithi16030472-251 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: oid 251 updating version 0 to 1060 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: oid 251 version 1060 is already newer than 1059 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: oid 251 version 1060 is already newer than 1058 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 251 v 1060 (ObjNum 1616 snap 0 seq_num 1616) dirty exists 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: left oid 251 (ObjNum 1616 snap 0 seq_num 1616) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3938: got (0) Success 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 317 v 0 (ObjNum 1573 snap 0 seq_num 1573) dirty exists 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3927: done (7 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3930: done (6 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3933: done (5 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3936: done (4 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3937: done (3 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3938: done (2 left) 2024-09-21T18:10:27.644 INFO:tasks.rados.rados.0.smithi160.stdout:3939: done (1 left) 2024-09-21T18:10:27.645 INFO:tasks.rados.rados.0.smithi160.stdout:3941: delete oid 399 current snap is 0 2024-09-21T18:10:27.656 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:27.651+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:27.730 INFO:tasks.rados.rados.0.smithi160.stdout:3940: got (0) Success 2024-09-21T18:10:27.730 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 34 v 0 (ObjNum 129 snap 0 seq_num 3138374736) clean dne 2024-09-21T18:10:27.733 INFO:tasks.rados.rados.0.smithi160.stdout:3940: done (1 left) 2024-09-21T18:10:27.734 INFO:tasks.rados.rados.0.smithi160.stdout:3941: done (0 left) 2024-09-21T18:10:27.734 INFO:tasks.rados.rados.0.smithi160.stdout:3942: write (excl) oid 361 current snap is 0 2024-09-21T18:10:27.734 INFO:tasks.rados.rados.0.smithi160.stdout:3942: seq_num 1618 ranges {793726=419718,1614924=772570,2932520=261206} 2024-09-21T18:10:27.739 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:27.736+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:27.749 INFO:tasks.rados.rados.0.smithi160.stdout:3942: writing smithi16030472-361 from 793726 to 1213444 tid 1 2024-09-21T18:10:27.763 INFO:tasks.rados.rados.0.smithi160.stdout:3942: writing smithi16030472-361 from 1614924 to 2387494 tid 2 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3942: writing smithi16030472-361 from 2932520 to 3193726 tid 3 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3943: read oid 253 snap -1 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3943: expect deleted 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3944: cache_evict oid 112 snap -1 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3942: finishing write tid 1 to smithi16030472-361 2024-09-21T18:10:27.774 INFO:tasks.rados.rados.0.smithi160.stdout:3945: delete oid 18 current snap is 0 2024-09-21T18:10:27.783 INFO:teuthology.orchestra.run.smithi040.stdout:{ 2024-09-21T18:10:27.783 INFO:teuthology.orchestra.run.smithi040.stdout: "size": 20, 2024-09-21T18:10:27.783 INFO:teuthology.orchestra.run.smithi040.stdout: "duration": 600, 2024-09-21T18:10:27.783 INFO:teuthology.orchestra.run.smithi040.stdout: "ops": [] 2024-09-21T18:10:27.783 INFO:teuthology.orchestra.run.smithi040.stdout:} 2024-09-21T18:10:27.798 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:10:27.840 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:27.837+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:3942: finishing write tid 2 to smithi16030472-361 2024-09-21T18:10:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:3942: finishing write tid 3 to smithi16030472-361 2024-09-21T18:10:27.875 INFO:tasks.rados.rados.0.smithi160.stdout:3944: got (16) Device or resource busy 2024-09-21T18:10:27.878 INFO:tasks.rados.rados.0.smithi160.stdout:3944: done (3 left) 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3945: done (2 left) 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3946: cache_try_flush oid 270 snap -1 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3947: copy_from oid 191 from oid 259 current snap is 0 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3948: read oid 211 snap -1 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3948: expect (ObjNum 1278 snap 0 seq_num 1278) 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3949: cache_flush oid 91 snap -1 2024-09-21T18:10:27.879 INFO:tasks.rados.rados.0.smithi160.stdout:3950: write oid 433 current snap is 0 2024-09-21T18:10:27.880 INFO:tasks.rados.rados.0.smithi160.stdout:3950: seq_num 1620 ranges {427340=686569,1848241=776239,3103719=123621} 2024-09-21T18:10:27.892 INFO:tasks.rados.rados.0.smithi160.stdout:3950: writing smithi16030472-433 from 427340 to 1113909 tid 1 2024-09-21T18:10:27.911 INFO:tasks.rados.rados.0.smithi160.stdout:3950: writing smithi16030472-433 from 1848241 to 2624480 tid 2 2024-09-21T18:10:27.918 INFO:tasks.rados.rados.0.smithi160.stdout:3950: writing smithi16030472-433 from 3103719 to 3227340 tid 3 2024-09-21T18:10:27.918 INFO:tasks.rados.rados.0.smithi160.stdout:3951: write (excl) oid 250 current snap is 0 2024-09-21T18:10:27.918 INFO:tasks.rados.rados.0.smithi160.stdout:3951: seq_num 1621 ranges {444147=761818,1790370=519897,2898436=345711} 2024-09-21T18:10:27.931 INFO:tasks.rados.rados.0.smithi160.stdout:3951: writing smithi16030472-250 from 444147 to 1205965 tid 1 2024-09-21T18:10:27.934 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:27.934 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:10239 4.1 e128/33)", 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:27.782600+0000", 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.151529567, 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.15156439399999999, 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.782600+0000", 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:27.935 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.782600+0000", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.782602+0000", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.717e-06 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.782614+0000", 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1800000000000001e-05 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:27.936 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.782641+0000", 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7276000000000001e-05 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:27.882755+0000", 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100113463 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:10:27.937 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 1 2024-09-21T18:10:27.938 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:27.940 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:27.937+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:27.942 INFO:tasks.rados.rados.0.smithi160.stdout:3951: writing smithi16030472-250 from 1790370 to 2310267 tid 2 2024-09-21T18:10:27.950 INFO:tasks.rados.rados.0.smithi160.stdout:3951: writing smithi16030472-250 from 2898436 to 3244147 tid 3 2024-09-21T18:10:27.950 INFO:tasks.rados.rados.0.smithi160.stdout:3952: copy_from oid 322 from oid 92 current snap is 0 2024-09-21T18:10:27.950 INFO:tasks.rados.rados.0.smithi160.stdout:3951: finishing write tid 1 to smithi16030472-250 2024-09-21T18:10:27.951 INFO:tasks.rados.rados.0.smithi160.stdout:3951: finishing write tid 2 to smithi16030472-250 2024-09-21T18:10:27.951 INFO:tasks.rados.rados.0.smithi160.stdout:3953: copy_from oid 183 from oid 434 current snap is 0 2024-09-21T18:10:27.951 INFO:tasks.rados.rados.0.smithi160.stdout:3954: delete oid 216 current snap is 0 2024-09-21T18:10:27.952 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:10:28.040 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:28.037+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:28.058 INFO:tasks.rados.rados.0.smithi160.stdout:3951: finishing write tid 3 to smithi16030472-250 2024-09-21T18:10:28.058 INFO:tasks.rados.rados.0.smithi160.stdout:3951: finishing write tid 4 to smithi16030472-250 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: finishing write tid 5 to smithi16030472-250 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: oid 250 updating version 0 to 1160 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: oid 250 version 1160 is already newer than 1159 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: oid 250 version 1160 is already newer than 1157 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: oid 250 version 1160 is already newer than 1158 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 250 v 1160 (ObjNum 1621 snap 0 seq_num 1621) dirty exists 2024-09-21T18:10:28.059 INFO:tasks.rados.rados.0.smithi160.stdout:3951: left oid 250 (ObjNum 1621 snap 0 seq_num 1621) 2024-09-21T18:10:28.071 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:28.071 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:28.071 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:28.071 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:28.071 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:28.085 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:10:28.098 INFO:tasks.rados.rados.0.smithi160.stdout:3953: finishing copy_from racing read to smithi16030472-183 2024-09-21T18:10:28.098 INFO:tasks.rados.rados.0.smithi160.stdout:3953: finishing copy_from to smithi16030472-183 2024-09-21T18:10:28.099 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 183 v 274 (ObjNum 395 snap 0 seq_num 395) dirty exists 2024-09-21T18:10:28.177 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:28.177+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:28.180 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-09-21T18:10:28.242 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:28.235+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.262 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:524 3.3 e46/31 3:d9b5818f:::smithi16030472-117:head v 46'74, mlcod=46'64)", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 189.92853909799999, 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.302436072, 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.263 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330919+0000", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.330922+0000", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7839999999999999e-06 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331458+0000", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053620299999999996 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331484+0000", 2024-09-21T18:10:28.264 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6412999999999999e-05 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.331504+0000", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9905999999999999e-05 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530412+0000", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19890828799999999 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.530432+0000", 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9517999999999998e-05 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.265 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633309+0000", 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102877544 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.633355+0000", 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5416e-05 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.266 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:945 3.3 e48/31 3:e9a97588:::smithi16030472-213:head v 48'147, mlcod=48'143)", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.08868710199999, 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30864236900000003, 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.267 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170771+0000", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.170774+0000", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5470000000000001e-06 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171707+0000", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00093267600000000001 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.171735+0000", 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8399e-05 2024-09-21T18:10:28.268 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.271880+0000", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100144832 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372227+0000", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10034705100000001 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.372262+0000", 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.269 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479288+0000", 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107026418 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.479413+0000", 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012495 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.270 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:956 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'152, mlcod=48'145)", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.984966654, 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31671622799999999, 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.271 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274491+0000", 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.274493+0000", 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3060000000000001e-06 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275216+0000", 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072340100000000004 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.272 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275232+0000", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6002999999999999e-05 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275245+0000", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3263e-05 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481384+0000", 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206138452 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.273 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.481415+0000", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1316999999999998e-05 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591115+0000", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109699901 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591207+0000", 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.2584999999999995e-05 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.274 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:957 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'153, mlcod=48'145)", 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.98419191100001, 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31602933900000002, 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.275 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275266+0000", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275267+0000", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.407e-06 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275866+0000", 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059845999999999999 2024-09-21T18:10:28.276 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275882+0000", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6146000000000001e-05 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275896+0000", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.365e-05 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483827+0000", 2024-09-21T18:10:28.277 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207931006 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.483884+0000", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7538999999999999e-05 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591218+0000", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107334317 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591295+0000", 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6814e-05 2024-09-21T18:10:28.278 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:958 3.3 e48/31 3:e05635ff:::smithi16030472-215:head v 48'154, mlcod=48'145)", 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.983543892, 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31787257000000002, 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.279 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275914+0000", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275915+0000", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.313e-06 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.280 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275926+0000", 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0363000000000001e-05 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275943+0000", 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7560000000000001e-05 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.275956+0000", 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2437e-05 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.281 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485564+0000", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20960843300000001 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.485631+0000", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7265000000000001e-05 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593681+0000", 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.108049627 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.282 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593787+0000", 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105572 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:960 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'155, mlcod=48'145)", 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.96967219699999, 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304077243, 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.283 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289786+0000", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.289789+0000", 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.937e-06 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.284 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290628+0000", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00083910400000000002 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290657+0000", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9073000000000001e-05 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.290677+0000", 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0163e-05 2024-09-21T18:10:28.285 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486353+0000", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19567597 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.486383+0000", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9972999999999999e-05 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593795+0000", 2024-09-21T18:10:28.286 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10741223799999999 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593863+0000", 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7785000000000002e-05 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:961 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'156, mlcod=48'145)", 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.966731181, 2024-09-21T18:10:28.287 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40152941399999997, 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292727+0000", 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.288 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.292729+0000", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.807e-06 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293644+0000", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091491299999999995 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293673+0000", 2024-09-21T18:10:28.289 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8439e-05 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.293692+0000", 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8783e-05 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588412+0000", 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29472084999999998 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.290 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.588470+0000", 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.7389000000000002e-05 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694165+0000", 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105695335 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694256+0000", 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0897999999999996e-05 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.291 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:962 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'157, mlcod=48'145)", 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.96371433300001, 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40034693900000001, 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.292 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295744+0000", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.295746+0000", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8930000000000001e-06 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296391+0000", 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00064433700000000003 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.293 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296419+0000", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8282000000000001e-05 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.296438+0000", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.895e-05 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591402+0000", 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29496407699999999 2024-09-21T18:10:28.294 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.591435+0000", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3343000000000002e-05 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696027+0000", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104591422 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696090+0000", 2024-09-21T18:10:28.295 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3634999999999996e-05 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:963 3.3 e48/31 3:e73a30a0:::smithi16030472-216:head v 48'158, mlcod=48'145)", 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.96150275100001, 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.39819001300000001, 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.296 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297955+0000", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297958+0000", 2024-09-21T18:10:28.297 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.044e-06 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297970+0000", 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1868999999999999e-05 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.297997+0000", 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7033e-05 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.298 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.298016+0000", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8963000000000001e-05 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593175+0000", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29515935300000001 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.593229+0000", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3134999999999999e-05 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.299 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696097+0000", 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10286834 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.696145+0000", 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8276000000000003e-05 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:976 3.3 e48/31 3:f8256ac6:::smithi16030472-219:head v 48'160, mlcod=48'145)", 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.856165659, 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.393583715, 2024-09-21T18:10:28.300 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403292+0000", 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.301 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403295+0000", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1439999999999999e-06 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403924+0000", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062865200000000005 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403949+0000", 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4870000000000001e-05 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.302 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.403963+0000", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4316e-05 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694309+0000", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29034569199999999 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.694342+0000", 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2783000000000001e-05 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.303 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796806+0000", 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102463886 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:24.796876+0000", 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0371999999999996e-05 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1042 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'164, mlcod=49'161)", 2024-09-21T18:10:28.304 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.16474328499999, 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.306384885, 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.305 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094715+0000", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.094717+0000", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.903e-06 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095253+0000", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00053591900000000004 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.306 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095281+0000", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7336000000000001e-05 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.095295+0000", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4167e-05 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295664+0000", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.200368764 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.307 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.295727+0000", 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3301999999999998e-05 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401030+0000", 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105303512 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401099+0000", 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8981999999999998e-05 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.308 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1043 3.3 e49/31 3:cafbb15d:::smithi16030472-232:head v 49'165, mlcod=49'161)", 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.16171328499999, 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30339806400000002, 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097745+0000", 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.097747+0000", 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5639999999999999e-06 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098434+0000", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000686876 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.098454+0000", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0188999999999999e-05 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.198604+0000", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100150161 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298394+0000", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.099789972000000005 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.298428+0000", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4029999999999998e-05 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401108+0000", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10267945000000001 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.401143+0000", 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4822e-05 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1436 3.3 e50/31 3:eea6f21f:::smithi16030472-322:head v 50'225, mlcod=50'212)", 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 178.19514932300001, 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30726658400000001, 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064309+0000", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064317+0000", 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6889999999999993e-06 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064332+0000", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.437e-05 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.064359+0000", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7600999999999999e-05 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.164502+0000", 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142589 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264801+0000", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100299031 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.264836+0000", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4992e-05 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371418+0000", 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.106581999 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:30.371575+0000", 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015731299999999999 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1597 3.3 e51/31 3:ec0abe08:::smithi16030472-354:head v 51'244, mlcod=51'241)", 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 176.52831388800001, 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30046494800000001, 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.317 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731144+0000", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731152+0000", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8560000000000007e-06 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731832+0000", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068020600000000004 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731855+0000", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2806000000000002e-05 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.731869+0000", 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3902e-05 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926653+0000", 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19478403499999999 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.926699+0000", 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6044999999999997e-05 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031519+0000", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104819885 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.031609+0000", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.0212999999999998e-05 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2127 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'350, mlcod=53'348)", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:28.321 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.22482280700001, 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.309446165, 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034635+0000", 2024-09-21T18:10:28.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.034640+0000", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8470000000000003e-06 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035295+0000", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00065476900000000003 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035326+0000", 2024-09-21T18:10:28.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0923000000000002e-05 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035342+0000", 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6872e-05 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035499+0000", 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015631799999999999 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.035536+0000", 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7662e-05 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.343950+0000", 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30841353599999999 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344081+0000", 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013123799999999999 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2128 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'351, mlcod=53'348)", 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.22252577399999, 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30719240399999997, 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036932+0000", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.036936+0000", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.653e-06 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037600+0000", 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00066414299999999996 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037627+0000", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7594999999999999e-05 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.037642+0000", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.451e-05 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039742+0000", 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.002100059 2024-09-21T18:10:28.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039780+0000", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8269000000000002e-05 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344090+0000", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.304309783 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344124+0000", 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4391999999999999e-05 2024-09-21T18:10:28.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2129 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'352, mlcod=53'348)", 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.22047532600001, 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30517705099999998, 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038983+0000", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.038985+0000", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8339999999999999e-06 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039552+0000", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056650899999999998 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039579+0000", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7348999999999999e-05 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.039593+0000", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3635000000000001e-05 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042170+0000", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025773359999999999 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042204+0000", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3631999999999999e-05 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344129+0000", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.301925571 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344160+0000", 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0185e-05 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2130 3.3 e53/31 3:edf435bb:::smithi16030472-472:head v 53'353, mlcod=53'348)", 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.21831996500001, 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30306076900000001, 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041138+0000", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041141+0000", 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9699999999999999e-06 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041159+0000", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8233e-05 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041181+0000", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1591999999999999e-05 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041195+0000", 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3923e-05 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043761+0000", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0025660549999999998 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.043794+0000", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3312999999999998e-05 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344164+0000", 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30037047700000002 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.344199+0000", 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4205999999999997e-05 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2142 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'354, mlcod=53'348)", 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.217557207, 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.40314361500000001, 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.338 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041901+0000", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.041904+0000", 2024-09-21T18:10:28.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8569999999999999e-06 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042335+0000", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00043136100000000001 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042357+0000", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2243e-05 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.042375+0000", 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7411999999999999e-05 2024-09-21T18:10:28.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237918+0000", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.1955432 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.237955+0000", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7008999999999998e-05 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.444976+0000", 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207021701 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445044+0000", 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7831999999999997e-05 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2143 3.3 e53/31 3:e2ee5507:::smithi16030472-475:head v 53'355, mlcod=53'348)", 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 169.115652162, 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30131781000000002, 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:28.342 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143806+0000", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:28.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.143810+0000", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9770000000000002e-06 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144559+0000", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00074911200000000002 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144590+0000", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1389000000000001e-05 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:28.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.144606+0000", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5432999999999999e-05 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239875+0000", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.095269621999999998 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.239914+0000", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8379e-05 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:28.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445051+0000", 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.205137501 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.445124+0000", 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2397000000000005e-05 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:28.346 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:28.347 INFO:tasks.rados.rados.0.smithi160.stdout:3948: expect (ObjNum 1278 snap 0 seq_num 1278) 2024-09-21T18:10:28.348 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:10:28.348 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:28.301+0000 7fca4e93e640 -1 osd.2 128 handle_osd_map injecting map crc failure 2024-09-21T18:10:28.349 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:28.345+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:28.442 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 938025 2024-09-21T18:10:28.448 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:28.445+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:28.485 INFO:tasks.rados.rados.0.smithi160.stdout:3942: finishing write tid 4 to smithi16030472-361 2024-09-21T18:10:28.485 INFO:tasks.rados.rados.0.smithi160.stdout:3942: finishing write tid 5 to smithi16030472-361 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3942: oid 361 updating version 0 to 1102 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3942: oid 361 updating version 1102 to 1103 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3942: oid 361 updating version 1103 to 1105 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3942: oid 361 version 1105 is already newer than 1104 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 361 v 1105 (ObjNum 1618 snap 0 seq_num 1618) dirty exists 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3942: left oid 361 (ObjNum 1618 snap 0 seq_num 1618) 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: finishing write tid 1 to smithi16030472-433 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3952: finishing copy_from racing read to smithi16030472-322 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: finishing write tid 2 to smithi16030472-433 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: finishing write tid 3 to smithi16030472-433 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: finishing write tid 4 to smithi16030472-433 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: finishing write tid 5 to smithi16030472-433 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: oid 433 updating version 0 to 1107 2024-09-21T18:10:28.486 INFO:tasks.rados.rados.0.smithi160.stdout:3950: oid 433 updating version 1107 to 1109 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3950: oid 433 version 1109 is already newer than 1108 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3950: oid 433 version 1109 is already newer than 1106 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 433 v 1109 (ObjNum 1620 snap 0 seq_num 1620) dirty exists 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3950: left oid 433 (ObjNum 1620 snap 0 seq_num 1620) 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3947: finishing copy_from to smithi16030472-191 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3947: got expected ENOENT (src dne) 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3947: finishing copy_from racing read to smithi16030472-191 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3949: got (0) Success 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 91 v 0 (ObjNum 1491 snap 0 seq_num 1491) dirty exists 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3952: finishing copy_from to smithi16030472-322 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 322 v 541 (ObjNum 718 snap 0 seq_num 718) dirty exists 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3942: done (10 left) 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3943: done (9 left) 2024-09-21T18:10:28.487 INFO:tasks.rados.rados.0.smithi160.stdout:3947: done (8 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3948: done (7 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3949: done (6 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3950: done (5 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3951: done (4 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3952: done (3 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3953: done (2 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3954: done (1 left) 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3955: write (excl) oid 367 current snap is 0 2024-09-21T18:10:28.488 INFO:tasks.rados.rados.0.smithi160.stdout:3955: seq_num 1623 ranges {477761=512316,1507334=523566,2553282=599771} 2024-09-21T18:10:28.498 INFO:tasks.rados.rados.0.smithi160.stdout:3955: writing smithi16030472-367 from 477761 to 990077 tid 1 2024-09-21T18:10:28.512 INFO:tasks.rados.rados.0.smithi160.stdout:3955: writing smithi16030472-367 from 1507334 to 2030900 tid 2 2024-09-21T18:10:28.516 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:28.516 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:28.516 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:10:28.516 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:28.527 INFO:tasks.rados.rados.0.smithi160.stdout:3955: writing smithi16030472-367 from 2553282 to 3153053 tid 3 2024-09-21T18:10:28.527 INFO:tasks.rados.rados.0.smithi160.stdout:3956: write oid 158 current snap is 0 2024-09-21T18:10:28.527 INFO:tasks.rados.rados.0.smithi160.stdout:3956: seq_num 1624 ranges {494568=587565,1565816=667224,2980705=313863} 2024-09-21T18:10:28.530 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:10:28.540 INFO:tasks.rados.rados.0.smithi160.stdout:3956: writing smithi16030472-158 from 494568 to 1082133 tid 1 2024-09-21T18:10:28.551 INFO:tasks.rados.rados.0.smithi160.stdout:3956: writing smithi16030472-158 from 1565816 to 2233040 tid 2 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3956: writing smithi16030472-158 from 2980705 to 3294568 tid 3 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3957: cache_try_flush oid 272 snap -1 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3958: copy_from oid 408 from oid 405 current snap is 0 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3959: cache_evict oid 47 snap -1 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3960: cache_evict oid 87 snap -1 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3961: write oid 88 current snap is 0 2024-09-21T18:10:28.561 INFO:tasks.rados.rados.0.smithi160.stdout:3961: seq_num 1625 ranges {511375=662814,1907945=527235,2891775=419600} 2024-09-21T18:10:28.569 INFO:tasks.rados.rados.0.smithi160.stdout:3961: writing smithi16030472-88 from 511375 to 1174189 tid 1 2024-09-21T18:10:28.574 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:28.573+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:28.578 INFO:tasks.rados.rados.0.smithi160.stdout:3961: writing smithi16030472-88 from 1907945 to 2435180 tid 2 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3961: writing smithi16030472-88 from 2891775 to 3311375 tid 3 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3962: cache_try_flush oid 11 snap -1 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3963: cache_try_flush oid 230 snap -1 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3964: cache_try_flush oid 336 snap -1 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3965: read oid 142 snap -1 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3965: expect (ObjNum 1395 snap 0 seq_num 1395) 2024-09-21T18:10:28.584 INFO:tasks.rados.rados.0.smithi160.stdout:3966: delete oid 255 current snap is 0 2024-09-21T18:10:28.649 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:10:28.645+0000 7fb93058f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 8 (PID: 26363) UID: 0 2024-09-21T18:10:28.669 INFO:tasks.rados.rados.0.smithi160.stdout:3956: finishing write tid 1 to smithi16030472-158 2024-09-21T18:10:28.670 INFO:tasks.rados.rados.0.smithi160.stdout:3956: finishing write tid 2 to smithi16030472-158 2024-09-21T18:10:28.672 INFO:tasks.rados.rados.0.smithi160.stdout:3956: finishing write tid 3 to smithi16030472-158 2024-09-21T18:10:28.672 INFO:tasks.rados.rados.0.smithi160.stdout:3960: got (16) Device or resource busy 2024-09-21T18:10:28.703 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:28.704 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:28.704 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:28.704 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:28.704 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:28.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:10:28.750 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:28.745+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:28.778 INFO:tasks.rados.rados.0.smithi160.stdout:3956: finishing write tid 4 to smithi16030472-158 2024-09-21T18:10:28.778 INFO:tasks.rados.rados.0.smithi160.stdout:3956: finishing write tid 5 to smithi16030472-158 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: oid 158 updating version 0 to 1143 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: oid 158 version 1143 is already newer than 1141 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: oid 158 updating version 1143 to 1144 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: oid 158 version 1144 is already newer than 1142 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 158 v 1144 (ObjNum 1624 snap 0 seq_num 1624) dirty exists 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: left oid 158 (ObjNum 1624 snap 0 seq_num 1624) 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3956: done (12 left) 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3960: done (11 left) 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3966: done (10 left) 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3967: cache_try_flush oid 485 snap -1 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3968: cache_flush oid 82 snap -1 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3969: write (excl) oid 210 current snap is 0 2024-09-21T18:10:28.779 INFO:tasks.rados.rados.0.smithi160.stdout:3969: seq_num 1627 ranges {544989=529665,1741262=530904,3062974=282015} 2024-09-21T18:10:28.784 INFO:tasks.rados.rados.0.smithi160.stdout:3969: writing smithi16030472-210 from 544989 to 1074654 tid 1 2024-09-21T18:10:28.791 INFO:tasks.rados.rados.0.smithi160.stdout:3969: writing smithi16030472-210 from 1741262 to 2272166 tid 2 2024-09-21T18:10:28.798 INFO:tasks.rados.rados.0.smithi160.stdout:3969: writing smithi16030472-210 from 3062974 to 3344989 tid 3 2024-09-21T18:10:28.798 INFO:tasks.rados.rados.0.smithi160.stdout:3970: cache_evict oid 254 snap -1 2024-09-21T18:10:28.798 INFO:tasks.rados.rados.0.smithi160.stdout:3971: write oid 240 current snap is 0 2024-09-21T18:10:28.798 INFO:tasks.rados.rados.0.smithi160.stdout:3971: seq_num 1628 ranges {561796=604914,1799744=674562,3090397=271399} 2024-09-21T18:10:28.805 INFO:tasks.rados.rados.0.smithi160.stdout:3971: writing smithi16030472-240 from 561796 to 1166710 tid 1 2024-09-21T18:10:28.813 INFO:tasks.rados.rados.0.smithi160.stdout:3971: writing smithi16030472-240 from 1799744 to 2474306 tid 2 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:3971: writing smithi16030472-240 from 3090397 to 3361796 tid 3 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:3972: read oid 250 snap -1 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:3972: expect (ObjNum 1621 snap 0 seq_num 1621) 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 16 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:3963: got (0) Success 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 0 (ObjNum 33 snap 0 seq_num 3147953352) clean dne 2024-09-21T18:10:28.819 INFO:tasks.rados.rados.0.smithi160.stdout:3965: expect (ObjNum 1395 snap 0 seq_num 1395) 2024-09-21T18:10:28.842 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:28.842 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:28.842 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:28.842 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [] 2024-09-21T18:10:28.842 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:28.851 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:28.849+0000 7f9bc4e70640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 10 (PID: 26345) UID: 0 2024-09-21T18:10:28.856 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:10:28.952 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:28.948+0000 7f6a48282640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 26586) UID: 0 2024-09-21T18:10:28.978 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2466400 2024-09-21T18:10:28.996 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:29.025 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:10299 4.2 e129/33 4:6513c01e:::smithi16030472-210:head v 129'1168, mlcod=129'1164)", 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:28.899335+0000", 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.126141848, 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.12618347399999999, 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.026 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.899335+0000", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.899335+0000", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.899336+0000", 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0130000000000001e-06 2024-09-21T18:10:29.027 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.899593+0000", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00025753100000000003 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.899601+0000", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9670000000000001e-06 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999686+0000", 2024-09-21T18:10:29.028 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100084225 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999811+0000", 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012553099999999999 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999866+0000", 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5226000000000003e-05 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.029 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:10300 4.2 e129/33 4:6513c01e:::smithi16030472-210:head v 129'1169, mlcod=129'1164)", 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:28.999739+0000", 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.025737869, 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025868687000000001, 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "started", 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999739+0000", 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.030 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999739+0000", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999741+0000", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.5799999999999999e-06 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999755+0000", 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3376e-05 2024-09-21T18:10:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999770+0000", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.5543e-05 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999780+0000", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9569999999999997e-06 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:29.002456+0000", 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0026758889999999999 2024-09-21T18:10:29.032 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:29.002512+0000", 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5828999999999997e-05 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:1887 4.2 e129/33)", 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:28.999800+0000", 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 0.025676926999999999, 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.025880996999999999, 2024-09-21T18:10:29.033 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "queued for pg", 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999800+0000", 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999800+0000", 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.034 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999801+0000", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.2699999999999999e-07 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999807+0000", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1299999999999998e-06 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999815+0000", 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.2490000000000001e-06 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.035 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:28.999821+0000", 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3600000000000001e-06 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: ], 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 3 2024-09-21T18:10:29.036 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:10:29.053 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:29.052+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3955: finishing write tid 1 to smithi16030472-367 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3957: got (0) Success 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 272 v 0 (ObjNum 568 snap 0 seq_num 568) dirty exists 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3968: got (0) Success 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 82 v 0 (ObjNum 964 snap 0 seq_num 964) dirty exists 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3970: got (16) Device or resource busy 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3958: finishing copy_from to smithi16030472-408 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 408 v 829 (ObjNum 1146 snap 0 seq_num 1146) dirty exists 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3958: finishing copy_from racing read to smithi16030472-408 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3961: finishing write tid 1 to smithi16030472-88 2024-09-21T18:10:29.063 INFO:tasks.rados.rados.0.smithi160.stdout:3955: finishing write tid 2 to smithi16030472-367 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3957: done (15 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3958: done (14 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3963: done (13 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3965: done (12 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3968: done (11 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3970: done (10 left) 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3973: cache_evict oid 127 snap -1 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: finishing write tid 3 to smithi16030472-367 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: finishing write tid 4 to smithi16030472-367 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: finishing write tid 5 to smithi16030472-367 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: oid 367 updating version 0 to 1164 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: oid 367 version 1164 is already newer than 1162 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: oid 367 version 1164 is already newer than 1163 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:3955: oid 367 version 1164 is already newer than 1161 2024-09-21T18:10:29.064 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 367 v 1164 (ObjNum 1623 snap 0 seq_num 1623) dirty exists 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3955: left oid 367 (ObjNum 1623 snap 0 seq_num 1623) 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3959: got (0) Success 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3962: got (0) Success 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 11 v 0 (ObjNum 539 snap 0 seq_num 539) dirty exists 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: finishing write tid 2 to smithi16030472-88 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: finishing write tid 3 to smithi16030472-88 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: finishing write tid 4 to smithi16030472-88 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: finishing write tid 5 to smithi16030472-88 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: oid 88 updating version 0 to 1062 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: oid 88 updating version 1062 to 1064 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: oid 88 version 1064 is already newer than 1063 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: oid 88 updating version 1064 to 1065 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 88 v 1065 (ObjNum 1625 snap 0 seq_num 1625) dirty exists 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3961: left oid 88 (ObjNum 1625 snap 0 seq_num 1625) 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3946: got (0) Success 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 270 v 0 (ObjNum 1031 snap 0 seq_num 1031) dirty exists 2024-09-21T18:10:29.065 INFO:tasks.rados.rados.0.smithi160.stdout:3971: finishing write tid 1 to smithi16030472-240 2024-09-21T18:10:29.066 INFO:tasks.rados.rados.0.smithi160.stdout:3972: expect (ObjNum 1621 snap 0 seq_num 1621) 2024-09-21T18:10:29.081 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1798752 2024-09-21T18:10:29.126 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1742154 2024-09-21T18:10:29.154 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:29.152+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:29.165 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:29.165 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:29.165 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.181 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:10:29.197 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:10:29.193+0000 7f728550b640 -1 osd.14 129 handle_osd_map injecting map crc failure 2024-09-21T18:10:29.197 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:29.191+0000 7f440980f640 -1 osd.13 129 handle_osd_map injecting map crc failure 2024-09-21T18:10:29.202 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2603851 2024-09-21T18:10:29.213 INFO:tasks.rados.rados.0.smithi160.stdout:3969: finishing write tid 1 to smithi16030472-210 2024-09-21T18:10:29.213 INFO:tasks.rados.rados.0.smithi160.stdout:3969: finishing write tid 2 to smithi16030472-210 2024-09-21T18:10:29.213 INFO:tasks.rados.rados.0.smithi160.stdout:3946: done (10 left) 2024-09-21T18:10:29.213 INFO:tasks.rados.rados.0.smithi160.stdout:3955: done (9 left) 2024-09-21T18:10:29.213 INFO:tasks.rados.rados.0.smithi160.stdout:3959: done (8 left) 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3961: done (7 left) 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3962: done (6 left) 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3972: done (5 left) 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3974: delete oid 133 current snap is 0 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3973: got (16) Device or resource busy 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3971: finishing write tid 2 to smithi16030472-240 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3971: finishing write tid 3 to smithi16030472-240 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3971: finishing write tid 4 to smithi16030472-240 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: finishing write tid 3 to smithi16030472-210 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: finishing write tid 4 to smithi16030472-210 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: finishing write tid 5 to smithi16030472-210 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: oid 210 updating version 0 to 1166 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: oid 210 updating version 1166 to 1167 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: oid 210 updating version 1167 to 1169 2024-09-21T18:10:29.214 INFO:tasks.rados.rados.0.smithi160.stdout:3969: oid 210 version 1169 is already newer than 1168 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 210 v 1169 (ObjNum 1627 snap 0 seq_num 1627) dirty exists 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3969: left oid 210 (ObjNum 1627 snap 0 seq_num 1627) 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3967: got (0) Success 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 485 v 0 (ObjNum 817 snap 0 seq_num 817) dirty exists 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3964: got (0) Success 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 336 v 0 (ObjNum 813 snap 0 seq_num 813) dirty exists 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: finishing write tid 5 to smithi16030472-240 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: oid 240 updating version 0 to 1069 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: oid 240 version 1069 is already newer than 1068 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: oid 240 version 1069 is already newer than 1066 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: oid 240 version 1069 is already newer than 1067 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 240 v 1069 (ObjNum 1628 snap 0 seq_num 1628) dirty exists 2024-09-21T18:10:29.215 INFO:tasks.rados.rados.0.smithi160.stdout:3971: left oid 240 (ObjNum 1628 snap 0 seq_num 1628) 2024-09-21T18:10:29.256 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:29.251+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:29.307 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1531 4.0 e51/33 4:2aa3e9e6:::smithi16030472-341:head v 51'121, mlcod=51'117)", 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 177.709297193, 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30050849899999998, 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.308 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595801+0000", 2024-09-21T18:10:29.309 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595804+0000", 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.869e-06 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595815+0000", 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1253000000000001e-05 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.310 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595842+0000", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6395999999999999e-05 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.595855+0000", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3599e-05 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895125+0000", 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.299269656 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.311 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.895159+0000", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3862000000000002e-05 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896189+0000", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001030556 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:31.896310+0000", 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000120308 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.312 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2891 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'237, mlcod=59'234)", 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.27932724600001, 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31284122399999997, 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.313 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025771+0000", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.025773+0000", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8199999999999999e-06 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.314 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026759+0000", 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00098598799999999993 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026781+0000", 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2480000000000002e-05 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.026800+0000", 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8609000000000001e-05 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.315 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129145+0000", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10234452500000001 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.129180+0000", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5366e-05 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338452+0000", 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.2092725 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.316 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.338612+0000", 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015993600000000001 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2893 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'239, mlcod=59'234)", 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.18180687099999, 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42264381400000001, 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.317 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123291+0000", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.318 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123300+0000", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.0940000000000008e-06 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123451+0000", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015106200000000001 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123477+0000", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6693e-05 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123492+0000", 2024-09-21T18:10:29.319 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4195999999999999e-05 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233343+0000", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109851112 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.233377+0000", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4496000000000003e-05 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545841+0000", 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31246387599999997 2024-09-21T18:10:29.320 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545935+0000", 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4284999999999996e-05 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2894 4.2 e60/33 4:70bf26e5:::smithi16030472-249:head v 60'240, mlcod=59'234)", 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.18112262099999, 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.422038898, 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.321 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123976+0000", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.322 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123984+0000", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5920000000000001e-06 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.123996+0000", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1970000000000001e-05 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124023+0000", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6356e-05 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124036+0000", 2024-09-21T18:10:29.323 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3443e-05 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234376+0000", 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110340122 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.234408+0000", 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1766e-05 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.324 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.545945+0000", 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31153713700000002 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.546015+0000", 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.9511999999999994e-05 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2896 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'241, mlcod=59'234)", 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:29.325 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.18028926599999, 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42445530599999998, 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.326 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124809+0000", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.124832+0000", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.2685999999999999e-05 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125748+0000", 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00091599300000000004 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.327 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125775+0000", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6979e-05 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.125791+0000", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6379999999999999e-05 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335008+0000", 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20921717200000001 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.328 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.335064+0000", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5689000000000002e-05 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549188+0000", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21412368200000001 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549264+0000", 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.6724999999999995e-05 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.329 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2897 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'242, mlcod=59'234)", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.17757063600001, 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.42180716400000001, 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.330 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127528+0000", 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.127531+0000", 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.416e-06 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128100+0000", 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00056844600000000001 2024-09-21T18:10:29.331 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128127+0000", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7832999999999999e-05 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.128141+0000", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3991999999999999e-05 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438906+0000", 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31076425800000002 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.332 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.438940+0000", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4022000000000001e-05 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549271+0000", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.110330856 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549335+0000", 2024-09-21T18:10:29.333 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.4340999999999999e-05 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2898 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'243, mlcod=59'234)", 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.08176223000001, 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.326073803, 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.334 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223336+0000", 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.223369+0000", 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2665e-05 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.335 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224269+0000", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090022400000000003 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224296+0000", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7138999999999999e-05 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.224311+0000", 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4610000000000001e-05 2024-09-21T18:10:29.336 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541275+0000", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31696466099999998 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.541310+0000", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4739000000000001e-05 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549338+0000", 2024-09-21T18:10:29.337 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0080279570000000005 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549410+0000", 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.1808000000000001e-05 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2899 4.2 e60/33 4:58b6fd03:::smithi16030472-180:head v 60'244, mlcod=59'234)", 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.07945768900001, 2024-09-21T18:10:29.338 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32384043699999998, 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225641+0000", 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.339 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225649+0000", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9559999999999997e-06 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225694+0000", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.5108999999999998e-05 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.225735+0000", 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.0917999999999997e-05 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.340 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.325877+0000", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100142762 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544083+0000", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21820531100000001 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.544118+0000", 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4829000000000001e-05 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.341 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549412+0000", 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052949169999999997 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.549481+0000", 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8634999999999996e-05 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.342 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3799 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'370, mlcod=72'364)", 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 136.10182906899999, 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.350174767, 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.343 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203269+0000", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.203271+0000", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3540000000000001e-06 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204249+0000", 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0009786459999999999 2024-09-21T18:10:29.344 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.204274+0000", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.4471999999999999e-05 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304380+0000", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10010635700000001 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404645+0000", 2024-09-21T18:10:29.345 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10026493 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.404682+0000", 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.7185999999999999e-05 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553377+0000", 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.14869442999999999 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.346 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.553444+0000", 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.7391999999999994e-05 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3800 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'371, mlcod=72'364)", 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 136.00067338700001, 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25689167800000001, 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.347 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304425+0000", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.304428+0000", 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9359999999999999e-06 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.348 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305162+0000", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073421399999999996 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305183+0000", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1072000000000001e-05 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305196+0000", 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.257e-05 2024-09-21T18:10:29.349 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406751+0000", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10155505099999999 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.406790+0000", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.8933000000000001e-05 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561174+0000", 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.154384258 2024-09-21T18:10:29.350 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561317+0000", 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00014264400000000001 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3801 4.0 e73/33 4:29dccdbd:::smithi16030472-144:head v 73'372, mlcod=72'364)", 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 135.999877725, 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.25614924900000002, 2024-09-21T18:10:29.351 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305221+0000", 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.352 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.305222+0000", 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6199999999999999e-06 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306310+0000", 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001087974 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.353 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306328+0000", 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.7969999999999999e-05 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.306339+0000", 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1263e-05 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.408967+0000", 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102627518 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.354 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.409018+0000", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.0881999999999998e-05 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561323+0000", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.152305199 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.561370+0000", 2024-09-21T18:10:29.355 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.6823000000000002e-05 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:856 4.2 e73/33 4:692fac94:::smithi16030472-329:head v 73'341, mlcod=72'339)", 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 135.978024312, 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.28786457100000001, 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.356 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327074+0000", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.327075+0000", 2024-09-21T18:10:29.357 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330909+0000", 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0038347419999999999 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330936+0000", 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7140000000000001e-05 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.358 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.330951+0000", 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4749000000000001e-05 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407644+0000", 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.076692477999999994 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.407697+0000", 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.3856000000000001e-05 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.359 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614855+0000", 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207157708 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614939+0000", 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.3415e-05 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.360 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(osd.15.0:857 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'342, mlcod=72'339)", 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 135.96867162800001, 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27859368200000001, 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.361 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336427+0000", 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.336430+0000", 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0240000000000002e-06 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.362 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339771+0000", 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0033414870000000002 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339800+0000", 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8866000000000001e-05 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.339815+0000", 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4823e-05 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.363 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413428+0000", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.073613095000000003 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.413464+0000", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6109999999999998e-05 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.614947+0000", 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20148285799999999 2024-09-21T18:10:29.364 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615020+0000", 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3418999999999996e-05 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3782 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'343, mlcod=72'339)", 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 135.96329698, 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27329392600000002, 2024-09-21T18:10:29.365 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341801+0000", 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.366 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.341804+0000", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.977e-06 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342693+0000", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00088848400000000004 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342720+0000", 2024-09-21T18:10:29.367 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6716e-05 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.342734+0000", 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.395e-05 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417887+0000", 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075153954999999995 2024-09-21T18:10:29.368 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.417918+0000", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0423e-05 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615026+0000", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.197108546 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615095+0000", 2024-09-21T18:10:29.369 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.8874999999999994e-05 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:3783 4.2 e73/33 4:6f38095b:::smithi16030472-243:head v 73'344, mlcod=72'339)", 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 135.96074870499999, 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.27081757299999998, 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.370 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344350+0000", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344353+0000", 2024-09-21T18:10:29.371 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9960000000000002e-06 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344365+0000", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2597e-05 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344392+0000", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6296e-05 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.344405+0000", 2024-09-21T18:10:29.372 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3596e-05 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419764+0000", 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.075358779000000001 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.419851+0000", 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6984999999999995e-05 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.373 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615101+0000", 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19525023899999999 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:13.615167+0000", 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.6085000000000002e-05 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4138 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'379, mlcod=75'375)", 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:29.374 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 129.73304816800001, 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31439997200000003, 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572050+0000", 2024-09-21T18:10:29.375 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572051+0000", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.263e-06 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572673+0000", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062116700000000005 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572689+0000", 2024-09-21T18:10:29.376 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6509999999999999e-05 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572699+0000", 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5270000000000004e-06 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674806+0000", 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10210781100000001 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.377 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.674859+0000", 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2862999999999999e-05 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886296+0000", 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.21143626300000001 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886450+0000", 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000154568 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.378 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4139 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'380, mlcod=75'375)", 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 129.73238187499999, 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.313843127, 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572716+0000", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572717+0000", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.5000000000000001e-07 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572747+0000", 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0023e-05 2024-09-21T18:10:29.380 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572757+0000", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.4530000000000008e-06 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572764+0000", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0999999999999998e-06 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677084+0000", 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10431988 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.381 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.677144+0000", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.0365999999999998e-05 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886460+0000", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20931606699999999 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886560+0000", 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.9288000000000002e-05 2024-09-21T18:10:29.382 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4140 4.2 e75/33 4:6e9cef1a:::smithi16030472-436:head v 75'381, mlcod=75'375)", 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 129.73228976300001, 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31380345399999998, 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.383 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572809+0000", 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.7000000000000004e-07 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.384 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572816+0000", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3960000000000003e-06 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572827+0000", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1464000000000001e-05 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.572835+0000", 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.1559999999999995e-06 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.385 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778857+0000", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20602174600000001 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.778890+0000", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3139000000000002e-05 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886570+0000", 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10767932099999999 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.386 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:19.886612+0000", 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.2462000000000001e-05 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:6894 4.2 e96/33 4:60b99da3:::smithi16030472-227:head v 96'728, mlcod=96'726)", 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 73.461649320000006, 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.303226251, 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.387 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843449+0000", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.843452+0000", 2024-09-21T18:10:29.388 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0429999999999999e-06 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844181+0000", 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00072923900000000002 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.844209+0000", 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7313000000000001e-05 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.389 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944352+0000", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100143339 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944508+0000", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015568600000000001 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:15.944543+0000", 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5649999999999999e-05 2024-09-21T18:10:29.390 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146570+0000", 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20202621900000001 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:09:16.146675+0000", 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00010576199999999999 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.391 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:9447 4.2 e121/33 4:4279d287:::smithi16030472-294:head v 121'1069, mlcod=121'1065)", 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 20.657079237000001, 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30225417700000001, 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.392 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648019+0000", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648022+0000", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.1599999999999998e-06 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648035+0000", 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2643000000000001e-05 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.393 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.648063+0000", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8229999999999999e-05 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.748203+0000", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10013965800000001 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.948607+0000", 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20040434600000001 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.394 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.948647+0000", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.9344999999999997e-05 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.950195+0000", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.001548787 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:10:08.950273+0000", 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8008000000000003e-05 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.395 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.396 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.396 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.396 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3964: done (5 left) 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3967: done (4 left) 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3969: done (3 left) 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3971: done (2 left) 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3973: done (1 left) 2024-09-21T18:10:29.397 INFO:tasks.rados.rados.0.smithi160.stdout:3974: done (0 left) 2024-09-21T18:10:29.398 INFO:tasks.rados.rados.0.smithi160.stdout:3975: delete oid 277 current snap is 0 2024-09-21T18:10:29.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:10:29.399 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:29.391+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:29.498 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:29.495+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:29.568 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.568 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:29.568 INFO:teuthology.orchestra.run.smithi070.stdout: "num_ops": 0 2024-09-21T18:10:29.568 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.583 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:10:29.600 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:29.596+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:29.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T18:10:29.701 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:29.695+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:29.703 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.703 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [], 2024-09-21T18:10:29.703 INFO:teuthology.orchestra.run.smithi070.stdout: "complaint_time": 30, 2024-09-21T18:10:29.704 INFO:teuthology.orchestra.run.smithi070.stdout: "num_blocked_ops": 0 2024-09-21T18:10:29.704 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.718 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:10:29.722 INFO:tasks.rados.rados.0.smithi160.stdout:3975: done (0 left) 2024-09-21T18:10:29.722 INFO:tasks.rados.rados.0.smithi160.stdout:3976: write oid 191 current snap is 0 2024-09-21T18:10:29.722 INFO:tasks.rados.rados.0.smithi160.stdout:3976: seq_num 1631 ranges {612217=430661,1458837=538242,2656313=522653} 2024-09-21T18:10:29.734 INFO:tasks.rados.rados.0.smithi160.stdout:3976: writing smithi16030472-191 from 612217 to 1042878 tid 1 2024-09-21T18:10:29.746 INFO:tasks.rados.rados.0.smithi160.stdout:3976: writing smithi16030472-191 from 1458837 to 1997079 tid 2 2024-09-21T18:10:29.760 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-09-21T18:10:29.760 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T18:10:29.761 INFO:tasks.rados.rados.0.smithi160.stdout:3976: writing smithi16030472-191 from 2656313 to 3178966 tid 3 2024-09-21T18:10:29.761 INFO:tasks.rados.rados.0.smithi160.stdout:3977: cache_evict oid 203 snap -1 2024-09-21T18:10:29.761 INFO:tasks.rados.rados.0.smithi160.stdout:3978: cache_evict oid 244 snap -1 2024-09-21T18:10:29.761 INFO:tasks.rados.rados.0.smithi160.stdout:3979: write oid 27 current snap is 0 2024-09-21T18:10:29.761 INFO:tasks.rados.rados.0.smithi160.stdout:3979: seq_num 1632 ranges {629024=505910,1917319=681900,3083736=345288} 2024-09-21T18:10:29.763 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:29.761+0000 7f3ce267c640 -1 osd.6 129 handle_osd_map injecting map crc failure 2024-09-21T18:10:29.774 INFO:tasks.rados.rados.0.smithi160.stdout:3979: writing smithi16030472-27 from 629024 to 1134934 tid 1 2024-09-21T18:10:29.781 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-09-21T18:10:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T18:10:29.788 INFO:tasks.rados.rados.0.smithi160.stdout:3979: writing smithi16030472-27 from 1917319 to 2599219 tid 2 2024-09-21T18:10:29.795 INFO:tasks.rados.rados.0.smithi160.stdout:3979: writing smithi16030472-27 from 3083736 to 3429024 tid 3 2024-09-21T18:10:29.796 INFO:tasks.rados.rados.0.smithi160.stdout:3980: delete oid 436 current snap is 0 2024-09-21T18:10:29.801 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-09-21T18:10:29.802 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T18:10:29.802 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:29.801+0000 7f3cf1c5d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 26343) UID: 0 2024-09-21T18:10:29.822 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-09-21T18:10:29.822 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "size": 20, 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 600, 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "ops": [ 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:536 3.1 e47/31 3:92ed8e1e:::smithi16030472-119:head v 47'87, mlcod=47'83)", 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 191.001800632, 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204533561, 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.843 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838544+0000", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838545+0000", 2024-09-21T18:10:29.844 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.3799999999999996e-07 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838551+0000", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.4859999999999997e-06 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.838559+0000", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.8080000000000005e-06 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938687+0000", 2024-09-21T18:10:29.845 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100128736 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938785+0000", 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.7205000000000002e-05 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:18.938818+0000", 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3127999999999999e-05 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.846 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.042990+0000", 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10417233200000001 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:19.043078+0000", 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.7928000000000002e-05 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1001 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'193, mlcod=49'190)", 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.59015833800001, 2024-09-21T18:10:29.847 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.32058132900000003, 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250187+0000", 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.848 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250190+0000", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.2270000000000002e-06 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250691+0000", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00050105899999999999 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250719+0000", 2024-09-21T18:10:29.849 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8099999999999999e-05 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.250734+0000", 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4409e-05 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351006+0000", 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100272839 2024-09-21T18:10:29.850 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.351069+0000", 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2149000000000004e-05 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570664+0000", 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.219595238 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.851 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570768+0000", 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000104308 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1002 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'194, mlcod=49'190)", 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.587327466, 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31782431500000002, 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.852 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253018+0000", 2024-09-21T18:10:29.853 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253021+0000", 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9560000000000002e-06 2024-09-21T18:10:29.854 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253755+0000", 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00073406899999999995 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253816+0000", 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1561000000000002e-05 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.855 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.253833+0000", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.6682999999999999e-05 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453692+0000", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19985925600000001 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.453744+0000", 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.2031999999999999e-05 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.856 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570779+0000", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11703487899999999 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.570842+0000", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2879000000000006e-05 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1003 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'195, mlcod=49'190)", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:29.857 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.583354558, 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31610690299999999, 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.858 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256991+0000", 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.256994+0000", 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.039e-06 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257587+0000", 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059370400000000002 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257614+0000", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.707e-05 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.257632+0000", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8136000000000001e-05 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.455956+0000", 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19832360800000001 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.860 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.456069+0000", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00011317 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.572912+0000", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.116842417 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573097+0000", 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00018575899999999999 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.861 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1004 3.1 e49/31 3:837e1005:::smithi16030472-224:head v 49'196, mlcod=49'190)", 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.58023760699999, 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.31307449399999998, 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:29.862 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260107+0000", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260110+0000", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9100000000000001e-06 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260123+0000", 2024-09-21T18:10:29.863 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2500000000000001e-05 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260149+0000", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6346000000000002e-05 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.260163+0000", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3543000000000001e-05 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458534+0000", 2024-09-21T18:10:29.864 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19837121299999999 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.458594+0000", 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.9979000000000003e-05 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573109+0000", 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11451463100000001 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.865 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573182+0000", 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.3372000000000002e-05 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1006 3.1 e49/31 3:8894617b:::smithi16030472-225:head v 49'197, mlcod=49'191)", 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.57830660299999, 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.311313063, 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.866 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262038+0000", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262041+0000", 2024-09-21T18:10:29.867 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7630000000000001e-06 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262882+0000", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00084083900000000002 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262909+0000", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7345999999999999e-05 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.262928+0000", 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.8094e-05 2024-09-21T18:10:29.868 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459202+0000", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19627407899999999 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.459257+0000", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5222000000000001e-05 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573189+0000", 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11393253 2024-09-21T18:10:29.869 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573352+0000", 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016218999999999999 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1036 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'198, mlcod=49'191)", 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.575199844, 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.308289176, 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.870 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265145+0000", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.871 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.265148+0000", 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0000000000000001e-06 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266236+0000", 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0010881739999999999 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266264+0000", 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7305999999999999e-05 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.872 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.266277+0000", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3315999999999999e-05 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461372+0000", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19509494799999999 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.461428+0000", 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.5557999999999999e-05 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.873 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573360+0000", 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.111932632 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573434+0000", 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.4241999999999998e-05 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.874 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.875 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.875 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.875 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1037 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'199, mlcod=49'191)", 2024-09-21T18:10:29.875 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.57204216, 2024-09-21T18:10:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30523983300000002, 2024-09-21T18:10:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.876 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268303+0000", 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.877 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268306+0000", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9299999999999999e-06 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268713+0000", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00040703899999999999 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268740+0000", 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7282000000000001e-05 2024-09-21T18:10:29.878 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.879 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.879 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.268754+0000", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3573e-05 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463840+0000", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.19508631100000001 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.463896+0000", 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6229e-05 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.880 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573442+0000", 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.109545711 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573543+0000", 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000100758 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.881 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1038 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'200, mlcod=49'191)", 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.569809109, 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30310639299999997, 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.882 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270536+0000", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270539+0000", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8760000000000001e-06 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270924+0000", 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038559599999999999 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.883 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270951+0000", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6729000000000001e-05 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.270965+0000", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3753e-05 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.565970+0000", 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.29500490499999998 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.884 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.566003+0000", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.3448999999999998e-05 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573551+0000", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.007547237 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573642+0000", 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 9.1848000000000006e-05 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.885 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1039 3.1 e49/31 3:93f3a785:::smithi16030472-231:head v 49'201, mlcod=49'191)", 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.56755743599999, 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.30093925500000002, 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:29.886 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272788+0000", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272791+0000", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9500000000000001e-06 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272803+0000", 2024-09-21T18:10:29.887 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.2146e-05 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272829+0000", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.6400000000000001e-05 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.272843+0000", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3383e-05 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567528+0000", 2024-09-21T18:10:29.888 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.294685325 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.567584+0000", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 5.6295000000000002e-05 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573651+0000", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0060667680000000002 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.573727+0000", 2024-09-21T18:10:29.889 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.5988000000000005e-05 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1046 3.1 e49/31 3:956c7121:::smithi16030472-233:head v 49'202, mlcod=49'191)", 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 184.474979848, 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.210846596, 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.890 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365365+0000", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.365368+0000", 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.027e-06 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.891 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366180+0000", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00081138300000000005 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.366209+0000", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9016000000000001e-05 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.466320+0000", 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100111332 2024-09-21T18:10:29.892 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568256+0000", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.101935519 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.568345+0000", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.9561999999999994e-05 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576125+0000", 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0077802549999999998 2024-09-21T18:10:29.893 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:25.576212+0000", 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6502000000000006e-05 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1097 3.1 e49/31 3:9ce45f1e:::smithi16030472-244:head v 49'210, mlcod=49'204)", 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 183.698310472, 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.24095656100000001, 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.894 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.895 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142035+0000", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 4.8299999999999997e-07 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142418+0000", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00038336199999999999 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142432+0000", 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.363e-05 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.896 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.142440+0000", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.9130000000000003e-06 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247049+0000", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104608993 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.247083+0000", 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4344999999999997e-05 2024-09-21T18:10:29.897 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382929+0000", 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.13584527299999999 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:26.382991+0000", 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2562000000000002e-05 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.898 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1623 3.1 e51/31 3:badc2fa3:::smithi16030472-360:head v 51'304, mlcod=51'300)", 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 177.688802888, 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20678823199999999, 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.899 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151542+0000", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.151544+0000", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0899999999999999e-06 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152069+0000", 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00052430599999999999 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.900 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152089+0000", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.0343000000000002e-05 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.152101+0000", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.1888999999999999e-05 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252620+0000", 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100519068 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.901 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.252799+0000", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00017869500000000001 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358225+0000", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.105425989 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.358330+0000", 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.000105852 2024-09-21T18:10:29.902 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:1627 3.1 e51/31 3:96ca449d:::smithi16030472-361:head v 51'307, mlcod=51'302)", 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 177.581972909, 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.204676887, 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.903 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258372+0000", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258375+0000", 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.543e-06 2024-09-21T18:10:29.904 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258970+0000", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00059547300000000001 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.258990+0000", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9939e-05 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.359072+0000", 2024-09-21T18:10:29.905 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100081975 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459326+0000", 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100254303 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.459388+0000", 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.1408000000000006e-05 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.906 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.462964+0000", 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0035758029999999998 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:32.463049+0000", 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.5443000000000002e-05 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2171 3.1 e54/31 3:b57571cd:::smithi16030472-481:head v 54'404, mlcod=53'401)", 2024-09-21T18:10:29.907 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 170.24860135700001, 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.207507252, 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.908 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591744+0000", 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591755+0000", 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.0890000000000001e-05 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591769+0000", 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.4182000000000001e-05 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.909 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.591792+0000", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.3269e-05 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.691939+0000", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014743299999999 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692098+0000", 2024-09-21T18:10:29.910 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015844500000000001 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.692135+0000", 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.6751000000000001e-05 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799181+0000", 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.107046107 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.911 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:39.799251+0000", 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0174999999999998e-05 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2443 3.1 e55/31 3:9332d87c:::smithi16030472-126:head v 55'421, mlcod=55'419)", 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 165.47660988800001, 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.209157064, 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.912 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363735+0000", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.363738+0000", 2024-09-21T18:10:29.913 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.824e-06 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364365+0000", 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00062699900000000005 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364387+0000", 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.1807e-05 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.914 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.364406+0000", 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9636e-05 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.464881+0000", 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10047500600000001 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.465021+0000", 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013981200000000001 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.915 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572724+0000", 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10770310900000001 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:44.572892+0000", 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00016787099999999999 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.916 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:2912 3.1 e60/31 3:867cc668:::smithi16030472-302:head v 60'439, mlcod=60'435)", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 155.59446143299999, 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.206110025, 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245884+0000", 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.917 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.245887+0000", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.0730000000000001e-06 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246574+0000", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00068702499999999996 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246601+0000", 2024-09-21T18:10:29.918 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.7087e-05 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.246614+0000", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.3638999999999999e-05 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349051+0000", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10243622600000001 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.349086+0000", 2024-09-21T18:10:29.919 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.5615999999999999e-05 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451907+0000", 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.102820458 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:07:54.451994+0000", 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 8.6901e-05 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:4078 3.1 e74/31 3:9f8299d6:::smithi16030472-63:head v 74'484, mlcod=74'479)", 2024-09-21T18:10:29.920 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 131.05433698499999, 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20972542399999999, 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786008+0000", 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.786028+0000", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 1.9830000000000002e-05 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791239+0000", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0052115349999999998 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.791269+0000", 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9529000000000001e-05 2024-09-21T18:10:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891400+0000", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100131497 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891540+0000", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00013984100000000001 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.891603+0000", 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.2725999999999995e-05 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995663+0000", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.104059717 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:18.995734+0000", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 7.0748999999999996e-05 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5021 3.1 e80/31 3:937a2b85:::smithi16030472-221:head v 80'516, mlcod=79'514)", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 113.744378525, 2024-09-21T18:10:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.212824341, 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095967+0000", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.095970+0000", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.41e-06 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097241+0000", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00127137 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.097270+0000", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.8464999999999999e-05 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197410+0000", 2024-09-21T18:10:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.100139929 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197566+0000", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015609 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.197629+0000", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 6.3256000000000006e-05 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308667+0000", 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.11103747999999999 2024-09-21T18:10:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:36.308791+0000", 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00012434100000000001 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "description": "osd_repop(client.4876.0:5159 3.1 e81/31 3:af0b33ab:::smithi16030472-25:head v 81'524, mlcod=81'522)", 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "initiated_at": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "age": 110.499976214, 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.20535973399999999, 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "continuous": false, 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "type_data": { 2024-09-21T18:10:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "events": [ 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "initiated", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "header_read", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340369+0000", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "throttled", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.340372+0000", 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9170000000000002e-06 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "all_read", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341275+0000", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00090325799999999999 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "dispatched", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.341304+0000", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 2.9221999999999999e-05 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "queued_for_pg", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.441450+0000", 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10014619299999999 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "reached_pg", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541779+0000", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.10032864499999999 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "started", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.541814+0000", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 3.4632000000000003e-05 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "commit_sent", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545575+0000", 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.0037613880000000001 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: }, 2024-09-21T18:10:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: { 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: "event": "done", 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: "time": "2024-09-21T18:08:39.545729+0000", 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: "duration": 0.00015347900000000001 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: } 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-09-21T18:10:29.932 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-09-21T18:10:29.933 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-09-21T18:10:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T18:10:29.934 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-09-21T18:10:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T18:10:29.935 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:10:29.937 INFO:tasks.rados.rados.0.smithi160.stdout:3978: got (16) Device or resource busy 2024-09-21T18:10:29.937 INFO:tasks.rados.rados.0.smithi160.stdout:3978: done (4 left) 2024-09-21T18:10:29.937 INFO:tasks.rados.rados.0.smithi160.stdout:3980: done (3 left) 2024-09-21T18:10:29.937 INFO:tasks.rados.rados.0.smithi160.stdout:3981: copy_from oid 273 from oid 190 current snap is 0 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3982: delete oid 18 current snap is 0 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3976: finishing write tid 1 to smithi16030472-191 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3976: finishing write tid 2 to smithi16030472-191 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3976: finishing write tid 3 to smithi16030472-191 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3977: got (16) Device or resource busy 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3977: done (4 left) 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3982: done (3 left) 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3983: write oid 371 current snap is 0 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3983: seq_num 1635 ranges {679445=448010,1692765=429227,2766005=567741} 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3983: writing smithi16030472-371 from 679445 to 1127455 tid 1 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3983: writing smithi16030472-371 from 1692765 to 2121992 tid 2 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3983: writing smithi16030472-371 from 2766005 to 3333746 tid 3 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3984: read oid 294 snap -1 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3984: expect (ObjNum 1515 snap 0 seq_num 1515) 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3985: copy_from oid 76 from oid 481 current snap is 0 2024-09-21T18:10:29.938 INFO:tasks.rados.rados.0.smithi160.stdout:3976: finishing write tid 4 to smithi16030472-191 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3986: write (excl) oid 307 current snap is 0 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3986: seq_num 1636 ranges {696252=523259,1751247=689238,3193428=302824} 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3986: writing smithi16030472-307 from 696252 to 1219511 tid 1 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3986: writing smithi16030472-307 from 1751247 to 2440485 tid 2 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3986: writing smithi16030472-307 from 3193428 to 3496252 tid 3 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3987: copy_from oid 230 from oid 125 current snap is 0 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: finishing write tid 5 to smithi16030472-191 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: oid 191 updating version 0 to 1073 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: oid 191 version 1073 is already newer than 1072 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: oid 191 version 1073 is already newer than 1071 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: oid 191 updating version 1073 to 1074 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 191 v 1074 (ObjNum 1631 snap 0 seq_num 1631) dirty exists 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: left oid 191 (ObjNum 1631 snap 0 seq_num 1631) 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3976: done (7 left) 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3988: cache_try_flush oid 180 snap -1 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3989: read oid 178 snap -1 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3989: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:10:29.939 INFO:tasks.rados.rados.0.smithi160.stdout:3990: write oid 359 current snap is 0 2024-09-21T18:10:29.940 INFO:tasks.rados.rados.0.smithi160.stdout:3990: seq_num 1637 ranges {713059=598508,2093376=432896,3104498=408561} 2024-09-21T18:10:29.940 INFO:tasks.rados.rados.0.smithi160.stdout:3990: writing smithi16030472-359 from 713059 to 1311567 tid 1 2024-09-21T18:10:29.940 INFO:tasks.rados.rados.0.smithi160.stdout:3990: writing smithi16030472-359 from 2093376 to 2526272 tid 2 2024-09-21T18:10:29.940 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:29.927+0000 7f4416dec640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 13 (PID: 26236) UID: 0 2024-09-21T18:10:29.940 INFO:tasks.rados.rados.0.smithi160.stdout:3990: writing smithi16030472-359 from 3104498 to 3513059 tid 3 2024-09-21T18:10:29.941 INFO:tasks.rados.rados.0.smithi160.stdout:3991: delete oid 493 current snap is 0 2024-09-21T18:10:29.941 INFO:tasks.rados.rados.0.smithi160.stdout:3984: expect (ObjNum 1515 snap 0 seq_num 1515) 2024-09-21T18:10:29.958 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-09-21T18:10:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T18:10:29.959 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T18:10:29.983 INFO:teuthology.orchestra.run.smithi029.stdout:osd.8: {} 2024-09-21T18:10:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:osd.8: osd_enable_op_tracker = '' 2024-09-21T18:10:30.015 INFO:teuthology.orchestra.run.smithi029.stdout:osd.9: {} 2024-09-21T18:10:30.015 INFO:teuthology.orchestra.run.smithi029.stderr:osd.9: osd_enable_op_tracker = '' 2024-09-21T18:10:30.034 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:10:30.032+0000 7f94e8b23640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 15 (PID: 26590) UID: 0 2024-09-21T18:10:30.041 INFO:teuthology.orchestra.run.smithi029.stdout:osd.10: {} 2024-09-21T18:10:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:osd.10: osd_enable_op_tracker = '' 2024-09-21T18:10:30.063 INFO:teuthology.orchestra.run.smithi029.stdout:osd.11: {} 2024-09-21T18:10:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:osd.11: osd_enable_op_tracker = '' 2024-09-21T18:10:30.065 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.065 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.079 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:10:30.086 INFO:teuthology.orchestra.run.smithi029.stdout:osd.12: {} 2024-09-21T18:10:30.087 INFO:teuthology.orchestra.run.smithi029.stderr:osd.12: osd_enable_op_tracker = '' 2024-09-21T18:10:30.093 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:30.114 INFO:teuthology.orchestra.run.smithi029.stdout:osd.13: {} 2024-09-21T18:10:30.114 INFO:teuthology.orchestra.run.smithi029.stderr:osd.13: osd_enable_op_tracker = '' 2024-09-21T18:10:30.135 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:30.133+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:30.138 INFO:teuthology.orchestra.run.smithi029.stdout:osd.14: {} 2024-09-21T18:10:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:osd.14: osd_enable_op_tracker = '' 2024-09-21T18:10:30.162 INFO:tasks.rados.rados.0.smithi160.stdout:3979: finishing write tid 1 to smithi16030472-27 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3979: finishing write tid 2 to smithi16030472-27 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3984: done (10 left) 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3991: done (9 left) 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3992: delete oid 451 current snap is 0 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3981: finishing copy_from racing read to smithi16030472-273 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3983: finishing write tid 1 to smithi16030472-371 2024-09-21T18:10:30.163 INFO:tasks.rados.rados.0.smithi160.stdout:3989: expect (ObjNum 1 snap 0 seq_num 1) 2024-09-21T18:10:30.163 INFO:teuthology.orchestra.run.smithi029.stdout:osd.15: {} 2024-09-21T18:10:30.164 INFO:teuthology.orchestra.run.smithi029.stderr:osd.15: osd_enable_op_tracker = '' 2024-09-21T18:10:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T18:10:30.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T18:10:30.185 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:30.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:10:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.214 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.216 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 1805101 2024-09-21T18:10:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:3988: got (0) Success 2024-09-21T18:10:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 180 v 0 (ObjNum 1577 snap 0 seq_num 1577) dirty exists 2024-09-21T18:10:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:3985: finishing copy_from racing read to smithi16030472-76 2024-09-21T18:10:30.218 INFO:tasks.rados.rados.0.smithi160.stdout:3987: finishing copy_from racing read to smithi16030472-230 2024-09-21T18:10:30.219 INFO:tasks.rados.rados.0.smithi160.stdout:3979: finishing write tid 3 to smithi16030472-27 2024-09-21T18:10:30.219 INFO:tasks.rados.rados.0.smithi160.stdout:3983: finishing write tid 2 to smithi16030472-371 2024-09-21T18:10:30.219 INFO:tasks.rados.rados.0.smithi160.stdout:3983: finishing write tid 3 to smithi16030472-371 2024-09-21T18:10:30.219 INFO:tasks.rados.rados.0.smithi160.stdout:3983: finishing write tid 4 to smithi16030472-371 2024-09-21T18:10:30.228 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:30.236 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:30.231+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:30.279 INFO:tasks.rados.rados.0.smithi160.stdout:3979: finishing write tid 4 to smithi16030472-27 2024-09-21T18:10:30.280 INFO:tasks.rados.rados.0.smithi160.stdout:3979: finishing write tid 5 to smithi16030472-27 2024-09-21T18:10:30.280 INFO:tasks.rados.rados.0.smithi160.stdout:3979: oid 27 updating version 0 to 1151 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3979: oid 27 version 1151 is already newer than 1149 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3979: oid 27 version 1151 is already newer than 1150 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3979: oid 27 version 1151 is already newer than 1148 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 27 v 1151 (ObjNum 1632 snap 0 seq_num 1632) dirty exists 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3979: left oid 27 (ObjNum 1632 snap 0 seq_num 1632) 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: finishing write tid 1 to smithi16030472-307 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: finishing write tid 2 to smithi16030472-307 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: finishing write tid 3 to smithi16030472-307 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: finishing write tid 4 to smithi16030472-307 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: finishing write tid 5 to smithi16030472-307 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: oid 307 updating version 0 to 1153 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: oid 307 updating version 1153 to 1154 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: oid 307 updating version 1154 to 1155 2024-09-21T18:10:30.281 INFO:tasks.rados.rados.0.smithi160.stdout:3986: oid 307 version 1155 is already newer than 1152 2024-09-21T18:10:30.282 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 307 v 1155 (ObjNum 1636 snap 0 seq_num 1636) dirty exists 2024-09-21T18:10:30.282 INFO:tasks.rados.rados.0.smithi160.stdout:3986: left oid 307 (ObjNum 1636 snap 0 seq_num 1636) 2024-09-21T18:10:30.292 INFO:tasks.rados.rados.0.smithi160.stdout:3987: finishing copy_from to smithi16030472-230 2024-09-21T18:10:30.292 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 230 v 940 (ObjNum 1333 snap 0 seq_num 1333) dirty exists 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: finishing write tid 5 to smithi16030472-371 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: oid 371 updating version 0 to 1078 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: oid 371 version 1078 is already newer than 1077 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: oid 371 version 1078 is already newer than 1075 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: oid 371 version 1078 is already newer than 1076 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 371 v 1078 (ObjNum 1635 snap 0 seq_num 1635) dirty exists 2024-09-21T18:10:30.296 INFO:tasks.rados.rados.0.smithi160.stdout:3983: left oid 371 (ObjNum 1635 snap 0 seq_num 1635) 2024-09-21T18:10:30.297 INFO:tasks.rados.rados.0.smithi160.stdout:3981: finishing copy_from to smithi16030472-273 2024-09-21T18:10:30.297 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 273 v 772 (ObjNum 1150 snap 0 seq_num 1150) dirty exists 2024-09-21T18:10:30.337 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:30.331+0000 7f0b757eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 9 (PID: 26234) UID: 0 2024-09-21T18:10:30.345 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.345 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:30.394 INFO:tasks.rados.rados.0.smithi160.stdout:3990: finishing write tid 1 to smithi16030472-359 2024-09-21T18:10:30.394 INFO:tasks.rados.rados.0.smithi160.stdout:3990: finishing write tid 2 to smithi16030472-359 2024-09-21T18:10:30.394 INFO:tasks.rados.rados.0.smithi160.stdout:3990: finishing write tid 3 to smithi16030472-359 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3979: done (9 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3981: done (8 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3983: done (7 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3986: done (6 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3987: done (5 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3988: done (4 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3989: done (3 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3992: done (2 left) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3993: read oid 6 snap -1 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3993: expect (ObjNum 1571 snap 0 seq_num 1571) 2024-09-21T18:10:30.400 INFO:tasks.rados.rados.0.smithi160.stdout:3994: delete oid 483 current snap is 0 2024-09-21T18:10:30.408 INFO:tasks.rados.rados.0.smithi160.stdout:3993: expect (ObjNum 1571 snap 0 seq_num 1571) 2024-09-21T18:10:30.408 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:30.439 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:30.437+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:30.484 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.484 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.498 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:30.498 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:30.539 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:10:30.535+0000 7f49815fc640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 26232) UID: 0 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: finishing write tid 4 to smithi16030472-359 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: finishing write tid 5 to smithi16030472-359 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: oid 359 updating version 0 to 1112 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: oid 359 version 1112 is already newer than 1110 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: oid 359 updating version 1112 to 1113 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: oid 359 version 1113 is already newer than 1111 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 359 v 1113 (ObjNum 1637 snap 0 seq_num 1637) dirty exists 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3990: left oid 359 (ObjNum 1637 snap 0 seq_num 1637) 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:3985: finishing copy_from to smithi16030472-76 2024-09-21T18:10:30.543 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 76 v 1057 (ObjNum 1529 snap 0 seq_num 1529) dirty exists 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3985: done (3 left) 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3990: done (2 left) 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3993: done (1 left) 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3994: done (0 left) 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3995: cache_flush oid 39 snap -1 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3996: read oid 377 snap -1 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3996: expect deleted 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3997: cache_flush oid 179 snap -1 2024-09-21T18:10:30.605 INFO:tasks.rados.rados.0.smithi160.stdout:3998: delete oid 235 current snap is 0 2024-09-21T18:10:30.626 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.626 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.635 INFO:tasks.rados.rados.0.smithi160.stdout:3997: got (0) Success 2024-09-21T18:10:30.635 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 179 v 0 (ObjNum 334 snap 0 seq_num 334) dirty exists 2024-09-21T18:10:30.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:30.641 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:30.635+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:30.705 INFO:tasks.rados.rados.0.smithi160.stdout:3995: got (0) Success 2024-09-21T18:10:30.705 INFO:tasks.rados.rados.0.smithi160.stdout:update_object_version oid 39 v 0 (ObjNum 1301 snap 0 seq_num 1301) dirty exists 2024-09-21T18:10:30.741 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:30.737+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:30.763 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.763 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.777 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:30.809 INFO:tasks.rados.rados.0.smithi160.stdout:3995: done (3 left) 2024-09-21T18:10:30.809 INFO:tasks.rados.rados.0.smithi160.stdout:3996: done (2 left) 2024-09-21T18:10:30.809 INFO:tasks.rados.rados.0.smithi160.stdout:3997: done (1 left) 2024-09-21T18:10:30.810 INFO:tasks.rados.rados.0.smithi160.stdout:3998: done (0 left) 2024-09-21T18:10:30.810 INFO:tasks.rados.rados.0.smithi160.stdout:3999: read oid 450 snap -1 2024-09-21T18:10:30.810 INFO:tasks.rados.rados.0.smithi160.stdout:3999: expect (ObjNum 449 snap 0 seq_num 449) 2024-09-21T18:10:30.810 INFO:tasks.rados.rados.0.smithi160.stdout: waiting on 1 2024-09-21T18:10:30.842 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:30.837+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:30.899 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:30.899 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:30.913 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:30.957 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:30.956+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:31.044 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:10:31.041+0000 7f8c456a3640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 12 (PID: 26366) UID: 0 2024-09-21T18:10:31.079 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.080 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.094 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:31.145 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:31.144+0000 7fb26e1eb640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 11 (PID: 26588) UID: 0 2024-09-21T18:10:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:10:31.219 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.220 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T18:10:31.246 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:10:31.239+0000 7fd7a9ade640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 26230) UID: 0 2024-09-21T18:10:31.295 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:10:31.293+0000 7f9bb588f640 -1 osd.10 130 handle_osd_map injecting map crc failure 2024-09-21T18:10:31.327 INFO:tasks.rados.rados.0.smithi160.stdout:3999: expect (ObjNum 449 snap 0 seq_num 449) 2024-09-21T18:10:31.346 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:10:31.345+0000 7f09587b7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 26360) UID: 0 2024-09-21T18:10:31.348 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 2340889 2024-09-21T18:10:31.369 INFO:tasks.rados.rados.0.smithi160.stdout:seeking to 0 2024-09-21T18:10:31.406 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.407 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T18:10:31.448 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:31.445+0000 7fca5df1f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 26341) UID: 0 2024-09-21T18:10:31.456 INFO:tasks.rados.rados.0.smithi160.stdout:3999: done (0 left) 2024-09-21T18:10:31.465 INFO:tasks.rados.rados.0.smithi160.stderr:0 errors. 2024-09-21T18:10:31.465 INFO:tasks.rados.rados.0.smithi160.stderr: 2024-09-21T18:10:31.479 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T18:10:31.487 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T18:10:31.589 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.589 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T18:10:31.730 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.730 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T18:10:31.872 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:31.872 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:31.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T18:10:32.012 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.013 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T18:10:32.154 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.155 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_ops_in_flight 2024-09-21T18:10:32.292 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.292 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_blocked_ops 2024-09-21T18:10:32.434 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.434 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.8.asok dump_historic_ops 2024-09-21T18:10:32.576 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.577 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_ops_in_flight 2024-09-21T18:10:32.712 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.712 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_blocked_ops 2024-09-21T18:10:32.854 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:32.855 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:32.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.12.asok dump_historic_ops 2024-09-21T18:10:33.000 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.000 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T18:10:33.152 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.152 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.167 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T18:10:33.230 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:10:33.228+0000 7f6a3cca9640 -1 osd.3 131 handle_osd_map injecting map crc failure 2024-09-21T18:10:33.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T18:10:33.235 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:33.232+0000 7fb25ec0a640 -1 osd.11 131 handle_osd_map injecting map crc failure 2024-09-21T18:10:33.296 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.296 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.311 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T18:10:33.431 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.432 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.446 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T18:10:33.575 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.575 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.591 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T18:10:33.716 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.716 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.731 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T18:10:33.859 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:33.859 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-09-21T18:10:33.875 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_ops_in_flight 2024-09-21T18:10:33.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T18:10:34.000 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.000 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.015 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_blocked_ops 2024-09-21T18:10:34.138 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:10:34.137+0000 7fca4e93e640 -1 osd.2 132 handle_osd_map injecting map crc failure 2024-09-21T18:10:34.140 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.140 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.155 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.9.asok dump_historic_ops 2024-09-21T18:10:34.285 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.285 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.301 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_ops_in_flight 2024-09-21T18:10:34.428 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.428 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.442 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_blocked_ops 2024-09-21T18:10:34.563 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.563 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.577 DEBUG:teuthology.orchestra.run.smithi040:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.13.asok dump_historic_ops 2024-09-21T18:10:34.702 INFO:teuthology.orchestra.run.smithi040.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.702 INFO:teuthology.orchestra.run.smithi040.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.716 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T18:10:34.850 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.850 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T18:10:34.866 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-09-21T18:10:34.866 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:34.861+0000 7ff40b999640 -1 osd.4 133 handle_osd_map injecting map crc failure 2024-09-21T18:10:34.984 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:34.984 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:34.998 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T18:10:35.114 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.115 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.129 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T18:10:35.255 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.255 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.269 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T18:10:35.391 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.391 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.407 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T18:10:35.530 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.530 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.543 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_ops_in_flight 2024-09-21T18:10:35.666 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.666 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.680 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_blocked_ops 2024-09-21T18:10:35.757 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:10:35.751+0000 7f0b6a212640 -1 osd.9 133 handle_osd_map injecting map crc failure 2024-09-21T18:10:35.799 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.799 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.813 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.10.asok dump_historic_ops 2024-09-21T18:10:35.938 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:35.939 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:35.951 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_ops_in_flight 2024-09-21T18:10:36.074 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.074 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.088 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_blocked_ops 2024-09-21T18:10:36.211 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.211 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.226 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.14.asok dump_historic_ops 2024-09-21T18:10:36.343 INFO:teuthology.orchestra.run.smithi070.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.345 INFO:teuthology.orchestra.run.smithi070.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.359 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T18:10:36.491 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.491 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.505 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T18:10:36.626 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.626 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.640 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T18:10:36.758 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.758 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.774 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_ops_in_flight 2024-09-21T18:10:36.898 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:36.898 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:36.914 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_blocked_ops 2024-09-21T18:10:37.034 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:37.035 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:37.048 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.11.asok dump_historic_ops 2024-09-21T18:10:37.171 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:37.171 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:37.185 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_ops_in_flight 2024-09-21T18:10:37.304 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:37.304 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:37.318 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_blocked_ops 2024-09-21T18:10:37.436 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:37.436 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:37.450 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.15.asok dump_historic_ops 2024-09-21T18:10:37.571 INFO:teuthology.orchestra.run.smithi160.stdout:ERROR: (22) Invalid argument 2024-09-21T18:10:37.571 INFO:teuthology.orchestra.run.smithi160.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T18:10:38.479 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:10:38.477+0000 7f3ce267c640 -1 osd.6 134 handle_osd_map injecting map crc failure 2024-09-21T18:10:38.499 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:10:38.497+0000 7ff409995640 -1 osd.4 134 handle_osd_map injecting map crc failure 2024-09-21T18:10:49.351 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:10:49.347+0000 7f440780b640 -1 osd.13 135 handle_osd_map injecting map crc failure 2024-09-21T18:10:49.351 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:10:49.348+0000 7fb25ec0a640 -1 osd.11 135 handle_osd_map injecting map crc failure 2024-09-21T18:11:24.540 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-21T18:11:24.540 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-21T18:11:24.541 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-21T18:11:24.544 INFO:tasks.ceph.osd.7:Started 2024-09-21T18:11:24.544 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:no valid command found; 10 closest matches: 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:0 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:1 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:2 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:abort 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:assert 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:config diff 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:config diff get 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:config get 2024-09-21T18:11:24.666 INFO:teuthology.orchestra.run.smithi160.stderr:config help [] 2024-09-21T18:11:24.667 INFO:teuthology.orchestra.run.smithi160.stderr:config set ... 2024-09-21T18:11:24.667 INFO:teuthology.orchestra.run.smithi160.stderr:admin_socket: invalid command 2024-09-21T18:11:24.668 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T18:11:24.669 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-21T18:11:25.867 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:25.864+0000 7fc0471d87c0 -1 Falling back to public interface 2024-09-21T18:11:26.993 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:11:26.988+0000 7fb924fb6640 -1 osd.8 139 handle_osd_map injecting map crc failure 2024-09-21T18:11:26.995 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:11:26.992+0000 7fb924fb6640 -1 osd.8 139 handle_osd_map injecting map crc failure 2024-09-21T18:11:29.669 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:11:29.732 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:29.732+0000 7fc0471d87c0 -1 osd.7 61 log_to_monitors true 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:no valid command found; 10 closest matches: 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:0 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:1 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:2 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:abort 2024-09-21T18:11:29.798 INFO:teuthology.orchestra.run.smithi160.stderr:assert 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs debug_inject_read_zeros 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs files list 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:bluefs stats 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore allocator dump block 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:bluestore allocator fragmentation block 2024-09-21T18:11:29.799 INFO:teuthology.orchestra.run.smithi160.stderr:admin_socket: invalid command 2024-09-21T18:11:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T18:11:29.801 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-21T18:11:30.395 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:11:30.392+0000 7f8c3a0ca640 -1 osd.12 140 handle_osd_map injecting map crc failure 2024-09-21T18:11:30.395 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:11:30.391+0000 7f7283507640 -1 osd.14 140 handle_osd_map injecting map crc failure 2024-09-21T18:11:30.400 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.396+0000 7fc03abbf640 -1 osd.7 61 handle_osd_map injecting map crc failure 2024-09-21T18:11:30.514 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 62, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 63, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 64, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 65, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 66, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 67, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 68, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 69, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 70, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 71, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 72, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 73, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 74, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 75, got 0 bytes 2024-09-21T18:11:30.515 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 76, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 77, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 78, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 79, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 80, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 81, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 82, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 83, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 84, got 0 bytes 2024-09-21T18:11:30.516 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.512+0000 7fc029f80640 -1 osd.7 98 failed to load OSD map for epoch 85, got 0 bytes 2024-09-21T18:11:30.528 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.524+0000 7fc03abbf640 -1 osd.7 98 handle_osd_map injecting map crc failure 2024-09-21T18:11:30.660 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:30.656+0000 7fc03efbb640 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T18:11:31.281 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:11:31.276+0000 7f8c3a0ca640 -1 osd.12 141 handle_osd_map injecting map crc failure 2024-09-21T18:11:31.859 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:11:31.855+0000 7f7283507640 -1 osd.14 141 handle_osd_map injecting map crc failure 2024-09-21T18:11:31.970 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:11:31.967+0000 7f0b6a212640 -1 osd.9 141 handle_osd_map injecting map crc failure 2024-09-21T18:11:32.281 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:11:32.276+0000 7ff40b999640 -1 osd.4 141 handle_osd_map injecting map crc failure 2024-09-21T18:11:32.412 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:11:32.408+0000 7f6a3cca9640 -1 osd.3 142 handle_osd_map injecting map crc failure 2024-09-21T18:11:34.802 DEBUG:teuthology.orchestra.run.smithi160:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T18:11:34.935 INFO:teuthology.orchestra.run.smithi160.stdout:{ 2024-09-21T18:11:34.935 INFO:teuthology.orchestra.run.smithi160.stdout: "ops": [], 2024-09-21T18:11:34.935 INFO:teuthology.orchestra.run.smithi160.stdout: "num_ops": 0 2024-09-21T18:11:34.935 INFO:teuthology.orchestra.run.smithi160.stdout:} 2024-09-21T18:11:34.949 INFO:tasks.thrashosds.thrasher:choose_action: min_in 2 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T18:11:34.949 INFO:tasks.thrashosds.thrasher:check thrash_hosts: in_osds > minin 2024-09-21T18:11:34.949 INFO:tasks.thrashosds.thrasher:Checking the number of in OSDs in host ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:11:34.950 INFO:tasks.thrashosds.thrasher:Removing all OSDs in host ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:11:34.950 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-21T18:11:34.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-21T18:11:35.287 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:35.284+0000 7f1072a45640 -1 mon.a@0(leader).osd e144 definitely_dead 0 2024-09-21T18:11:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.2. 2024-09-21T18:11:35.907 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-21T18:11:35.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 6 2024-09-21T18:11:36.196 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:11:36.192+0000 7fb924fb6640 -1 osd.8 144 handle_osd_map injecting map crc failure 2024-09-21T18:11:36.204 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:11:36.199+0000 7f94d9542640 -1 osd.15 144 handle_osd_map injecting map crc failure 2024-09-21T18:11:36.236 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:36.232+0000 7f1072a45640 -1 mon.a@0(leader).osd e145 definitely_dead 0 2024-09-21T18:11:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.6. 2024-09-21T18:11:36.901 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:11:36.895+0000 7f9bb588f640 -1 osd.10 145 handle_osd_map injecting map crc failure 2024-09-21T18:11:36.919 INFO:tasks.thrashosds.thrasher:Removing osd 10, in_osds are: [0, 1, 3, 4, 5, 8, 9, 10, 11, 12, 13, 14, 15] 2024-09-21T18:11:36.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 10 2024-09-21T18:11:37.252 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:37.248+0000 7f1072a45640 -1 mon.a@0(leader).osd e146 definitely_dead 0 2024-09-21T18:11:37.902 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.10. 2024-09-21T18:11:37.927 INFO:tasks.thrashosds.thrasher:Removing osd 14, in_osds are: [0, 1, 3, 4, 5, 8, 9, 11, 12, 13, 14, 15] 2024-09-21T18:11:37.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 14 2024-09-21T18:11:38.255 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:38.252+0000 7f1072a45640 -1 mon.a@0(leader).osd e147 definitely_dead 0 2024-09-21T18:11:38.911 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.14. 2024-09-21T18:11:38.920 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:11:38.915+0000 7fb25ec0a640 -1 osd.11 147 handle_osd_map injecting map crc failure 2024-09-21T18:11:39.384 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:11:39.379+0000 7f440780b640 -1 osd.13 147 handle_osd_map injecting map crc failure 2024-09-21T18:11:40.298 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:11:40.295+0000 7f0b6a212640 -1 osd.9 148 handle_osd_map injecting map crc failure 2024-09-21T18:11:40.298 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:11:40.292+0000 7f8c3a0ca640 -1 osd.12 148 handle_osd_map injecting map crc failure 2024-09-21T18:11:43.940 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T18:11:43.940 INFO:tasks.thrashosds.thrasher:Adding osd 10 2024-09-21T18:11:43.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 10 2024-09-21T18:11:44.276 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:44.272+0000 7f1072a45640 -1 mon.a@0(leader).osd e149 definitely_dead 0 2024-09-21T18:11:44.328 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.10. 2024-09-21T18:11:44.329 INFO:tasks.ceph.osd.0.smithi029.stderr:2024-09-21T18:11:44.324+0000 7f094f1e2640 -1 osd.0 149 handle_osd_map injecting map crc failure 2024-09-21T18:11:44.348 INFO:tasks.thrashosds.thrasher:Added osd 10 2024-09-21T18:11:44.348 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T18:11:44.348 INFO:tasks.thrashosds.thrasher:Adding osd 14 2024-09-21T18:11:44.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 14 2024-09-21T18:11:44.669 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:44.664+0000 7f1072a45640 -1 mon.a@0(leader).osd e150 definitely_dead 0 2024-09-21T18:11:45.345 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.14. 2024-09-21T18:11:45.354 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:11:45.351+0000 7f9bb588f640 -1 osd.10 150 handle_osd_map injecting map crc failure 2024-09-21T18:11:45.374 INFO:tasks.thrashosds.thrasher:Added osd 14 2024-09-21T18:11:45.374 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T18:11:45.374 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-21T18:11:45.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-21T18:11:45.709 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:45.704+0000 7f1072a45640 -1 mon.a@0(leader).osd e151 definitely_dead 0 2024-09-21T18:11:46.348 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.2. 2024-09-21T18:11:46.375 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-21T18:11:46.375 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T18:11:46.375 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-21T18:11:46.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 6 2024-09-21T18:11:46.695 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:46.692+0000 7f1072a45640 -1 mon.a@0(leader).osd e152 definitely_dead 0 2024-09-21T18:11:47.364 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.6. 2024-09-21T18:11:47.370 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:11:47.367+0000 7fca50942640 -1 osd.2 152 handle_osd_map injecting map crc failure 2024-09-21T18:11:47.389 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-21T18:11:47.389 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T18:11:47.389 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-21T18:11:47.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-09-21T18:11:47.450 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:47.447+0000 7fc038bbb640 -1 osd.7 152 handle_osd_map injecting map crc failure 2024-09-21T18:11:47.451 INFO:tasks.ceph.osd.9.smithi040.stderr:2024-09-21T18:11:47.447+0000 7f0b6a212640 -1 osd.9 152 handle_osd_map injecting map crc failure 2024-09-21T18:11:47.729 INFO:tasks.ceph.mon.a.smithi029.stderr:2024-09-21T18:11:47.724+0000 7f1072a45640 -1 mon.a@0(leader).osd e153 definitely_dead 0 2024-09-21T18:11:48.373 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.7. 2024-09-21T18:11:48.374 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:11:48.371+0000 7f440980f640 -1 osd.13 153 handle_osd_map injecting map crc failure 2024-09-21T18:11:48.396 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-21T18:11:48.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T18:11:48.540 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a: {} 2024-09-21T18:11:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:11:48.554 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b: {} 2024-09-21T18:11:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:11:48.568 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c: {} 2024-09-21T18:11:48.568 INFO:teuthology.orchestra.run.smithi029.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T18:11:48.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T18:11:49.382 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T18:11:49.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T18:11:50.044 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:11:50.039+0000 7fb25ec0a640 -1 osd.11 154 handle_osd_map injecting map crc failure 2024-09-21T18:11:50.392 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T18:11:50.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T18:11:50.987 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:11:50.984+0000 7f8c3a0ca640 -1 osd.12 155 handle_osd_map injecting map crc failure 2024-09-21T18:11:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T18:11:51.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-21T18:11:52.152 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:11:52.147+0000 7fd79a4fd640 -1 osd.1 155 handle_osd_map injecting map crc failure 2024-09-21T18:11:52.403 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T18:11:52.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 8 1 2024-09-21T18:11:52.843 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:11:52.839+0000 7fc038bbb640 -1 osd.7 157 handle_osd_map injecting map crc failure 2024-09-21T18:11:52.843 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:11:52.839+0000 7f9bb588f640 -1 osd.10 156 handle_osd_map injecting map crc failure 2024-09-21T18:11:52.843 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:11:52.839+0000 7f94d9542640 -1 osd.15 155 handle_osd_map injecting map crc failure 2024-09-21T18:11:52.850 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:11:52.847+0000 7f94db546640 -1 osd.15 157 handle_osd_map injecting map crc failure 2024-09-21T18:11:52.962 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:11:52.960+0000 7ff409995640 -1 osd.4 156 handle_osd_map injecting map crc failure 2024-09-21T18:11:53.421 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.8 to 1 (10000) 2024-09-21T18:11:53.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 8 1 2024-09-21T18:11:54.424 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.8 primary-affinity to 1 (8655362) 2024-09-21T18:11:54.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 12 1 2024-09-21T18:11:54.616 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:11:54.612+0000 7ff409995640 -1 osd.4 159 handle_osd_map injecting map crc failure 2024-09-21T18:11:54.875 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:11:54.872+0000 7f8c3a0ca640 -1 osd.12 159 handle_osd_map injecting map crc failure 2024-09-21T18:11:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.12 to 1 (10000) 2024-09-21T18:11:55.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 12 1 2024-09-21T18:11:55.653 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:11:55.647+0000 7fd79a4fd640 -1 osd.1 159 handle_osd_map injecting map crc failure 2024-09-21T18:11:55.655 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:11:55.650+0000 7fd79a4fd640 -1 osd.1 160 handle_osd_map injecting map crc failure 2024-09-21T18:11:55.717 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:11:55.712+0000 7ff409995640 -1 osd.4 160 handle_osd_map injecting map crc failure 2024-09-21T18:11:55.763 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:11:55.759+0000 7f7283507640 -1 osd.14 160 handle_osd_map injecting map crc failure 2024-09-21T18:11:56.263 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:11:56.259+0000 7f6a3cca9640 -1 osd.3 160 handle_osd_map injecting map crc failure 2024-09-21T18:11:56.442 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.12 primary-affinity to 1 (8655362) 2024-09-21T18:11:56.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T18:11:57.459 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T18:11:57.459 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:11:57.456+0000 7ff40b999640 -1 osd.4 161 handle_osd_map injecting map crc failure 2024-09-21T18:11:57.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T18:11:57.490 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:11:57.487+0000 7f3ce267c640 -1 osd.6 161 handle_osd_map injecting map crc failure 2024-09-21T18:11:57.952 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:11:57.946+0000 7fd79a4fd640 -1 osd.1 162 handle_osd_map injecting map crc failure 2024-09-21T18:11:58.461 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T18:11:58.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T18:11:59.476 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T18:11:59.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T18:11:59.541 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:11:59.539+0000 7f6a3cca9640 -1 osd.3 164 handle_osd_map injecting map crc failure 2024-09-21T18:12:00.189 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:12:00.182+0000 7f4976023640 -1 osd.5 164 handle_osd_map injecting map crc failure 2024-09-21T18:12:00.254 INFO:tasks.ceph.osd.1.smithi040.stderr:2024-09-21T18:12:00.250+0000 7fd79a4fd640 -1 osd.1 163 handle_osd_map injecting map crc failure 2024-09-21T18:12:00.478 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T18:12:00.478 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:12:00.475+0000 7fb260c0e640 -1 osd.11 165 handle_osd_map injecting map crc failure 2024-09-21T18:12:00.479 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:12:00.475+0000 7f6a3ecad640 -1 osd.3 165 handle_osd_map injecting map crc failure 2024-09-21T18:12:00.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 9 1 2024-09-21T18:12:01.310 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:12:01.306+0000 7f4978027640 -1 osd.5 165 handle_osd_map injecting map crc failure 2024-09-21T18:12:01.487 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.9 to 1 (10000) 2024-09-21T18:12:01.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 9 1 2024-09-21T18:12:01.702 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:12:01.699+0000 7f3ce267c640 -1 osd.6 165 handle_osd_map injecting map crc failure 2024-09-21T18:12:01.743 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:12:01.739+0000 7fb25ec0a640 -1 osd.11 166 handle_osd_map injecting map crc failure 2024-09-21T18:12:02.497 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.9 primary-affinity to 1 (8655362) 2024-09-21T18:12:02.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 13 1 2024-09-21T18:12:02.719 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:12:02.715+0000 7f7283507640 -1 osd.14 167 handle_osd_map injecting map crc failure 2024-09-21T18:12:03.514 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.13 to 1 (10000) 2024-09-21T18:12:03.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 13 1 2024-09-21T18:12:04.516 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.13 primary-affinity to 1 (8655362) 2024-09-21T18:12:04.517 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:12:04.515+0000 7f9bb7893640 -1 osd.10 169 handle_osd_map injecting map crc failure 2024-09-21T18:12:04.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T18:12:04.866 INFO:tasks.ceph.osd.14.smithi070.stderr:2024-09-21T18:12:04.863+0000 7f7283507640 -1 osd.14 168 handle_osd_map injecting map crc failure 2024-09-21T18:12:05.526 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T18:12:05.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-21T18:12:05.904 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:12:05.899+0000 7fc038bbb640 -1 osd.7 168 handle_osd_map injecting map crc failure 2024-09-21T18:12:06.016 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:12:06.014+0000 7f4976023640 -1 osd.5 170 handle_osd_map injecting map crc failure 2024-09-21T18:12:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T18:12:06.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T18:12:07.546 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T18:12:07.547 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:12:07.543+0000 7f9bb7893640 -1 osd.10 172 handle_osd_map injecting map crc failure 2024-09-21T18:12:07.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-21T18:12:08.555 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T18:12:08.556 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:12:08.551+0000 7f9bb7893640 -1 osd.10 173 handle_osd_map injecting map crc failure 2024-09-21T18:12:08.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 10 1 2024-09-21T18:12:09.491 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:12:09.488+0000 7f8c3a0ca640 -1 osd.12 173 handle_osd_map injecting map crc failure 2024-09-21T18:12:09.565 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.10 to 1 (10000) 2024-09-21T18:12:09.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 10 1 2024-09-21T18:12:09.757 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:12:09.755+0000 7fca4e93e640 -1 osd.2 173 handle_osd_map injecting map crc failure 2024-09-21T18:12:10.578 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.10 primary-affinity to 1 (8655362) 2024-09-21T18:12:10.578 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:12:10.572+0000 7ff40b999640 -1 osd.4 175 handle_osd_map injecting map crc failure 2024-09-21T18:12:10.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 14 1 2024-09-21T18:12:11.506 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:12:11.502+0000 7f440780b640 -1 osd.13 175 handle_osd_map injecting map crc failure 2024-09-21T18:12:11.584 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.14 to 1 (10000) 2024-09-21T18:12:11.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 14 1 2024-09-21T18:12:12.419 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:12:12.416+0000 7f8c3a0ca640 -1 osd.12 175 handle_osd_map injecting map crc failure 2024-09-21T18:12:12.593 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.14 primary-affinity to 1 (8655362) 2024-09-21T18:12:12.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T18:12:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T18:12:13.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-21T18:12:14.902 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T18:12:14.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 11 1 2024-09-21T18:12:14.959 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:12:14.955+0000 7fca4e93e640 -1 osd.2 179 handle_osd_map injecting map crc failure 2024-09-21T18:12:15.508 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:12:15.503+0000 7f3ce267c640 -1 osd.6 179 handle_osd_map injecting map crc failure 2024-09-21T18:12:15.508 INFO:tasks.ceph.osd.3.smithi160.stderr:2024-09-21T18:12:15.507+0000 7f6a3cca9640 -1 osd.3 179 handle_osd_map injecting map crc failure 2024-09-21T18:12:15.911 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.11 to 1 (10000) 2024-09-21T18:12:15.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 11 1 2024-09-21T18:12:16.913 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.11 primary-affinity to 1 (8655362) 2024-09-21T18:12:16.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 15 1 2024-09-21T18:12:17.672 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:12:17.667+0000 7f94d9542640 -1 osd.15 181 handle_osd_map injecting map crc failure 2024-09-21T18:12:17.925 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.15 to 1 (10000) 2024-09-21T18:12:17.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 15 1 2024-09-21T18:12:17.993 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:12:17.991+0000 7f3ce267c640 -1 osd.6 181 handle_osd_map injecting map crc failure 2024-09-21T18:12:18.933 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.15 primary-affinity to 1 (8655362) 2024-09-21T18:12:18.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T18:12:19.305 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:12:19.300+0000 7fb924fb6640 -1 osd.8 182 handle_osd_map injecting map crc failure 2024-09-21T18:12:19.524 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:12:19.518+0000 7f4976023640 -1 osd.5 182 handle_osd_map injecting map crc failure 2024-09-21T18:12:19.524 INFO:tasks.ceph.osd.15.smithi160.stderr:2024-09-21T18:12:19.519+0000 7f94d9542640 -1 osd.15 183 handle_osd_map injecting map crc failure 2024-09-21T18:12:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T18:12:19.943 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:12:19.939+0000 7fb260c0e640 -1 osd.11 184 handle_osd_map injecting map crc failure 2024-09-21T18:12:19.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-21T18:12:20.669 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:12:20.664+0000 7f8c3a0ca640 -1 osd.12 184 handle_osd_map injecting map crc failure 2024-09-21T18:12:20.953 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T18:12:20.980 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T18:12:20.980 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T18:12:20.980 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T18:12:20.980 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T18:12:20.980 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T18:12:20.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:12:21.224 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:12:21.222+0000 7f440780b640 -1 osd.13 185 handle_osd_map injecting map crc failure 2024-09-21T18:12:21.245 INFO:tasks.ceph.osd.11.smithi160.stderr:2024-09-21T18:12:21.243+0000 7fb25ec0a640 -1 osd.11 185 handle_osd_map injecting map crc failure 2024-09-21T18:12:21.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:21.300 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":186,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:12:20.951078+0000","last_up_change":"2024-09-21T18:11:31.261580+0000","last_in_change":"2024-09-21T18:11:48.364704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-21T18:06:59.604621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-21T18:07:01.499947+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":3600,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":61,"up_from":142,"up_thru":0,"down_at":62,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":655055502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":655055502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":655055502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":655055502}]},"public_addr":"172.21.15.160:6801/655055502","cluster_addr":"172.21.15.160:6802/655055502","heartbeat_back_addr":"172.21.15.160:6804/655055502","heartbeat_front_addr":"172.21.15.160:6803/655055502","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:54.549709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:37.644910+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.319783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:35.985784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:34.786377+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:23.492947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:49.682678+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:30.774098+0000","dead_epoch":62},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:51.196099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.981028+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:10.964912+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:56.102384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:48.713280+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:26.051863+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:07.753636+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:05.599605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:12:21.318 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T18:12:21.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:12:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:21.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":186,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:12:20.951078+0000","last_up_change":"2024-09-21T18:11:31.261580+0000","last_in_change":"2024-09-21T18:11:48.364704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-21T18:06:59.604621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-21T18:07:01.499947+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":3600,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":61,"up_from":142,"up_thru":0,"down_at":62,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":655055502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":655055502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":655055502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":655055502}]},"public_addr":"172.21.15.160:6801/655055502","cluster_addr":"172.21.15.160:6802/655055502","heartbeat_back_addr":"172.21.15.160:6804/655055502","heartbeat_front_addr":"172.21.15.160:6803/655055502","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:54.549709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:37.644910+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.319783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:35.985784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:34.786377+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:23.492947+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:49.682678+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:30.774098+0000","dead_epoch":62},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:51.196099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.981028+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:10.964912+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:56.102384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:48.713280+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:26.051863+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:07.753636+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:05.599605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:12:21.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T18:12:21.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T18:12:21.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T18:12:21.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T18:12:21.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T18:12:21.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T18:12:21.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T18:12:21.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T18:12:21.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 flush_pg_stats 2024-09-21T18:12:21.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 flush_pg_stats 2024-09-21T18:12:21.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 flush_pg_stats 2024-09-21T18:12:21.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 flush_pg_stats 2024-09-21T18:12:21.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 flush_pg_stats 2024-09-21T18:12:21.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 flush_pg_stats 2024-09-21T18:12:21.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 flush_pg_stats 2024-09-21T18:12:21.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 flush_pg_stats 2024-09-21T18:12:22.065 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:22.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-21T18:12:22.070 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:22.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-21T18:12:22.071 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:22.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-21T18:12:22.089 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:22.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T18:12:22.121 INFO:teuthology.orchestra.run.smithi029.stdout:98784247879 2024-09-21T18:12:22.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T18:12:22.132 INFO:teuthology.orchestra.run.smithi029.stdout:609885356044 2024-09-21T18:12:22.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T18:12:22.157 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:22.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-21T18:12:22.159 INFO:teuthology.orchestra.run.smithi029.stdout:98784247880 2024-09-21T18:12:22.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-21T18:12:22.168 INFO:teuthology.orchestra.run.smithi029.stdout:90194313288 2024-09-21T18:12:22.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T18:12:22.176 INFO:teuthology.orchestra.run.smithi029.stdout:98784247878 2024-09-21T18:12:22.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-21T18:12:22.181 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:22.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-21T18:12:22.211 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:22.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T18:12:22.233 INFO:teuthology.orchestra.run.smithi029.stdout:90194313289 2024-09-21T18:12:22.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T18:12:22.244 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:22.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T18:12:22.256 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:22.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T18:12:22.311 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:22.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-21T18:12:23.006 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.067 INFO:teuthology.orchestra.run.smithi029.stdout:98784247879 2024-09-21T18:12:23.072 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280582 for osd.11 2024-09-21T18:12:23.080 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.114 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247880 got 98784247879 for osd.15 2024-09-21T18:12:23.130 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280582 for osd.8 2024-09-21T18:12:23.236 INFO:teuthology.orchestra.run.smithi029.stdout:609885356043 2024-09-21T18:12:23.238 INFO:teuthology.orchestra.run.smithi029.stdout:94489280581 2024-09-21T18:12:23.252 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.271 INFO:teuthology.orchestra.run.smithi029.stdout:98784247878 2024-09-21T18:12:23.275 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280581 for osd.9 2024-09-21T18:12:23.287 INFO:tasks.ceph.ceph_manager.ceph:need seq 609885356044 got 609885356043 for osd.7 2024-09-21T18:12:23.287 INFO:teuthology.orchestra.run.smithi029.stdout:90194313287 2024-09-21T18:12:23.294 INFO:teuthology.orchestra.run.smithi029.stdout:94489280581 2024-09-21T18:12:23.316 INFO:teuthology.orchestra.run.smithi029.stdout:90194313288 2024-09-21T18:12:23.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280582 for osd.5 2024-09-21T18:12:23.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247879 got 98784247878 for osd.1 2024-09-21T18:12:23.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280582 for osd.14 2024-09-21T18:12:23.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280581 for osd.10 2024-09-21T18:12:23.334 INFO:teuthology.orchestra.run.smithi029.stdout:94489280581 2024-09-21T18:12:23.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313288 got 90194313287 for osd.4 2024-09-21T18:12:23.336 INFO:teuthology.orchestra.run.smithi029.stdout:94489280581 2024-09-21T18:12:23.339 INFO:teuthology.orchestra.run.smithi029.stdout:98784247877 2024-09-21T18:12:23.351 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313289 got 90194313288 for osd.6 2024-09-21T18:12:23.358 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280581 for osd.3 2024-09-21T18:12:23.362 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247878 got 98784247877 for osd.13 2024-09-21T18:12:23.363 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280581 for osd.2 2024-09-21T18:12:23.368 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280582 for osd.0 2024-09-21T18:12:23.392 INFO:teuthology.orchestra.run.smithi029.stdout:94489280582 2024-09-21T18:12:23.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280582 for osd.12 2024-09-21T18:12:24.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.11 2024-09-21T18:12:24.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.15 2024-09-21T18:12:24.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.8 2024-09-21T18:12:24.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.9 2024-09-21T18:12:24.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T18:12:24.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T18:12:24.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T18:12:24.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.14 2024-09-21T18:12:24.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.10 2024-09-21T18:12:24.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T18:12:24.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T18:12:24.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T18:12:24.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.13 2024-09-21T18:12:24.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T18:12:24.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T18:12:24.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.12 2024-09-21T18:12:24.496 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:24.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280583 for osd.11 2024-09-21T18:12:24.545 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:24.830 INFO:teuthology.orchestra.run.smithi029.stdout:98784247880 2024-09-21T18:12:24.846 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:24.915 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280584 for osd.8 2024-09-21T18:12:24.915 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:24.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247880 got 98784247880 for osd.15 2024-09-21T18:12:24.935 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.041 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:25.099 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280583 for osd.9 2024-09-21T18:12:25.099 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.185 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:25.212 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:25.224 INFO:teuthology.orchestra.run.smithi029.stdout:90194313289 2024-09-21T18:12:25.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280583 for osd.10 2024-09-21T18:12:25.225 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280584 for osd.5 2024-09-21T18:12:25.246 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.264 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313289 got 90194313289 for osd.6 2024-09-21T18:12:25.264 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.303 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:25.312 INFO:teuthology.orchestra.run.smithi029.stdout:98784247880 2024-09-21T18:12:25.313 INFO:teuthology.orchestra.run.smithi029.stdout:609885356044 2024-09-21T18:12:25.328 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280582 got 94489280583 for osd.2 2024-09-21T18:12:25.328 INFO:teuthology.orchestra.run.smithi029.stdout:90194313288 2024-09-21T18:12:25.328 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.342 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247879 got 98784247880 for osd.1 2024-09-21T18:12:25.342 INFO:teuthology.orchestra.run.smithi029.stdout:98784247879 2024-09-21T18:12:25.342 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.353 INFO:tasks.ceph.ceph_manager.ceph:need seq 609885356044 got 609885356044 for osd.7 2024-09-21T18:12:25.353 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313288 got 90194313288 for osd.4 2024-09-21T18:12:25.355 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.365 INFO:tasks.ceph.ceph_manager.ceph:need seq 98784247878 got 98784247879 for osd.13 2024-09-21T18:12:25.365 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.368 INFO:teuthology.orchestra.run.smithi029.stdout:94489280583 2024-09-21T18:12:25.368 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:25.388 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:25.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280584 for osd.14 2024-09-21T18:12:25.389 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280583 got 94489280583 for osd.3 2024-09-21T18:12:25.389 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.398 INFO:teuthology.orchestra.run.smithi029.stdout:94489280584 2024-09-21T18:12:25.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280584 for osd.12 2024-09-21T18:12:25.405 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.416 INFO:tasks.ceph.ceph_manager.ceph:need seq 94489280584 got 94489280584 for osd.0 2024-09-21T18:12:25.416 DEBUG:teuthology.parallel:result is None 2024-09-21T18:12:25.416 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T18:12:25.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:25.739 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:25.740 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:25.756 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-09-21T18:12:25.151568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2904684,"kb_used_data":2383844,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044756,"statfs":{"total":1546188226560,"available":1543213830144,"internally_reserved":0,"allocated":2441056256,"data_stored":2542871702,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.217711"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":482,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250982+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:21.250982+0000","last_peered":"2024-09-21T18:12:21.250982+0000","last_clean":"2024-09-21T18:12:21.250982+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:21.250982+0000","last_undegraded":"2024-09-21T18:12:21.250982+0000","last_fullsized":"2024-09-21T18:12:21.250982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":375,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251108+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:21.251108+0000","last_peered":"2024-09-21T18:12:21.251108+0000","last_clean":"2024-09-21T18:12:21.251108+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:21.251108+0000","last_undegraded":"2024-09-21T18:12:21.251108+0000","last_fullsized":"2024-09-21T18:12:21.251108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1235,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250979+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:21.250979+0000","last_peered":"2024-09-21T18:12:21.250979+0000","last_clean":"2024-09-21T18:12:21.250979+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:21.250979+0000","last_undegraded":"2024-09-21T18:12:21.250979+0000","last_fullsized":"2024-09-21T18:12:21.250979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363668+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:21.363668+0000","last_peered":"2024-09-21T18:12:21.363668+0000","last_clean":"2024-09-21T18:12:21.363668+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:21.363668+0000","last_undegraded":"2024-09-21T18:12:21.363668+0000","last_fullsized":"2024-09-21T18:12:21.363668+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1691,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957272+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:20.957272+0000","last_peered":"2024-09-21T18:12:20.957272+0000","last_clean":"2024-09-21T18:12:20.957272+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:20.957272+0000","last_undegraded":"2024-09-21T18:12:20.957272+0000","last_fullsized":"2024-09-21T18:12:20.957272+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:27.164579+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":339,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327449+0000","last_change":"2024-09-21T18:11:18.907283+0000","last_active":"2024-09-21T18:12:21.327449+0000","last_peered":"2024-09-21T18:12:21.327449+0000","last_clean":"2024-09-21T18:12:21.327449+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:21.327449+0000","last_undegraded":"2024-09-21T18:12:21.327449+0000","last_fullsized":"2024-09-21T18:12:21.327449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:18.907114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:18.907114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:29.818720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:22.461498+0000","last_change":"2024-09-21T18:12:06.352918+0000","last_active":"2024-09-21T18:12:22.461498+0000","last_peered":"2024-09-21T18:12:22.461498+0000","last_clean":"2024-09-21T18:12:22.461498+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:22.461498+0000","last_undegraded":"2024-09-21T18:12:22.461498+0000","last_fullsized":"2024-09-21T18:12:22.461498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":104,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:35.851886+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363675+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:21.363675+0000","last_peered":"2024-09-21T18:12:21.363675+0000","last_clean":"2024-09-21T18:12:21.363675+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:21.363675+0000","last_undegraded":"2024-09-21T18:12:21.363675+0000","last_fullsized":"2024-09-21T18:12:21.363675+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1740,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327443+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:21.327443+0000","last_peered":"2024-09-21T18:12:21.327443+0000","last_clean":"2024-09-21T18:12:21.327443+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:21.327443+0000","last_undegraded":"2024-09-21T18:12:21.327443+0000","last_fullsized":"2024-09-21T18:12:21.327443+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2678,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.427662+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:21.427662+0000","last_peered":"2024-09-21T18:12:21.427662+0000","last_clean":"2024-09-21T18:12:21.427662+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:21.427662+0000","last_undegraded":"2024-09-21T18:12:21.427662+0000","last_fullsized":"2024-09-21T18:12:21.427662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":342,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957263+0000","last_change":"2024-09-21T18:11:15.583692+0000","last_active":"2024-09-21T18:12:20.957263+0000","last_peered":"2024-09-21T18:12:20.957263+0000","last_clean":"2024-09-21T18:12:20.957263+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:20.957263+0000","last_undegraded":"2024-09-21T18:12:20.957263+0000","last_fullsized":"2024-09-21T18:12:20.957263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:15.583514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:15.583514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:31.683887+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2037,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251041+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:21.251041+0000","last_peered":"2024-09-21T18:12:21.251041+0000","last_clean":"2024-09-21T18:12:21.251041+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:21.251041+0000","last_undegraded":"2024-09-21T18:12:21.251041+0000","last_fullsized":"2024-09-21T18:12:21.251041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.060290+0000","last_change":"2024-09-21T18:12:01.314179+0000","last_active":"2024-09-21T18:12:21.060290+0000","last_peered":"2024-09-21T18:12:21.060290+0000","last_clean":"2024-09-21T18:12:21.060290+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:21.060290+0000","last_undegraded":"2024-09-21T18:12:21.060290+0000","last_fullsized":"2024-09-21T18:12:21.060290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:21.770766+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2774,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957285+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:20.957285+0000","last_peered":"2024-09-21T18:12:20.957285+0000","last_clean":"2024-09-21T18:12:20.957285+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:20.957285+0000","last_undegraded":"2024-09-21T18:12:20.957285+0000","last_fullsized":"2024-09-21T18:12:20.957285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":349,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.798316+0000","last_change":"2024-09-21T18:11:30.157639+0000","last_active":"2024-09-21T18:12:21.798316+0000","last_peered":"2024-09-21T18:12:21.798316+0000","last_clean":"2024-09-21T18:12:21.798316+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:21.798316+0000","last_undegraded":"2024-09-21T18:12:21.798316+0000","last_fullsized":"2024-09-21T18:12:21.798316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:30.157362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:30.157362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:37.787991+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251043+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:21.251043+0000","last_peered":"2024-09-21T18:12:21.251043+0000","last_clean":"2024-09-21T18:12:21.251043+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:21.251043+0000","last_undegraded":"2024-09-21T18:12:21.251043+0000","last_fullsized":"2024-09-21T18:12:21.251043+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.228145+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:21.228145+0000","last_peered":"2024-09-21T18:12:21.228145+0000","last_clean":"2024-09-21T18:12:21.228145+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:21.228145+0000","last_undegraded":"2024-09-21T18:12:21.228145+0000","last_fullsized":"2024-09-21T18:12:21.228145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323336,"kb_used_data":283816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048504,"statfs":{"total":96636764160,"available":96305668096,"internally_reserved":0,"allocated":290627584,"data_stored":301155780,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182680,"kb_used_data":151492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189160,"statfs":{"total":96636764160,"available":96449699840,"internally_reserved":0,"allocated":155127808,"data_stored":162763463,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350572,"kb_used_data":311056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021268,"statfs":{"total":96636764160,"available":96277778432,"internally_reserved":0,"allocated":318521344,"data_stored":331198622,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313289,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":700416,"data_stored":455234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338224,"kb_used_data":298712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033616,"statfs":{"total":96636764160,"available":96290422784,"internally_reserved":0,"allocated":305881088,"data_stored":319105146,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27712,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":696320,"data_stored":454840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323532,"kb_used_data":292344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048308,"statfs":{"total":96636764160,"available":96305467392,"internally_reserved":0,"allocated":299360256,"data_stored":314015725,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356044,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":672,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":688128,"data_stored":454294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":704512,"data_stored":455565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178544,"kb_used_data":151508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193296,"statfs":{"total":96636764160,"available":96453935104,"internally_reserved":0,"allocated":155144192,"data_stored":162764681,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330260,"kb_used_data":286588,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041580,"statfs":{"total":96636764160,"available":96298577920,"internally_reserved":0,"allocated":293466112,"data_stored":303377403,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482356,"kb_used_data":442844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889484,"statfs":{"total":96636764160,"available":96142831616,"internally_reserved":0,"allocated":453472256,"data_stored":474503901,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":700416,"data_stored":453448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191916,"kb_used_data":160724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179924,"statfs":{"total":96636764160,"available":96440242176,"internally_reserved":0,"allocated":164581376,"data_stored":170804603,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:25.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:26.058 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:26.075 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-21T18:12:25.992494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2904684,"kb_used_data":2383844,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044756,"statfs":{"total":1546188226560,"available":1543213830144,"internally_reserved":0,"allocated":2441056256,"data_stored":2542871702,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.842776"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":482,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250982+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:21.250982+0000","last_peered":"2024-09-21T18:12:21.250982+0000","last_clean":"2024-09-21T18:12:21.250982+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:21.250982+0000","last_undegraded":"2024-09-21T18:12:21.250982+0000","last_fullsized":"2024-09-21T18:12:21.250982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":375,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251108+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:21.251108+0000","last_peered":"2024-09-21T18:12:21.251108+0000","last_clean":"2024-09-21T18:12:21.251108+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:21.251108+0000","last_undegraded":"2024-09-21T18:12:21.251108+0000","last_fullsized":"2024-09-21T18:12:21.251108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1235,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250979+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:21.250979+0000","last_peered":"2024-09-21T18:12:21.250979+0000","last_clean":"2024-09-21T18:12:21.250979+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:21.250979+0000","last_undegraded":"2024-09-21T18:12:21.250979+0000","last_fullsized":"2024-09-21T18:12:21.250979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363668+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:21.363668+0000","last_peered":"2024-09-21T18:12:21.363668+0000","last_clean":"2024-09-21T18:12:21.363668+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:21.363668+0000","last_undegraded":"2024-09-21T18:12:21.363668+0000","last_fullsized":"2024-09-21T18:12:21.363668+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1691,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957272+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:20.957272+0000","last_peered":"2024-09-21T18:12:20.957272+0000","last_clean":"2024-09-21T18:12:20.957272+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:20.957272+0000","last_undegraded":"2024-09-21T18:12:20.957272+0000","last_fullsized":"2024-09-21T18:12:20.957272+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:27.164579+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":339,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327449+0000","last_change":"2024-09-21T18:11:18.907283+0000","last_active":"2024-09-21T18:12:21.327449+0000","last_peered":"2024-09-21T18:12:21.327449+0000","last_clean":"2024-09-21T18:12:21.327449+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:21.327449+0000","last_undegraded":"2024-09-21T18:12:21.327449+0000","last_fullsized":"2024-09-21T18:12:21.327449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:18.907114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:18.907114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:29.818720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:22.461498+0000","last_change":"2024-09-21T18:12:06.352918+0000","last_active":"2024-09-21T18:12:22.461498+0000","last_peered":"2024-09-21T18:12:22.461498+0000","last_clean":"2024-09-21T18:12:22.461498+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:22.461498+0000","last_undegraded":"2024-09-21T18:12:22.461498+0000","last_fullsized":"2024-09-21T18:12:22.461498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":104,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:35.851886+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363675+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:21.363675+0000","last_peered":"2024-09-21T18:12:21.363675+0000","last_clean":"2024-09-21T18:12:21.363675+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:21.363675+0000","last_undegraded":"2024-09-21T18:12:21.363675+0000","last_fullsized":"2024-09-21T18:12:21.363675+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1740,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327443+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:21.327443+0000","last_peered":"2024-09-21T18:12:21.327443+0000","last_clean":"2024-09-21T18:12:21.327443+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:21.327443+0000","last_undegraded":"2024-09-21T18:12:21.327443+0000","last_fullsized":"2024-09-21T18:12:21.327443+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2678,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.427662+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:21.427662+0000","last_peered":"2024-09-21T18:12:21.427662+0000","last_clean":"2024-09-21T18:12:21.427662+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:21.427662+0000","last_undegraded":"2024-09-21T18:12:21.427662+0000","last_fullsized":"2024-09-21T18:12:21.427662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":342,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957263+0000","last_change":"2024-09-21T18:11:15.583692+0000","last_active":"2024-09-21T18:12:20.957263+0000","last_peered":"2024-09-21T18:12:20.957263+0000","last_clean":"2024-09-21T18:12:20.957263+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:20.957263+0000","last_undegraded":"2024-09-21T18:12:20.957263+0000","last_fullsized":"2024-09-21T18:12:20.957263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:15.583514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:15.583514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:31.683887+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2037,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251041+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:21.251041+0000","last_peered":"2024-09-21T18:12:21.251041+0000","last_clean":"2024-09-21T18:12:21.251041+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:21.251041+0000","last_undegraded":"2024-09-21T18:12:21.251041+0000","last_fullsized":"2024-09-21T18:12:21.251041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.060290+0000","last_change":"2024-09-21T18:12:01.314179+0000","last_active":"2024-09-21T18:12:21.060290+0000","last_peered":"2024-09-21T18:12:21.060290+0000","last_clean":"2024-09-21T18:12:21.060290+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:21.060290+0000","last_undegraded":"2024-09-21T18:12:21.060290+0000","last_fullsized":"2024-09-21T18:12:21.060290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:21.770766+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2774,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957285+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:20.957285+0000","last_peered":"2024-09-21T18:12:20.957285+0000","last_clean":"2024-09-21T18:12:20.957285+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:20.957285+0000","last_undegraded":"2024-09-21T18:12:20.957285+0000","last_fullsized":"2024-09-21T18:12:20.957285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":349,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.798316+0000","last_change":"2024-09-21T18:11:30.157639+0000","last_active":"2024-09-21T18:12:21.798316+0000","last_peered":"2024-09-21T18:12:21.798316+0000","last_clean":"2024-09-21T18:12:21.798316+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:21.798316+0000","last_undegraded":"2024-09-21T18:12:21.798316+0000","last_fullsized":"2024-09-21T18:12:21.798316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:30.157362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:30.157362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:37.787991+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251043+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:21.251043+0000","last_peered":"2024-09-21T18:12:21.251043+0000","last_clean":"2024-09-21T18:12:21.251043+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:21.251043+0000","last_undegraded":"2024-09-21T18:12:21.251043+0000","last_fullsized":"2024-09-21T18:12:21.251043+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.228145+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:21.228145+0000","last_peered":"2024-09-21T18:12:21.228145+0000","last_clean":"2024-09-21T18:12:21.228145+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:21.228145+0000","last_undegraded":"2024-09-21T18:12:21.228145+0000","last_fullsized":"2024-09-21T18:12:21.228145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323336,"kb_used_data":283816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048504,"statfs":{"total":96636764160,"available":96305668096,"internally_reserved":0,"allocated":290627584,"data_stored":301155780,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182680,"kb_used_data":151492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189160,"statfs":{"total":96636764160,"available":96449699840,"internally_reserved":0,"allocated":155127808,"data_stored":162763463,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350572,"kb_used_data":311056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021268,"statfs":{"total":96636764160,"available":96277778432,"internally_reserved":0,"allocated":318521344,"data_stored":331198622,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":700416,"data_stored":455234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338224,"kb_used_data":298712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033616,"statfs":{"total":96636764160,"available":96290422784,"internally_reserved":0,"allocated":305881088,"data_stored":319105146,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27712,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":696320,"data_stored":454840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323532,"kb_used_data":292344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048308,"statfs":{"total":96636764160,"available":96305467392,"internally_reserved":0,"allocated":299360256,"data_stored":314015725,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":672,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":688128,"data_stored":454294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":704512,"data_stored":455565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178544,"kb_used_data":151508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193296,"statfs":{"total":96636764160,"available":96453935104,"internally_reserved":0,"allocated":155144192,"data_stored":162764681,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330260,"kb_used_data":286588,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041580,"statfs":{"total":96636764160,"available":96298577920,"internally_reserved":0,"allocated":293466112,"data_stored":303377403,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482356,"kb_used_data":442844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889484,"statfs":{"total":96636764160,"available":96142831616,"internally_reserved":0,"allocated":453472256,"data_stored":474503901,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":700416,"data_stored":453448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191916,"kb_used_data":160724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179924,"statfs":{"total":96636764160,"available":96440242176,"internally_reserved":0,"allocated":164581376,"data_stored":170804603,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:26.076 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T18:12:26.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T18:12:26.403 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T18:12:26.418 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T18:12:26.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:26.716 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:26.733 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-21T18:12:25.992494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2904684,"kb_used_data":2383844,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044756,"statfs":{"total":1546188226560,"available":1543213830144,"internally_reserved":0,"allocated":2441056256,"data_stored":2542871702,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.842776"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":482,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250982+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:21.250982+0000","last_peered":"2024-09-21T18:12:21.250982+0000","last_clean":"2024-09-21T18:12:21.250982+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:21.250982+0000","last_undegraded":"2024-09-21T18:12:21.250982+0000","last_fullsized":"2024-09-21T18:12:21.250982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":375,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251108+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:21.251108+0000","last_peered":"2024-09-21T18:12:21.251108+0000","last_clean":"2024-09-21T18:12:21.251108+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:21.251108+0000","last_undegraded":"2024-09-21T18:12:21.251108+0000","last_fullsized":"2024-09-21T18:12:21.251108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1235,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250979+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:21.250979+0000","last_peered":"2024-09-21T18:12:21.250979+0000","last_clean":"2024-09-21T18:12:21.250979+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:21.250979+0000","last_undegraded":"2024-09-21T18:12:21.250979+0000","last_fullsized":"2024-09-21T18:12:21.250979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363668+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:21.363668+0000","last_peered":"2024-09-21T18:12:21.363668+0000","last_clean":"2024-09-21T18:12:21.363668+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:21.363668+0000","last_undegraded":"2024-09-21T18:12:21.363668+0000","last_fullsized":"2024-09-21T18:12:21.363668+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1691,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957272+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:20.957272+0000","last_peered":"2024-09-21T18:12:20.957272+0000","last_clean":"2024-09-21T18:12:20.957272+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:20.957272+0000","last_undegraded":"2024-09-21T18:12:20.957272+0000","last_fullsized":"2024-09-21T18:12:20.957272+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:27.164579+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":339,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327449+0000","last_change":"2024-09-21T18:11:18.907283+0000","last_active":"2024-09-21T18:12:21.327449+0000","last_peered":"2024-09-21T18:12:21.327449+0000","last_clean":"2024-09-21T18:12:21.327449+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:21.327449+0000","last_undegraded":"2024-09-21T18:12:21.327449+0000","last_fullsized":"2024-09-21T18:12:21.327449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:18.907114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:18.907114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:29.818720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:22.461498+0000","last_change":"2024-09-21T18:12:06.352918+0000","last_active":"2024-09-21T18:12:22.461498+0000","last_peered":"2024-09-21T18:12:22.461498+0000","last_clean":"2024-09-21T18:12:22.461498+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:22.461498+0000","last_undegraded":"2024-09-21T18:12:22.461498+0000","last_fullsized":"2024-09-21T18:12:22.461498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":104,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:35.851886+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363675+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:21.363675+0000","last_peered":"2024-09-21T18:12:21.363675+0000","last_clean":"2024-09-21T18:12:21.363675+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:21.363675+0000","last_undegraded":"2024-09-21T18:12:21.363675+0000","last_fullsized":"2024-09-21T18:12:21.363675+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1740,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327443+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:21.327443+0000","last_peered":"2024-09-21T18:12:21.327443+0000","last_clean":"2024-09-21T18:12:21.327443+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:21.327443+0000","last_undegraded":"2024-09-21T18:12:21.327443+0000","last_fullsized":"2024-09-21T18:12:21.327443+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2678,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.427662+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:21.427662+0000","last_peered":"2024-09-21T18:12:21.427662+0000","last_clean":"2024-09-21T18:12:21.427662+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:21.427662+0000","last_undegraded":"2024-09-21T18:12:21.427662+0000","last_fullsized":"2024-09-21T18:12:21.427662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":342,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957263+0000","last_change":"2024-09-21T18:11:15.583692+0000","last_active":"2024-09-21T18:12:20.957263+0000","last_peered":"2024-09-21T18:12:20.957263+0000","last_clean":"2024-09-21T18:12:20.957263+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:20.957263+0000","last_undegraded":"2024-09-21T18:12:20.957263+0000","last_fullsized":"2024-09-21T18:12:20.957263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:15.583514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:15.583514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:31.683887+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2037,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251041+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:21.251041+0000","last_peered":"2024-09-21T18:12:21.251041+0000","last_clean":"2024-09-21T18:12:21.251041+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:21.251041+0000","last_undegraded":"2024-09-21T18:12:21.251041+0000","last_fullsized":"2024-09-21T18:12:21.251041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.060290+0000","last_change":"2024-09-21T18:12:01.314179+0000","last_active":"2024-09-21T18:12:21.060290+0000","last_peered":"2024-09-21T18:12:21.060290+0000","last_clean":"2024-09-21T18:12:21.060290+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:21.060290+0000","last_undegraded":"2024-09-21T18:12:21.060290+0000","last_fullsized":"2024-09-21T18:12:21.060290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:21.770766+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2774,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957285+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:20.957285+0000","last_peered":"2024-09-21T18:12:20.957285+0000","last_clean":"2024-09-21T18:12:20.957285+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:20.957285+0000","last_undegraded":"2024-09-21T18:12:20.957285+0000","last_fullsized":"2024-09-21T18:12:20.957285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":349,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.798316+0000","last_change":"2024-09-21T18:11:30.157639+0000","last_active":"2024-09-21T18:12:21.798316+0000","last_peered":"2024-09-21T18:12:21.798316+0000","last_clean":"2024-09-21T18:12:21.798316+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:21.798316+0000","last_undegraded":"2024-09-21T18:12:21.798316+0000","last_fullsized":"2024-09-21T18:12:21.798316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:30.157362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:30.157362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:37.787991+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251043+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:21.251043+0000","last_peered":"2024-09-21T18:12:21.251043+0000","last_clean":"2024-09-21T18:12:21.251043+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:21.251043+0000","last_undegraded":"2024-09-21T18:12:21.251043+0000","last_fullsized":"2024-09-21T18:12:21.251043+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.228145+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:21.228145+0000","last_peered":"2024-09-21T18:12:21.228145+0000","last_clean":"2024-09-21T18:12:21.228145+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:21.228145+0000","last_undegraded":"2024-09-21T18:12:21.228145+0000","last_fullsized":"2024-09-21T18:12:21.228145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323336,"kb_used_data":283816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048504,"statfs":{"total":96636764160,"available":96305668096,"internally_reserved":0,"allocated":290627584,"data_stored":301155780,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182680,"kb_used_data":151492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189160,"statfs":{"total":96636764160,"available":96449699840,"internally_reserved":0,"allocated":155127808,"data_stored":162763463,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350572,"kb_used_data":311056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021268,"statfs":{"total":96636764160,"available":96277778432,"internally_reserved":0,"allocated":318521344,"data_stored":331198622,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":700416,"data_stored":455234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338224,"kb_used_data":298712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033616,"statfs":{"total":96636764160,"available":96290422784,"internally_reserved":0,"allocated":305881088,"data_stored":319105146,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27712,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":696320,"data_stored":454840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323532,"kb_used_data":292344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048308,"statfs":{"total":96636764160,"available":96305467392,"internally_reserved":0,"allocated":299360256,"data_stored":314015725,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":672,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":688128,"data_stored":454294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":704512,"data_stored":455565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178544,"kb_used_data":151508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193296,"statfs":{"total":96636764160,"available":96453935104,"internally_reserved":0,"allocated":155144192,"data_stored":162764681,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330260,"kb_used_data":286588,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041580,"statfs":{"total":96636764160,"available":96298577920,"internally_reserved":0,"allocated":293466112,"data_stored":303377403,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482356,"kb_used_data":442844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889484,"statfs":{"total":96636764160,"available":96142831616,"internally_reserved":0,"allocated":453472256,"data_stored":474503901,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":700416,"data_stored":453448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191916,"kb_used_data":160724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179924,"statfs":{"total":96636764160,"available":96440242176,"internally_reserved":0,"allocated":164581376,"data_stored":170804603,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:26.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:27.027 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:27.027 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:27.043 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-21T18:12:25.992494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2904684,"kb_used_data":2383844,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044756,"statfs":{"total":1546188226560,"available":1543213830144,"internally_reserved":0,"allocated":2441056256,"data_stored":2542871702,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.842776"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":482,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250982+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:21.250982+0000","last_peered":"2024-09-21T18:12:21.250982+0000","last_clean":"2024-09-21T18:12:21.250982+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:21.250982+0000","last_undegraded":"2024-09-21T18:12:21.250982+0000","last_fullsized":"2024-09-21T18:12:21.250982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":375,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251108+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:21.251108+0000","last_peered":"2024-09-21T18:12:21.251108+0000","last_clean":"2024-09-21T18:12:21.251108+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:21.251108+0000","last_undegraded":"2024-09-21T18:12:21.251108+0000","last_fullsized":"2024-09-21T18:12:21.251108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1235,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250979+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:21.250979+0000","last_peered":"2024-09-21T18:12:21.250979+0000","last_clean":"2024-09-21T18:12:21.250979+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:21.250979+0000","last_undegraded":"2024-09-21T18:12:21.250979+0000","last_fullsized":"2024-09-21T18:12:21.250979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363668+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:21.363668+0000","last_peered":"2024-09-21T18:12:21.363668+0000","last_clean":"2024-09-21T18:12:21.363668+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:21.363668+0000","last_undegraded":"2024-09-21T18:12:21.363668+0000","last_fullsized":"2024-09-21T18:12:21.363668+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1691,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957272+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:20.957272+0000","last_peered":"2024-09-21T18:12:20.957272+0000","last_clean":"2024-09-21T18:12:20.957272+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:20.957272+0000","last_undegraded":"2024-09-21T18:12:20.957272+0000","last_fullsized":"2024-09-21T18:12:20.957272+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:27.164579+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":339,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327449+0000","last_change":"2024-09-21T18:11:18.907283+0000","last_active":"2024-09-21T18:12:21.327449+0000","last_peered":"2024-09-21T18:12:21.327449+0000","last_clean":"2024-09-21T18:12:21.327449+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:21.327449+0000","last_undegraded":"2024-09-21T18:12:21.327449+0000","last_fullsized":"2024-09-21T18:12:21.327449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:18.907114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:18.907114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:29.818720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:22.461498+0000","last_change":"2024-09-21T18:12:06.352918+0000","last_active":"2024-09-21T18:12:22.461498+0000","last_peered":"2024-09-21T18:12:22.461498+0000","last_clean":"2024-09-21T18:12:22.461498+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:22.461498+0000","last_undegraded":"2024-09-21T18:12:22.461498+0000","last_fullsized":"2024-09-21T18:12:22.461498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":104,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:35.851886+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363675+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:21.363675+0000","last_peered":"2024-09-21T18:12:21.363675+0000","last_clean":"2024-09-21T18:12:21.363675+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:21.363675+0000","last_undegraded":"2024-09-21T18:12:21.363675+0000","last_fullsized":"2024-09-21T18:12:21.363675+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1740,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327443+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:21.327443+0000","last_peered":"2024-09-21T18:12:21.327443+0000","last_clean":"2024-09-21T18:12:21.327443+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:21.327443+0000","last_undegraded":"2024-09-21T18:12:21.327443+0000","last_fullsized":"2024-09-21T18:12:21.327443+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2678,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.427662+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:21.427662+0000","last_peered":"2024-09-21T18:12:21.427662+0000","last_clean":"2024-09-21T18:12:21.427662+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:21.427662+0000","last_undegraded":"2024-09-21T18:12:21.427662+0000","last_fullsized":"2024-09-21T18:12:21.427662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":342,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957263+0000","last_change":"2024-09-21T18:11:15.583692+0000","last_active":"2024-09-21T18:12:20.957263+0000","last_peered":"2024-09-21T18:12:20.957263+0000","last_clean":"2024-09-21T18:12:20.957263+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:20.957263+0000","last_undegraded":"2024-09-21T18:12:20.957263+0000","last_fullsized":"2024-09-21T18:12:20.957263+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:15.583514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:15.583514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:31.683887+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2037,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251041+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:21.251041+0000","last_peered":"2024-09-21T18:12:21.251041+0000","last_clean":"2024-09-21T18:12:21.251041+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:21.251041+0000","last_undegraded":"2024-09-21T18:12:21.251041+0000","last_fullsized":"2024-09-21T18:12:21.251041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.060290+0000","last_change":"2024-09-21T18:12:01.314179+0000","last_active":"2024-09-21T18:12:21.060290+0000","last_peered":"2024-09-21T18:12:21.060290+0000","last_clean":"2024-09-21T18:12:21.060290+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:21.060290+0000","last_undegraded":"2024-09-21T18:12:21.060290+0000","last_fullsized":"2024-09-21T18:12:21.060290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:21.770766+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2774,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:20.957285+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:20.957285+0000","last_peered":"2024-09-21T18:12:20.957285+0000","last_clean":"2024-09-21T18:12:20.957285+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:20.957285+0000","last_undegraded":"2024-09-21T18:12:20.957285+0000","last_fullsized":"2024-09-21T18:12:20.957285+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":349,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.798316+0000","last_change":"2024-09-21T18:11:30.157639+0000","last_active":"2024-09-21T18:12:21.798316+0000","last_peered":"2024-09-21T18:12:21.798316+0000","last_clean":"2024-09-21T18:12:21.798316+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:21.798316+0000","last_undegraded":"2024-09-21T18:12:21.798316+0000","last_fullsized":"2024-09-21T18:12:21.798316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:30.157362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:30.157362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:37.787991+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251043+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:21.251043+0000","last_peered":"2024-09-21T18:12:21.251043+0000","last_clean":"2024-09-21T18:12:21.251043+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:21.251043+0000","last_undegraded":"2024-09-21T18:12:21.251043+0000","last_fullsized":"2024-09-21T18:12:21.251043+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.228145+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:21.228145+0000","last_peered":"2024-09-21T18:12:21.228145+0000","last_clean":"2024-09-21T18:12:21.228145+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:21.228145+0000","last_undegraded":"2024-09-21T18:12:21.228145+0000","last_fullsized":"2024-09-21T18:12:21.228145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323336,"kb_used_data":283816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048504,"statfs":{"total":96636764160,"available":96305668096,"internally_reserved":0,"allocated":290627584,"data_stored":301155780,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182680,"kb_used_data":151492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189160,"statfs":{"total":96636764160,"available":96449699840,"internally_reserved":0,"allocated":155127808,"data_stored":162763463,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313288,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350572,"kb_used_data":311056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021268,"statfs":{"total":96636764160,"available":96277778432,"internally_reserved":0,"allocated":318521344,"data_stored":331198622,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":700416,"data_stored":455234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338224,"kb_used_data":298712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033616,"statfs":{"total":96636764160,"available":96290422784,"internally_reserved":0,"allocated":305881088,"data_stored":319105146,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280583,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27712,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":696320,"data_stored":454840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280584,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323532,"kb_used_data":292344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048308,"statfs":{"total":96636764160,"available":96305467392,"internally_reserved":0,"allocated":299360256,"data_stored":314015725,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":672,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":688128,"data_stored":454294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":704512,"data_stored":455565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178544,"kb_used_data":151508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193296,"statfs":{"total":96636764160,"available":96453935104,"internally_reserved":0,"allocated":155144192,"data_stored":162764681,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330260,"kb_used_data":286588,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041580,"statfs":{"total":96636764160,"available":96298577920,"internally_reserved":0,"allocated":293466112,"data_stored":303377403,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482356,"kb_used_data":442844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889484,"statfs":{"total":96636764160,"available":96142831616,"internally_reserved":0,"allocated":453472256,"data_stored":474503901,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":700416,"data_stored":453448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280584,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191916,"kb_used_data":160724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179924,"statfs":{"total":96636764160,"available":96440242176,"internally_reserved":0,"allocated":164581376,"data_stored":170804603,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:27.045 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T18:12:27.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:27.341 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:27.358 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-09-21T18:12:27.152179+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2904704,"kb_used_data":2383868,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044736,"statfs":{"total":1546188226560,"available":1543213809664,"internally_reserved":0,"allocated":2441080832,"data_stored":2542887150,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.208481"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":482,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250982+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:21.250982+0000","last_peered":"2024-09-21T18:12:21.250982+0000","last_clean":"2024-09-21T18:12:21.250982+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:21.250982+0000","last_undegraded":"2024-09-21T18:12:21.250982+0000","last_fullsized":"2024-09-21T18:12:21.250982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":375,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251108+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:21.251108+0000","last_peered":"2024-09-21T18:12:21.251108+0000","last_clean":"2024-09-21T18:12:21.251108+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:21.251108+0000","last_undegraded":"2024-09-21T18:12:21.251108+0000","last_fullsized":"2024-09-21T18:12:21.251108+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1235,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.250979+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:21.250979+0000","last_peered":"2024-09-21T18:12:21.250979+0000","last_clean":"2024-09-21T18:12:21.250979+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:21.250979+0000","last_undegraded":"2024-09-21T18:12:21.250979+0000","last_fullsized":"2024-09-21T18:12:21.250979+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":80,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363668+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:21.363668+0000","last_peered":"2024-09-21T18:12:21.363668+0000","last_clean":"2024-09-21T18:12:21.363668+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:21.363668+0000","last_undegraded":"2024-09-21T18:12:21.363668+0000","last_fullsized":"2024-09-21T18:12:21.363668+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1693,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T18:12:25.995078+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:25.995078+0000","last_peered":"2024-09-21T18:12:25.995078+0000","last_clean":"2024-09-21T18:12:25.995078+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:25.995078+0000","last_undegraded":"2024-09-21T18:12:25.995078+0000","last_fullsized":"2024-09-21T18:12:25.995078+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:27.164579+0000","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":339,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327449+0000","last_change":"2024-09-21T18:11:18.907283+0000","last_active":"2024-09-21T18:12:21.327449+0000","last_peered":"2024-09-21T18:12:21.327449+0000","last_clean":"2024-09-21T18:12:21.327449+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:21.327449+0000","last_undegraded":"2024-09-21T18:12:21.327449+0000","last_fullsized":"2024-09-21T18:12:21.327449+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:18.907114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:18.907114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:29.818720+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1226,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:22.461498+0000","last_change":"2024-09-21T18:12:06.352918+0000","last_active":"2024-09-21T18:12:22.461498+0000","last_peered":"2024-09-21T18:12:22.461498+0000","last_clean":"2024-09-21T18:12:22.461498+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:22.461498+0000","last_undegraded":"2024-09-21T18:12:22.461498+0000","last_fullsized":"2024-09-21T18:12:22.461498+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":104,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:35.851886+0000","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":85,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.363675+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:21.363675+0000","last_peered":"2024-09-21T18:12:21.363675+0000","last_clean":"2024-09-21T18:12:21.363675+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:21.363675+0000","last_undegraded":"2024-09-21T18:12:21.363675+0000","last_fullsized":"2024-09-21T18:12:21.363675+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1740,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.327443+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:21.327443+0000","last_peered":"2024-09-21T18:12:21.327443+0000","last_clean":"2024-09-21T18:12:21.327443+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:21.327443+0000","last_undegraded":"2024-09-21T18:12:21.327443+0000","last_fullsized":"2024-09-21T18:12:21.327443+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2678,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.427662+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:21.427662+0000","last_peered":"2024-09-21T18:12:21.427662+0000","last_clean":"2024-09-21T18:12:21.427662+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:21.427662+0000","last_undegraded":"2024-09-21T18:12:21.427662+0000","last_fullsized":"2024-09-21T18:12:21.427662+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":344,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T18:12:26.095210+0000","last_change":"2024-09-21T18:11:15.583692+0000","last_active":"2024-09-21T18:12:26.095210+0000","last_peered":"2024-09-21T18:12:26.095210+0000","last_clean":"2024-09-21T18:12:26.095210+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:26.095210+0000","last_undegraded":"2024-09-21T18:12:26.095210+0000","last_fullsized":"2024-09-21T18:12:26.095210+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:15.583514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:15.583514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:31.683887+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2037,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251041+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:21.251041+0000","last_peered":"2024-09-21T18:12:21.251041+0000","last_clean":"2024-09-21T18:12:21.251041+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:21.251041+0000","last_undegraded":"2024-09-21T18:12:21.251041+0000","last_fullsized":"2024-09-21T18:12:21.251041+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":369,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.060290+0000","last_change":"2024-09-21T18:12:01.314179+0000","last_active":"2024-09-21T18:12:21.060290+0000","last_peered":"2024-09-21T18:12:21.060290+0000","last_clean":"2024-09-21T18:12:21.060290+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:21.060290+0000","last_undegraded":"2024-09-21T18:12:21.060290+0000","last_fullsized":"2024-09-21T18:12:21.060290+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:21.770766+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2776,"reported_epoch":187,"state":"active+clean","last_fresh":"2024-09-21T18:12:25.995142+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:25.995142+0000","last_peered":"2024-09-21T18:12:25.995142+0000","last_clean":"2024-09-21T18:12:25.995142+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:25.995142+0000","last_undegraded":"2024-09-21T18:12:25.995142+0000","last_fullsized":"2024-09-21T18:12:25.995142+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":349,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.798316+0000","last_change":"2024-09-21T18:11:30.157639+0000","last_active":"2024-09-21T18:12:21.798316+0000","last_peered":"2024-09-21T18:12:21.798316+0000","last_clean":"2024-09-21T18:12:21.798316+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:21.798316+0000","last_undegraded":"2024-09-21T18:12:21.798316+0000","last_fullsized":"2024-09-21T18:12:21.798316+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:30.157362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:30.157362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:37.787991+0000","scrub_duration":403,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2247,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.251043+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:21.251043+0000","last_peered":"2024-09-21T18:12:21.251043+0000","last_clean":"2024-09-21T18:12:21.251043+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:21.251043+0000","last_undegraded":"2024-09-21T18:12:21.251043+0000","last_fullsized":"2024-09-21T18:12:21.251043+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":72,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-21T18:12:21.228145+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:21.228145+0000","last_peered":"2024-09-21T18:12:21.228145+0000","last_clean":"2024-09-21T18:12:21.228145+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:21.228145+0000","last_undegraded":"2024-09-21T18:12:21.228145+0000","last_fullsized":"2024-09-21T18:12:21.228145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323336,"kb_used_data":283816,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048504,"statfs":{"total":96636764160,"available":96305668096,"internally_reserved":0,"allocated":290627584,"data_stored":301155780,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247879,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247880,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182680,"kb_used_data":151492,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189160,"statfs":{"total":96636764160,"available":96449699840,"internally_reserved":0,"allocated":155127808,"data_stored":162763463,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313289,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350572,"kb_used_data":311056,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021268,"statfs":{"total":96636764160,"available":96277778432,"internally_reserved":0,"allocated":318521344,"data_stored":331198622,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31868,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339972,"statfs":{"total":96636764160,"available":96604131328,"internally_reserved":0,"allocated":692224,"data_stored":454538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27712,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":700416,"data_stored":455234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280583,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338224,"kb_used_data":298712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033616,"statfs":{"total":96636764160,"available":96290422784,"internally_reserved":0,"allocated":305881088,"data_stored":319105146,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280584,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27712,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344128,"statfs":{"total":96636764160,"available":96608387072,"internally_reserved":0,"allocated":696320,"data_stored":454840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280585,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323532,"kb_used_data":292344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048308,"statfs":{"total":96636764160,"available":96305467392,"internally_reserved":0,"allocated":299360256,"data_stored":314015725,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356045,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28660,"kb_used_data":672,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343180,"statfs":{"total":96636764160,"available":96607416320,"internally_reserved":0,"allocated":688128,"data_stored":454294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27716,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344124,"statfs":{"total":96636764160,"available":96608382976,"internally_reserved":0,"allocated":704512,"data_stored":455565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280584,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178544,"kb_used_data":151508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193296,"statfs":{"total":96636764160,"available":96453935104,"internally_reserved":0,"allocated":155144192,"data_stored":162764681,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280583,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330260,"kb_used_data":286588,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041580,"statfs":{"total":96636764160,"available":96298577920,"internally_reserved":0,"allocated":293466112,"data_stored":303377403,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280584,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482356,"kb_used_data":442844,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889484,"statfs":{"total":96636764160,"available":96142831616,"internally_reserved":0,"allocated":453472256,"data_stored":474503901,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":700416,"data_stored":453448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280585,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191936,"kb_used_data":160748,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179904,"statfs":{"total":96636764160,"available":96440221696,"internally_reserved":0,"allocated":164605952,"data_stored":170820051,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:27.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T18:12:27.525 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:12:27.523+0000 7fc038bbb640 -1 osd.7 186 handle_osd_map injecting map crc failure 2024-09-21T18:12:27.658 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:27.658 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":188,"fsid":"ea312541-f204-4ac1-b2ae-f5daa6b7f078","created":"2024-09-21T18:06:28.087618+0000","modified":"2024-09-21T18:12:27.383569+0000","last_up_change":"2024-09-21T18:11:31.261580+0000","last_in_change":"2024-09-21T18:11:48.364704+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":16,"require_min_compat_client":"reef","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T18:06:41.093821+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":15.380000114440918,"score_stable":15.380000114440918,"optimal_score":0.12999999523162842,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T18:06:43.022470+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":28,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"base","create_time":"2024-09-21T18:06:59.604621+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[4],"tier_of":-1,"read_tier":4,"write_tier":4,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cache","create_time":"2024-09-21T18:07:01.499947+0000","flags":9,"flags_names":"hashpspool,incomplete_clones","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":4,"pg_placement_num":4,"pg_placement_num_target":4,"pg_num_target":4,"pg_num_pending":4,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"36","last_force_op_resend_prenautilus":"36","last_force_op_resend_preluminous":"36","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":34,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":3,"read_tier":-1,"write_tier":-1,"cache_mode":"writeback","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"bloom","false_positive_probability":0.050000000000000003,"target_size":0,"seed":0},"hit_set_period":3600,"hit_set_count":8,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[1000000,1000000,1000000,1000000,1000000,1000000,1000000,1000000],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"f0d6ae6f-6e5d-4f16-94dc-99c6afc000d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6808","nonce":3976261094}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6809","nonce":3976261094}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6811","nonce":3976261094}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6810","nonce":3976261094}]},"public_addr":"172.21.15.29:6808/3976261094","cluster_addr":"172.21.15.29:6809/3976261094","heartbeat_back_addr":"172.21.15.29:6811/3976261094","heartbeat_front_addr":"172.21.15.29:6810/3976261094","state":["exists","up"]},{"osd":1,"uuid":"020df4b2-13cb-4e14-88a1-a8219b5b34b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6808","nonce":408295210}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6809","nonce":408295210}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6811","nonce":408295210}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6810","nonce":408295210}]},"public_addr":"172.21.15.40:6808/408295210","cluster_addr":"172.21.15.40:6809/408295210","heartbeat_back_addr":"172.21.15.40:6811/408295210","heartbeat_front_addr":"172.21.15.40:6810/408295210","state":["exists","up"]},{"osd":2,"uuid":"57a83527-6610-4eb4-a01e-e22c3ade98a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6804","nonce":1064027255}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6805","nonce":1064027255}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6807","nonce":1064027255}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6806","nonce":1064027255}]},"public_addr":"172.21.15.70:6804/1064027255","cluster_addr":"172.21.15.70:6805/1064027255","heartbeat_back_addr":"172.21.15.70:6807/1064027255","heartbeat_front_addr":"172.21.15.70:6806/1064027255","state":["exists","up"]},{"osd":3,"uuid":"42758eff-818c-4e2b-98c8-5f9c01c883c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6813","nonce":1036992123}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6814","nonce":1036992123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6816","nonce":1036992123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6815","nonce":1036992123}]},"public_addr":"172.21.15.160:6813/1036992123","cluster_addr":"172.21.15.160:6814/1036992123","heartbeat_back_addr":"172.21.15.160:6816/1036992123","heartbeat_front_addr":"172.21.15.160:6815/1036992123","state":["exists","up"]},{"osd":4,"uuid":"c358d178-864a-4a0a-a1a8-0c42d1b10d86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":151,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6800","nonce":2670503742}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6801","nonce":2670503742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6803","nonce":2670503742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6802","nonce":2670503742}]},"public_addr":"172.21.15.29:6800/2670503742","cluster_addr":"172.21.15.29:6801/2670503742","heartbeat_back_addr":"172.21.15.29:6803/2670503742","heartbeat_front_addr":"172.21.15.29:6802/2670503742","state":["exists","up"]},{"osd":5,"uuid":"73790513-6074-4d9b-80b2-0403d3c1f0c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":145,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6800","nonce":3912255253}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6801","nonce":3912255253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6803","nonce":3912255253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6802","nonce":3912255253}]},"public_addr":"172.21.15.40:6800/3912255253","cluster_addr":"172.21.15.40:6801/3912255253","heartbeat_back_addr":"172.21.15.40:6803/3912255253","heartbeat_front_addr":"172.21.15.40:6802/3912255253","state":["exists","up"]},{"osd":6,"uuid":"e588347d-7853-4ac0-a871-03a7311247e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":154,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6800","nonce":3623713846}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6801","nonce":3623713846}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6803","nonce":3623713846}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6802","nonce":3623713846}]},"public_addr":"172.21.15.70:6800/3623713846","cluster_addr":"172.21.15.70:6801/3623713846","heartbeat_back_addr":"172.21.15.70:6803/3623713846","heartbeat_front_addr":"172.21.15.70:6802/3623713846","state":["exists","up"]},{"osd":7,"uuid":"f47cab83-6e0b-478f-88e0-0cf069bb3f3c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":22,"last_clean_end":61,"up_from":142,"up_thru":0,"down_at":62,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6801","nonce":655055502}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6802","nonce":655055502}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6804","nonce":655055502}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6803","nonce":655055502}]},"public_addr":"172.21.15.160:6801/655055502","cluster_addr":"172.21.15.160:6802/655055502","heartbeat_back_addr":"172.21.15.160:6804/655055502","heartbeat_front_addr":"172.21.15.160:6803/655055502","state":["exists","up"]},{"osd":8,"uuid":"427789c1-98e4-43d1-a12c-2606a8a6cd4b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6804","nonce":4039331719}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6805","nonce":4039331719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6807","nonce":4039331719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6806","nonce":4039331719}]},"public_addr":"172.21.15.29:6804/4039331719","cluster_addr":"172.21.15.29:6805/4039331719","heartbeat_back_addr":"172.21.15.29:6807/4039331719","heartbeat_front_addr":"172.21.15.29:6806/4039331719","state":["exists","up"]},{"osd":9,"uuid":"268cb87f-9f31-4543-8fc5-4e05b6d9ed0d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6812","nonce":3437338221}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6813","nonce":3437338221}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6815","nonce":3437338221}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6814","nonce":3437338221}]},"public_addr":"172.21.15.40:6812/3437338221","cluster_addr":"172.21.15.40:6813/3437338221","heartbeat_back_addr":"172.21.15.40:6815/3437338221","heartbeat_front_addr":"172.21.15.40:6814/3437338221","state":["exists","up"]},{"osd":10,"uuid":"85e66756-8ef8-4ba9-911f-08d74f04e0a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6808","nonce":3389947982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6809","nonce":3389947982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6811","nonce":3389947982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6810","nonce":3389947982}]},"public_addr":"172.21.15.70:6808/3389947982","cluster_addr":"172.21.15.70:6809/3389947982","heartbeat_back_addr":"172.21.15.70:6811/3389947982","heartbeat_front_addr":"172.21.15.70:6810/3389947982","state":["exists","up"]},{"osd":11,"uuid":"952e9412-fdb5-4b68-b109-def00c946b31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6809","nonce":524032079}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6810","nonce":524032079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6812","nonce":524032079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6811","nonce":524032079}]},"public_addr":"172.21.15.160:6809/524032079","cluster_addr":"172.21.15.160:6810/524032079","heartbeat_back_addr":"172.21.15.160:6812/524032079","heartbeat_front_addr":"172.21.15.160:6811/524032079","state":["exists","up"]},{"osd":12,"uuid":"1b4bd0f6-ed9f-426e-bb82-4e4328199d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":150,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6812","nonce":2242191432}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6813","nonce":2242191432}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6815","nonce":2242191432}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.29:6814","nonce":2242191432}]},"public_addr":"172.21.15.29:6812/2242191432","cluster_addr":"172.21.15.29:6813/2242191432","heartbeat_back_addr":"172.21.15.29:6815/2242191432","heartbeat_front_addr":"172.21.15.29:6814/2242191432","state":["exists","up"]},{"osd":13,"uuid":"a535d39d-7bee-4185-a28d-4155f9b17207","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6804","nonce":707207003}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6805","nonce":707207003}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6807","nonce":707207003}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.40:6806","nonce":707207003}]},"public_addr":"172.21.15.40:6804/707207003","cluster_addr":"172.21.15.40:6805/707207003","heartbeat_back_addr":"172.21.15.40:6807/707207003","heartbeat_front_addr":"172.21.15.40:6806/707207003","state":["exists","up"]},{"osd":14,"uuid":"95b8504b-67c0-4ed9-be7e-81eb3dacacf6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":22,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6812","nonce":2255369090}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6813","nonce":2255369090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6815","nonce":2255369090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.70:6814","nonce":2255369090}]},"public_addr":"172.21.15.70:6812/2255369090","cluster_addr":"172.21.15.70:6813/2255369090","heartbeat_back_addr":"172.21.15.70:6815/2255369090","heartbeat_front_addr":"172.21.15.70:6814/2255369090","state":["exists","up"]},{"osd":15,"uuid":"84b8cc82-bc18-4cce-901e-ed48e85ba7e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":23,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6805","nonce":4046424455}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6806","nonce":4046424455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6808","nonce":4046424455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.160:6807","nonce":4046424455}]},"public_addr":"172.21.15.160:6805/4046424455","cluster_addr":"172.21.15.160:6806/4046424455","heartbeat_back_addr":"172.21.15.160:6808/4046424455","heartbeat_front_addr":"172.21.15.160:6807/4046424455","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:54.549709+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:37.644910+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.319783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:35.985784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:34.786377+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:26.665903+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:49.682678+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:30.774098+0000","dead_epoch":62},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:51.196099+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:59.981028+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:10.964912+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:56.102384+0000","dead_epoch":0},{"osd":12,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:48.713280+0000","dead_epoch":0},{"osd":13,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:11:26.051863+0000","dead_epoch":0},{"osd":14,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:25.605094+0000","dead_epoch":0},{"osd":15,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T18:12:05.599605+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T18:12:28.676 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T18:12:28.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:28.840 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:28.840 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:28.840 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:28.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T18:12:29.151 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T18:12:29.168 INFO:tasks.ceph:Scrubbing osd.4 2024-09-21T18:12:29.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:29.317 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:29.317 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:29.317 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:29.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-21T18:12:29.617 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2024-09-21T18:12:29.635 INFO:tasks.ceph:Scrubbing osd.8 2024-09-21T18:12:29.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.8 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:29.783 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:29.783 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:29.783 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:29.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 8 2024-09-21T18:12:30.097 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 8 to deep-scrub 2024-09-21T18:12:30.115 INFO:tasks.ceph:Scrubbing osd.12 2024-09-21T18:12:30.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.12 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:30.259 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:30.259 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:30.259 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:30.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 12 2024-09-21T18:12:30.570 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 12 to deep-scrub 2024-09-21T18:12:30.587 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T18:12:30.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:30.729 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:30.730 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:30.730 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:30.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T18:12:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T18:12:31.049 INFO:tasks.ceph:Scrubbing osd.5 2024-09-21T18:12:31.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:31.201 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:31.201 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:31.201 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:31.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-21T18:12:31.509 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2024-09-21T18:12:31.529 INFO:tasks.ceph:Scrubbing osd.9 2024-09-21T18:12:31.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.9 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:31.672 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:31.672 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:31.672 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:31.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 9 2024-09-21T18:12:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 9 to deep-scrub 2024-09-21T18:12:32.000 INFO:tasks.ceph:Scrubbing osd.13 2024-09-21T18:12:32.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.13 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:32.144 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:32.144 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:32.144 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:32.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 13 2024-09-21T18:12:32.456 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 13 to deep-scrub 2024-09-21T18:12:32.474 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T18:12:32.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:32.623 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:32.623 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:32.623 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:32.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T18:12:32.928 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T18:12:32.946 INFO:tasks.ceph:Scrubbing osd.6 2024-09-21T18:12:32.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:33.097 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:33.097 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:33.097 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:33.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-21T18:12:33.404 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 6 to deep-scrub 2024-09-21T18:12:33.421 INFO:tasks.ceph:Scrubbing osd.10 2024-09-21T18:12:33.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.10 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:33.565 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:33.565 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:33.565 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:33.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 10 2024-09-21T18:12:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 10 to deep-scrub 2024-09-21T18:12:33.906 INFO:tasks.ceph:Scrubbing osd.14 2024-09-21T18:12:33.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.14 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:34.065 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:34.065 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:34.065 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:34.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 14 2024-09-21T18:12:34.385 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 14 to deep-scrub 2024-09-21T18:12:34.405 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T18:12:34.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:34.561 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:34.562 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:34.562 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:34.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T18:12:34.880 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T18:12:34.898 INFO:tasks.ceph:Scrubbing osd.7 2024-09-21T18:12:34.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:35.053 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:35.053 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:35.053 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:35.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-21T18:12:35.361 INFO:tasks.ceph.osd.12.smithi029.stderr:2024-09-21T18:12:35.356+0000 7f8c3a0ca640 -1 osd.12 188 handle_osd_map injecting map crc failure 2024-09-21T18:12:35.368 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 7 to deep-scrub 2024-09-21T18:12:35.389 INFO:tasks.ceph:Scrubbing osd.11 2024-09-21T18:12:35.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.11 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:35.547 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:35.547 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:35.547 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:35.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 11 2024-09-21T18:12:35.580 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:12:35.576+0000 7ff409995640 -1 osd.4 188 handle_osd_map injecting map crc failure 2024-09-21T18:12:35.873 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 11 to deep-scrub 2024-09-21T18:12:35.893 INFO:tasks.ceph:Scrubbing osd.15 2024-09-21T18:12:35.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.15 config set osd_debug_deep_scrub_sleep 0 2024-09-21T18:12:36.099 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-09-21T18:12:36.099 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T18:12:36.099 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-09-21T18:12:36.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 15 2024-09-21T18:12:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 15 to deep-scrub 2024-09-21T18:12:36.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:36.764 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:36.764 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:36.782 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-21T18:12:35.154668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6606,"ondisk_log_size":6606,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2905024,"kb_used_data":2384124,"kb_used_omap":27,"kb_used_meta":520420,"kb_avail":1507044416,"statfs":{"total":1546188226560,"available":1543213481984,"internally_reserved":0,"allocated":2441342976,"data_stored":2543050629,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":532910395},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002489"},"pg_stats":[{"pgid":"1.0","version":"29'112","reported_seq":485,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.597926+0000","last_change":"2024-09-21T18:11:32.414501+0000","last_active":"2024-09-21T18:12:27.597926+0000","last_peered":"2024-09-21T18:12:27.597926+0000","last_clean":"2024-09-21T18:12:27.597926+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:27.597926+0000","last_undegraded":"2024-09-21T18:12:27.597926+0000","last_fullsized":"2024-09-21T18:12:27.597926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"29'112","last_scrub_stamp":"2024-09-21T18:11:32.414336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:41.831877+0000","last_clean_scrub_stamp":"2024-09-21T18:11:32.414336+0000","objects_scrubbed":2,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:35.192342+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"26'1","reported_seq":378,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.497962+0000","last_change":"2024-09-21T18:11:47.483025+0000","last_active":"2024-09-21T18:12:27.497962+0000","last_peered":"2024-09-21T18:12:27.497962+0000","last_clean":"2024-09-21T18:12:27.497962+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:27.497962+0000","last_undegraded":"2024-09-21T18:12:27.497962+0000","last_fullsized":"2024-09-21T18:12:27.497962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"26'1","last_scrub_stamp":"2024-09-21T18:11:19.293763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:19.293763+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.084491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"130'504","reported_seq":1238,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.597936+0000","last_change":"2024-09-21T18:12:04.116161+0000","last_active":"2024-09-21T18:12:27.597936+0000","last_peered":"2024-09-21T18:12:27.597936+0000","last_clean":"2024-09-21T18:12:27.597936+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:27.597936+0000","last_undegraded":"2024-09-21T18:12:27.597936+0000","last_fullsized":"2024-09-21T18:12:27.597936+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'504","last_scrub_stamp":"2024-09-21T18:12:04.115977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:04.115977+0000","objects_scrubbed":108,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.640461+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":84,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:28.490226+0000","last_change":"2024-09-21T18:11:49.809980+0000","last_active":"2024-09-21T18:12:28.490226+0000","last_peered":"2024-09-21T18:12:28.490226+0000","last_clean":"2024-09-21T18:12:28.490226+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:28.490226+0000","last_undegraded":"2024-09-21T18:12:28.490226+0000","last_fullsized":"2024-09-21T18:12:28.490226+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:49.809811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:49.809811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:57.849881+0000","scrub_duration":423,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"129'709","reported_seq":1695,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.387382+0000","last_change":"2024-09-21T18:11:46.461255+0000","last_active":"2024-09-21T18:12:27.387382+0000","last_peered":"2024-09-21T18:12:27.387382+0000","last_clean":"2024-09-21T18:12:27.387382+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:27.387382+0000","last_undegraded":"2024-09-21T18:12:27.387382+0000","last_fullsized":"2024-09-21T18:12:27.387382+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'709","last_scrub_stamp":"2024-09-21T18:11:21.010979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:11:21.010979+0000","objects_scrubbed":100,"log_size":709,"log_dups_size":0,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":350,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:31.595618+0000","last_change":"2024-09-21T18:12:31.595618+0000","last_active":"2024-09-21T18:12:31.595618+0000","last_peered":"2024-09-21T18:12:31.595618+0000","last_clean":"2024-09-21T18:12:31.595618+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:31.595618+0000","last_undegraded":"2024-09-21T18:12:31.595618+0000","last_fullsized":"2024-09-21T18:12:31.595618+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:31.595435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:31.595435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:44.793853+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"130'482","reported_seq":1233,"reported_epoch":188,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-21T18:12:31.884362+0000","last_change":"2024-09-21T18:12:31.883681+0000","last_active":"2024-09-21T18:12:31.884362+0000","last_peered":"2024-09-21T18:12:31.884362+0000","last_clean":"2024-09-21T18:12:31.884362+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:31.884362+0000","last_undegraded":"2024-09-21T18:12:31.884362+0000","last_fullsized":"2024-09-21T18:12:31.884362+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'482","last_scrub_stamp":"2024-09-21T18:12:06.352709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:06.352709+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 1s","scrub_duration":419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"28'2","reported_seq":89,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:28.490321+0000","last_change":"2024-09-21T18:12:15.817271+0000","last_active":"2024-09-21T18:12:28.490321+0000","last_peered":"2024-09-21T18:12:28.490321+0000","last_clean":"2024-09-21T18:12:28.490321+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:28.490321+0000","last_undegraded":"2024-09-21T18:12:28.490321+0000","last_fullsized":"2024-09-21T18:12:28.490321+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"28'2","last_scrub_stamp":"2024-09-21T18:12:15.817079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:15.817079+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:23.496471+0000","scrub_duration":239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"129'718","reported_seq":1743,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.581264+0000","last_change":"2024-09-21T18:12:13.964130+0000","last_active":"2024-09-21T18:12:27.581264+0000","last_peered":"2024-09-21T18:12:27.581264+0000","last_clean":"2024-09-21T18:12:27.581264+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:27.581264+0000","last_undegraded":"2024-09-21T18:12:27.581264+0000","last_fullsized":"2024-09-21T18:12:27.581264+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'718","last_scrub_stamp":"2024-09-21T18:12:13.963848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:00.079908+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.963848+0000","objects_scrubbed":97,"log_size":718,"log_dups_size":0,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.699901+0000","scrub_duration":323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"130'1174","reported_seq":2681,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.530366+0000","last_change":"2024-09-21T18:11:45.460397+0000","last_active":"2024-09-21T18:12:27.530366+0000","last_peered":"2024-09-21T18:12:27.530366+0000","last_clean":"2024-09-21T18:12:27.530366+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:27.530366+0000","last_undegraded":"2024-09-21T18:12:27.530366+0000","last_fullsized":"2024-09-21T18:12:27.530366+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1174","last_scrub_stamp":"2024-09-21T18:11:20.204147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:20.204147+0000","objects_scrubbed":113,"log_size":1174,"log_dups_size":0,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:34.806630+0000","scrub_duration":331,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":354,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:30.615900+0000","last_change":"2024-09-21T18:12:30.615900+0000","last_active":"2024-09-21T18:12:30.615900+0000","last_peered":"2024-09-21T18:12:30.615900+0000","last_clean":"2024-09-21T18:12:30.615900+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:30.615900+0000","last_undegraded":"2024-09-21T18:12:30.615900+0000","last_fullsized":"2024-09-21T18:12:30.615900+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.615805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:46.219982+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"130'855","reported_seq":2040,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.497953+0000","last_change":"2024-09-21T18:12:01.873218+0000","last_active":"2024-09-21T18:12:27.497953+0000","last_peered":"2024-09-21T18:12:27.497953+0000","last_clean":"2024-09-21T18:12:27.497953+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:27.497953+0000","last_undegraded":"2024-09-21T18:12:27.497953+0000","last_fullsized":"2024-09-21T18:12:27.497953+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":117,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:27.528540+0000","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":376,"reported_epoch":188,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-21T18:12:32.035409+0000","last_change":"2024-09-21T18:12:32.034945+0000","last_active":"2024-09-21T18:12:32.035409+0000","last_peered":"2024-09-21T18:12:32.035409+0000","last_clean":"2024-09-21T18:12:32.035409+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:32.035409+0000","last_undegraded":"2024-09-21T18:12:32.035409+0000","last_fullsized":"2024-09-21T18:12:32.035409+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:01.313994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.313994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"130'1158","reported_seq":2778,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.387382+0000","last_change":"2024-09-21T18:11:50.613667+0000","last_active":"2024-09-21T18:12:27.387382+0000","last_peered":"2024-09-21T18:12:27.387382+0000","last_clean":"2024-09-21T18:12:27.387382+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:27.387382+0000","last_undegraded":"2024-09-21T18:12:27.387382+0000","last_fullsized":"2024-09-21T18:12:27.387382+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'1158","last_scrub_stamp":"2024-09-21T18:11:50.613565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:11:50.613565+0000","objects_scrubbed":124,"log_size":1158,"log_dups_size":0,"ondisk_log_size":1158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:14.767683+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":360,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:30.533934+0000","last_change":"2024-09-21T18:12:30.533934+0000","last_active":"2024-09-21T18:12:30.533934+0000","last_peered":"2024-09-21T18:12:30.533934+0000","last_clean":"2024-09-21T18:12:30.533934+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:30.533934+0000","last_undegraded":"2024-09-21T18:12:30.533934+0000","last_fullsized":"2024-09-21T18:12:30.533934+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.533780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:43.131707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"130'891","reported_seq":2250,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.497864+0000","last_change":"2024-09-21T18:12:13.151922+0000","last_active":"2024-09-21T18:12:27.497864+0000","last_peered":"2024-09-21T18:12:27.497864+0000","last_clean":"2024-09-21T18:12:27.497864+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:27.497864+0000","last_undegraded":"2024-09-21T18:12:27.497864+0000","last_fullsized":"2024-09-21T18:12:27.497864+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'891","last_scrub_stamp":"2024-09-21T18:12:13.151688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:13.151688+0000","objects_scrubbed":117,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:31.971925+0000","scrub_duration":731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":76,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-21T18:12:27.590860+0000","last_change":"2024-09-21T18:11:49.949100+0000","last_active":"2024-09-21T18:12:27.590860+0000","last_peered":"2024-09-21T18:12:27.590860+0000","last_clean":"2024-09-21T18:12:27.590860+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:27.590860+0000","last_undegraded":"2024-09-21T18:12:27.590860+0000","last_fullsized":"2024-09-21T18:12:27.590860+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:11:29.483793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:06:43.843471+0000","last_clean_scrub_stamp":"2024-09-21T18:11:29.483793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:12:45.746131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4078,"ondisk_log_size":4078,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2413,"ondisk_log_size":2413,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247883,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323344,"kb_used_data":283840,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048496,"statfs":{"total":96636764160,"available":96305659904,"internally_reserved":0,"allocated":290652160,"data_stored":301171270,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247881,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31888,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339952,"statfs":{"total":96636764160,"available":96604110848,"internally_reserved":0,"allocated":716800,"data_stored":469949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247882,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182704,"kb_used_data":151516,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189136,"statfs":{"total":96636764160,"available":96449675264,"internally_reserved":0,"allocated":155152384,"data_stored":162778953,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313290,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350592,"kb_used_data":311080,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021248,"statfs":{"total":96636764160,"available":96277757952,"internally_reserved":0,"allocated":318545920,"data_stored":331214112,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313291,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31892,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339948,"statfs":{"total":96636764160,"available":96604106752,"internally_reserved":0,"allocated":716800,"data_stored":470028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":724992,"data_stored":470724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280585,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338248,"kb_used_data":298620,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033592,"statfs":{"total":96636764160,"available":96290398208,"internally_reserved":0,"allocated":305786880,"data_stored":319044020,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280585,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27732,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344108,"statfs":{"total":96636764160,"available":96608366592,"internally_reserved":0,"allocated":720896,"data_stored":470330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280586,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323556,"kb_used_data":292368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94048284,"statfs":{"total":96636764160,"available":96305442816,"internally_reserved":0,"allocated":299384832,"data_stored":314031215,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356046,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28688,"kb_used_data":696,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343152,"statfs":{"total":96636764160,"available":96607387648,"internally_reserved":0,"allocated":712704,"data_stored":469784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":729088,"data_stored":471055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":178560,"kb_used_data":151532,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94193280,"statfs":{"total":96636764160,"available":96453918720,"internally_reserved":0,"allocated":155168768,"data_stored":162780171,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280585,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":330284,"kb_used_data":286612,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94041556,"statfs":{"total":96636764160,"available":96298553344,"internally_reserved":0,"allocated":293490688,"data_stored":303392893,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":44693952},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280586,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482376,"kb_used_data":442868,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889464,"statfs":{"total":96636764160,"available":96142811136,"internally_reserved":0,"allocated":453496832,"data_stored":474519391,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344104,"statfs":{"total":96636764160,"available":96608362496,"internally_reserved":0,"allocated":724992,"data_stored":468938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280586,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191948,"kb_used_data":160760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179892,"statfs":{"total":96636764160,"available":96440209408,"internally_reserved":0,"allocated":164618240,"data_stored":170827796,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:36.785 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T18:11:32.414336+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=32, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.785 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T18:11:19.293763+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=19, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-21T18:12:04.115977+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T18:11:49.809811+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-21T18:11:21.010979+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=21, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-21T18:12:06.352709+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=6, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T18:12:15.817079+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=15, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-21T18:12:13.963848+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.786 INFO:tasks.ceph:pgid 4.2 last_scrub_stamp 2024-09-21T18:11:20.204147+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=20, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-21T18:12:01.873042+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=1, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T18:12:01.313994+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=1, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:pgid 4.0 last_scrub_stamp 2024-09-21T18:11:50.613565+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=50, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:pgid 4.1 last_scrub_stamp 2024-09-21T18:12:13.151688+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=13, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T18:11:29.483793+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=11, tm_sec=29, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:36.787 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T18:12:45.384 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:12:45.382+0000 7f9bb588f640 -1 osd.10 189 handle_osd_map injecting map crc failure 2024-09-21T18:12:46.411 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:12:46.408+0000 7fb924fb6640 -1 osd.8 189 handle_osd_map injecting map crc failure 2024-09-21T18:12:56.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:12:57.112 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:12:57.113 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:12:57.132 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-09-21T18:12:55.161209+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7462,"ondisk_log_size":7462,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2895196,"kb_used_data":2382772,"kb_used_omap":27,"kb_used_meta":512100,"kb_avail":1507054244,"statfs":{"total":1546188226560,"available":1543223545856,"internally_reserved":0,"allocated":2439958528,"data_stored":2542150523,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":524390715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003326"},"pg_stats":[{"pgid":"1.0","version":"189'114","reported_seq":500,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.371824+0000","last_change":"2024-09-21T18:12:38.837877+0000","last_active":"2024-09-21T18:12:46.371824+0000","last_peered":"2024-09-21T18:12:46.371824+0000","last_clean":"2024-09-21T18:12:46.371824+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:12:46.371824+0000","last_undegraded":"2024-09-21T18:12:46.371824+0000","last_fullsized":"2024-09-21T18:12:46.371824+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'114","last_scrub_stamp":"2024-09-21T18:12:38.837698+0000","last_deep_scrub":"189'114","last_deep_scrub_stamp":"2024-09-21T18:12:38.837698+0000","last_clean_scrub_stamp":"2024-09-21T18:12:38.837698+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:41.520199+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"189'2","reported_seq":391,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.376055+0000","last_change":"2024-09-21T18:12:37.501642+0000","last_active":"2024-09-21T18:12:46.376055+0000","last_peered":"2024-09-21T18:12:46.376055+0000","last_clean":"2024-09-21T18:12:46.376055+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:12:46.376055+0000","last_undegraded":"2024-09-21T18:12:46.376055+0000","last_fullsized":"2024-09-21T18:12:46.376055+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'2","last_scrub_stamp":"2024-09-21T18:12:37.501471+0000","last_deep_scrub":"189'2","last_deep_scrub_stamp":"2024-09-21T18:12:37.501471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:37.501471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:05.725165+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"190'612","reported_seq":1366,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:53.113461+0000","last_change":"2024-09-21T18:12:53.113461+0000","last_active":"2024-09-21T18:12:53.113461+0000","last_peered":"2024-09-21T18:12:53.113461+0000","last_clean":"2024-09-21T18:12:53.113461+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:12:53.113461+0000","last_undegraded":"2024-09-21T18:12:53.113461+0000","last_fullsized":"2024-09-21T18:12:53.113461+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'612","last_scrub_stamp":"2024-09-21T18:12:53.113238+0000","last_deep_scrub":"190'612","last_deep_scrub_stamp":"2024-09-21T18:12:53.113238+0000","last_clean_scrub_stamp":"2024-09-21T18:12:53.113238+0000","objects_scrubbed":108,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:57.612064+0000","scrub_duration":16703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":96,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.288709+0000","last_change":"2024-09-21T18:12:34.947281+0000","last_active":"2024-09-21T18:12:46.288709+0000","last_peered":"2024-09-21T18:12:46.288709+0000","last_clean":"2024-09-21T18:12:46.288709+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:12:46.288709+0000","last_undegraded":"2024-09-21T18:12:46.288709+0000","last_fullsized":"2024-09-21T18:12:46.288709+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:34.947087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:34.947087+0000","last_clean_scrub_stamp":"2024-09-21T18:12:34.947087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:03.617178+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"190'809","reported_seq":1814,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:47.095823+0000","last_change":"2024-09-21T18:12:47.095823+0000","last_active":"2024-09-21T18:12:47.095823+0000","last_peered":"2024-09-21T18:12:47.095823+0000","last_clean":"2024-09-21T18:12:47.095823+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:12:47.095823+0000","last_undegraded":"2024-09-21T18:12:47.095823+0000","last_fullsized":"2024-09-21T18:12:47.095823+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'809","last_scrub_stamp":"2024-09-21T18:12:47.095606+0000","last_deep_scrub":"190'809","last_deep_scrub_stamp":"2024-09-21T18:12:47.095606+0000","last_clean_scrub_stamp":"2024-09-21T18:12:47.095606+0000","objects_scrubbed":100,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:15.817198+0000","scrub_duration":15571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":362,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:45.580979+0000","last_change":"2024-09-21T18:12:38.801785+0000","last_active":"2024-09-21T18:12:45.580979+0000","last_peered":"2024-09-21T18:12:45.580979+0000","last_clean":"2024-09-21T18:12:45.580979+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:12:45.580979+0000","last_undegraded":"2024-09-21T18:12:45.580979+0000","last_fullsized":"2024-09-21T18:12:45.580979+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:38.801502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:38.801502+0000","last_clean_scrub_stamp":"2024-09-21T18:12:38.801502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:08.539323+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"190'586","reported_seq":1352,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:47.641775+0000","last_change":"2024-09-21T18:12:47.641775+0000","last_active":"2024-09-21T18:12:47.641775+0000","last_peered":"2024-09-21T18:12:47.641775+0000","last_clean":"2024-09-21T18:12:47.641775+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:12:47.641775+0000","last_undegraded":"2024-09-21T18:12:47.641775+0000","last_fullsized":"2024-09-21T18:12:47.641775+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'586","last_scrub_stamp":"2024-09-21T18:12:47.641422+0000","last_deep_scrub":"190'586","last_deep_scrub_stamp":"2024-09-21T18:12:47.641422+0000","last_clean_scrub_stamp":"2024-09-21T18:12:47.641422+0000","objects_scrubbed":104,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:59.043205+0000","scrub_duration":15755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"188'3","reported_seq":102,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.288709+0000","last_change":"2024-09-21T18:12:33.731538+0000","last_active":"2024-09-21T18:12:46.288709+0000","last_peered":"2024-09-21T18:12:46.288709+0000","last_clean":"2024-09-21T18:12:46.288709+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:12:46.288709+0000","last_undegraded":"2024-09-21T18:12:46.288709+0000","last_fullsized":"2024-09-21T18:12:46.288709+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'3","last_scrub_stamp":"2024-09-21T18:12:33.731370+0000","last_deep_scrub":"188'3","last_deep_scrub_stamp":"2024-09-21T18:12:33.731370+0000","last_clean_scrub_stamp":"2024-09-21T18:12:33.731370+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:34.371870+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"190'815","reported_seq":1859,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:52.830439+0000","last_change":"2024-09-21T18:12:52.830439+0000","last_active":"2024-09-21T18:12:52.830439+0000","last_peered":"2024-09-21T18:12:52.830439+0000","last_clean":"2024-09-21T18:12:52.830439+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:12:52.830439+0000","last_undegraded":"2024-09-21T18:12:52.830439+0000","last_fullsized":"2024-09-21T18:12:52.830439+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'815","last_scrub_stamp":"2024-09-21T18:12:52.830195+0000","last_deep_scrub":"190'815","last_deep_scrub_stamp":"2024-09-21T18:12:52.830195+0000","last_clean_scrub_stamp":"2024-09-21T18:12:52.830195+0000","objects_scrubbed":97,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:06.567595+0000","scrub_duration":16307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"190'1287","reported_seq":2823,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:51.777380+0000","last_change":"2024-09-21T18:12:51.777380+0000","last_active":"2024-09-21T18:12:51.777380+0000","last_peered":"2024-09-21T18:12:51.777380+0000","last_clean":"2024-09-21T18:12:51.777380+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:12:51.777380+0000","last_undegraded":"2024-09-21T18:12:51.777380+0000","last_fullsized":"2024-09-21T18:12:51.777380+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1287","last_scrub_stamp":"2024-09-21T18:12:51.777116+0000","last_deep_scrub":"190'1287","last_deep_scrub_stamp":"2024-09-21T18:12:51.777116+0000","last_clean_scrub_stamp":"2024-09-21T18:12:51.777116+0000","objects_scrubbed":113,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:12.310648+0000","scrub_duration":14223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":358,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:45.225897+0000","last_change":"2024-09-21T18:12:30.615900+0000","last_active":"2024-09-21T18:12:45.225897+0000","last_peered":"2024-09-21T18:12:45.225897+0000","last_clean":"2024-09-21T18:12:45.225897+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:12:45.225897+0000","last_undegraded":"2024-09-21T18:12:45.225897+0000","last_fullsized":"2024-09-21T18:12:45.225897+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.615805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:46.219982+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"190'944","reported_seq":2144,"reported_epoch":190,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-21T18:12:52.889342+0000","last_change":"2024-09-21T18:12:40.435402+0000","last_active":"2024-09-21T18:12:52.889342+0000","last_peered":"2024-09-21T18:12:52.889342+0000","last_clean":"2024-09-21T18:12:52.889342+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:12:52.889342+0000","last_undegraded":"2024-09-21T18:12:52.889342+0000","last_fullsized":"2024-09-21T18:12:52.889342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'855","last_scrub_stamp":"2024-09-21T18:12:01.873042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:07:02.095335+0000","last_clean_scrub_stamp":"2024-09-21T18:12:01.873042+0000","objects_scrubbed":104,"log_size":944,"log_dups_size":0,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"deep scrubbing for 14s","scrub_duration":563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":385,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.126144+0000","last_change":"2024-09-21T18:12:32.137554+0000","last_active":"2024-09-21T18:12:46.126144+0000","last_peered":"2024-09-21T18:12:46.126144+0000","last_clean":"2024-09-21T18:12:46.126144+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:12:46.126144+0000","last_undegraded":"2024-09-21T18:12:46.126144+0000","last_fullsized":"2024-09-21T18:12:46.126144+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:32.137371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:32.137371+0000","last_clean_scrub_stamp":"2024-09-21T18:12:32.137371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:01.991717+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"190'1282","reported_seq":2923,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:50.640184+0000","last_change":"2024-09-21T18:12:50.640184+0000","last_active":"2024-09-21T18:12:50.640184+0000","last_peered":"2024-09-21T18:12:50.640184+0000","last_clean":"2024-09-21T18:12:50.640184+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:12:50.640184+0000","last_undegraded":"2024-09-21T18:12:50.640184+0000","last_fullsized":"2024-09-21T18:12:50.640184+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1282","last_scrub_stamp":"2024-09-21T18:12:50.640001+0000","last_deep_scrub":"190'1282","last_deep_scrub_stamp":"2024-09-21T18:12:50.640001+0000","last_clean_scrub_stamp":"2024-09-21T18:12:50.640001+0000","objects_scrubbed":124,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":19,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:54.193704+0000","scrub_duration":18159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":364,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:46.417812+0000","last_change":"2024-09-21T18:12:30.533934+0000","last_active":"2024-09-21T18:12:46.417812+0000","last_peered":"2024-09-21T18:12:46.417812+0000","last_clean":"2024-09-21T18:12:46.417812+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:12:46.417812+0000","last_undegraded":"2024-09-21T18:12:46.417812+0000","last_fullsized":"2024-09-21T18:12:46.417812+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.533780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:43.131707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"190'1008","reported_seq":2387,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:54.267271+0000","last_change":"2024-09-21T18:12:54.267271+0000","last_active":"2024-09-21T18:12:54.267271+0000","last_peered":"2024-09-21T18:12:54.267271+0000","last_clean":"2024-09-21T18:12:54.267271+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:12:54.267271+0000","last_undegraded":"2024-09-21T18:12:54.267271+0000","last_fullsized":"2024-09-21T18:12:54.267271+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1008","last_scrub_stamp":"2024-09-21T18:12:54.267048+0000","last_deep_scrub":"190'1008","last_deep_scrub_stamp":"2024-09-21T18:12:54.267048+0000","last_clean_scrub_stamp":"2024-09-21T18:12:54.267048+0000","objects_scrubbed":117,"log_size":1008,"log_dups_size":0,"ondisk_log_size":1008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:19.465376+0000","scrub_duration":14819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":88,"reported_epoch":190,"state":"active+clean","last_fresh":"2024-09-21T18:12:45.227453+0000","last_change":"2024-09-21T18:12:33.939411+0000","last_active":"2024-09-21T18:12:45.227453+0000","last_peered":"2024-09-21T18:12:45.227453+0000","last_clean":"2024-09-21T18:12:45.227453+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:12:45.227453+0000","last_undegraded":"2024-09-21T18:12:45.227453+0000","last_fullsized":"2024-09-21T18:12:45.227453+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:33.939244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:33.939244+0000","last_clean_scrub_stamp":"2024-09-21T18:12:33.939244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:59.178929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4521,"ondisk_log_size":4521,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2822,"ondisk_log_size":2822,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247887,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323244,"kb_used_data":283740,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048596,"statfs":{"total":96636764160,"available":96305762304,"internally_reserved":0,"allocated":290549760,"data_stored":301109606,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247885,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":630784,"data_stored":409511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182612,"kb_used_data":151424,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189228,"statfs":{"total":96636764160,"available":96449769472,"internally_reserved":0,"allocated":155058176,"data_stored":162717877,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313294,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350492,"kb_used_data":310988,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021348,"statfs":{"total":96636764160,"available":96277860352,"internally_reserved":0,"allocated":318451712,"data_stored":331153036,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31804,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340036,"statfs":{"total":96636764160,"available":96604196864,"internally_reserved":0,"allocated":630784,"data_stored":409540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":630784,"data_stored":409648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338144,"kb_used_data":298640,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033696,"statfs":{"total":96636764160,"available":96290504704,"internally_reserved":0,"allocated":305807360,"data_stored":319059216,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280589,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27640,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344200,"statfs":{"total":96636764160,"available":96608460800,"internally_reserved":0,"allocated":626688,"data_stored":409304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280590,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327624,"kb_used_data":292280,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94044216,"statfs":{"total":96636764160,"available":96301277184,"internally_reserved":0,"allocated":299294720,"data_stored":313970433,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356050,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28600,"kb_used_data":612,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343240,"statfs":{"total":96636764160,"available":96607477760,"internally_reserved":0,"allocated":626688,"data_stored":409346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344196,"statfs":{"total":96636764160,"available":96608456704,"internally_reserved":0,"allocated":630784,"data_stored":409685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182628,"kb_used_data":151440,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189212,"statfs":{"total":96636764160,"available":96449753088,"internally_reserved":0,"allocated":155074560,"data_stored":162719095,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280589,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313540,"kb_used_data":286516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94058300,"statfs":{"total":96636764160,"available":96315699200,"internally_reserved":0,"allocated":293392384,"data_stored":303331523,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280590,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482280,"kb_used_data":442776,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889560,"statfs":{"total":96636764160,"available":96142909440,"internally_reserved":0,"allocated":453402624,"data_stored":474458415,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":618496,"data_stored":406980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191864,"kb_used_data":160676,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179976,"statfs":{"total":96636764160,"available":96440295424,"internally_reserved":0,"allocated":164532224,"data_stored":170767308,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:12:57.133 INFO:tasks.ceph:pgid 4.3 last_scrub_stamp 2024-09-21T18:12:01.873042+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=1, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=18, tm_min=12, tm_sec=27, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T18:12:57.133 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T18:13:01.313 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:13:01.310+0000 7f3ce267c640 -1 osd.6 190 handle_osd_map injecting map crc failure 2024-09-21T18:13:05.383 INFO:tasks.ceph.osd.2.smithi070.stderr:2024-09-21T18:13:05.382+0000 7fca4e93e640 -1 osd.2 193 handle_osd_map injecting map crc failure 2024-09-21T18:13:10.229 INFO:tasks.ceph.osd.13.smithi040.stderr:2024-09-21T18:13:10.226+0000 7f440980f640 -1 osd.13 194 handle_osd_map injecting map crc failure 2024-09-21T18:13:10.269 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:13:10.266+0000 7f4976023640 -1 osd.5 194 handle_osd_map injecting map crc failure 2024-09-21T18:13:10.493 INFO:tasks.ceph.osd.4.smithi029.stderr:2024-09-21T18:13:10.491+0000 7ff409995640 -1 osd.4 194 handle_osd_map injecting map crc failure 2024-09-21T18:13:10.493 INFO:tasks.ceph.osd.7.smithi160.stderr:2024-09-21T18:13:10.490+0000 7fc038bbb640 -1 osd.7 194 handle_osd_map injecting map crc failure 2024-09-21T18:13:17.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T18:13:17.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-09-21T18:13:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-09-21T18:13:17.479 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-21T18:13:17.167966+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2182462619,"num_objects":884,"num_object_clones":0,"num_object_copies":1768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":126,"num_read":8810,"num_read_kb":5434597,"num_write":6884,"num_write_kb":3392218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":221,"num_bytes_recovered":255732555,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7490,"ondisk_log_size":7490,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":16,"num_per_pool_osds":16,"num_per_pool_omap_osds":15,"kb":1509949440,"kb_used":2896152,"kb_used_data":2383704,"kb_used_omap":27,"kb_used_meta":512100,"kb_avail":1507053288,"statfs":{"total":1546188226560,"available":1543222566912,"internally_reserved":0,"allocated":2440912896,"data_stored":2542769615,"data_compressed":101075248,"data_compressed_allocated":107626496,"data_compressed_original":220482048,"omap_allocated":28357,"internal_metadata":524390715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003082"},"pg_stats":[{"pgid":"1.0","version":"189'114","reported_seq":509,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272659+0000","last_change":"2024-09-21T18:12:38.837877+0000","last_active":"2024-09-21T18:13:10.272659+0000","last_peered":"2024-09-21T18:13:10.272659+0000","last_clean":"2024-09-21T18:13:10.272659+0000","last_became_active":"2024-09-21T18:06:42.955142+0000","last_became_peered":"2024-09-21T18:06:42.955142+0000","last_unstale":"2024-09-21T18:13:10.272659+0000","last_undegraded":"2024-09-21T18:13:10.272659+0000","last_fullsized":"2024-09-21T18:13:10.272659+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'114","last_scrub_stamp":"2024-09-21T18:12:38.837698+0000","last_deep_scrub":"189'114","last_deep_scrub_stamp":"2024-09-21T18:12:38.837698+0000","last_clean_scrub_stamp":"2024-09-21T18:12:38.837698+0000","objects_scrubbed":2,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:41.520199+0000","scrub_duration":311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.3","version":"189'2","reported_seq":401,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.273051+0000","last_change":"2024-09-21T18:12:37.501642+0000","last_active":"2024-09-21T18:13:10.273051+0000","last_peered":"2024-09-21T18:13:10.273051+0000","last_clean":"2024-09-21T18:13:10.273051+0000","last_became_active":"2024-09-21T18:11:47.377619+0000","last_became_peered":"2024-09-21T18:11:47.377619+0000","last_unstale":"2024-09-21T18:13:10.273051+0000","last_undegraded":"2024-09-21T18:13:10.273051+0000","last_fullsized":"2024-09-21T18:13:10.273051+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'2","last_scrub_stamp":"2024-09-21T18:12:37.501471+0000","last_deep_scrub":"189'2","last_deep_scrub_stamp":"2024-09-21T18:12:37.501471+0000","last_clean_scrub_stamp":"2024-09-21T18:12:37.501471+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:05.725165+0000","scrub_duration":107,"objects_trimmed":0,"snaptrim_duration":0.00095924200000000002,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2","version":"190'612","reported_seq":1375,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272802+0000","last_change":"2024-09-21T18:12:53.113461+0000","last_active":"2024-09-21T18:13:10.272802+0000","last_peered":"2024-09-21T18:13:10.272802+0000","last_clean":"2024-09-21T18:13:10.272802+0000","last_became_active":"2024-09-21T18:07:01.286983+0000","last_became_peered":"2024-09-21T18:07:01.286983+0000","last_unstale":"2024-09-21T18:13:10.272802+0000","last_undegraded":"2024-09-21T18:13:10.272802+0000","last_fullsized":"2024-09-21T18:13:10.272802+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'612","last_scrub_stamp":"2024-09-21T18:12:53.113238+0000","last_deep_scrub":"190'612","last_deep_scrub_stamp":"2024-09-21T18:12:53.113238+0000","last_clean_scrub_stamp":"2024-09-21T18:12:53.113238+0000","objects_scrubbed":108,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:57.612064+0000","scrub_duration":16703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295508805,"num_objects":108,"num_object_clones":0,"num_object_copies":216,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":108,"num_whiteouts":0,"num_read":507,"num_read_kb":102743,"num_write":581,"num_write_kb":467800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":106,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.234838+0000","last_change":"2024-09-21T18:12:34.947281+0000","last_active":"2024-09-21T18:13:10.234838+0000","last_peered":"2024-09-21T18:13:10.234838+0000","last_clean":"2024-09-21T18:13:10.234838+0000","last_became_active":"2024-09-21T18:11:47.471411+0000","last_became_peered":"2024-09-21T18:11:47.471411+0000","last_unstale":"2024-09-21T18:13:10.234838+0000","last_undegraded":"2024-09-21T18:13:10.234838+0000","last_fullsized":"2024-09-21T18:13:10.234838+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:34.947087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:34.947087+0000","last_clean_scrub_stamp":"2024-09-21T18:12:34.947087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:03.617178+0000","scrub_duration":503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"190'809","reported_seq":1824,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.599334+0000","last_change":"2024-09-21T18:12:47.095823+0000","last_active":"2024-09-21T18:13:10.599334+0000","last_peered":"2024-09-21T18:13:10.599334+0000","last_clean":"2024-09-21T18:13:10.599334+0000","last_became_active":"2024-09-21T18:11:46.460265+0000","last_became_peered":"2024-09-21T18:11:46.460265+0000","last_unstale":"2024-09-21T18:13:10.599334+0000","last_undegraded":"2024-09-21T18:13:10.599334+0000","last_fullsized":"2024-09-21T18:13:10.599334+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'809","last_scrub_stamp":"2024-09-21T18:12:47.095606+0000","last_deep_scrub":"190'809","last_deep_scrub_stamp":"2024-09-21T18:12:47.095606+0000","last_clean_scrub_stamp":"2024-09-21T18:12:47.095606+0000","objects_scrubbed":100,"log_size":809,"log_dups_size":0,"ondisk_log_size":809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:15.817198+0000","scrub_duration":15571,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291544299,"num_objects":100,"num_object_clones":0,"num_object_copies":200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":100,"num_whiteouts":0,"num_read":844,"num_read_kb":335558,"num_write":831,"num_write_kb":468506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":42607728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,14],"acting":[4,14],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":373,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.372951+0000","last_change":"2024-09-21T18:12:38.801785+0000","last_active":"2024-09-21T18:13:10.372951+0000","last_peered":"2024-09-21T18:13:10.372951+0000","last_clean":"2024-09-21T18:13:10.372951+0000","last_became_active":"2024-09-21T18:06:44.875040+0000","last_became_peered":"2024-09-21T18:06:44.875040+0000","last_unstale":"2024-09-21T18:13:10.372951+0000","last_undegraded":"2024-09-21T18:13:10.372951+0000","last_fullsized":"2024-09-21T18:13:10.372951+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:38.801502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:38.801502+0000","last_clean_scrub_stamp":"2024-09-21T18:12:38.801502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:08.539323+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0.00079891700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,11],"acting":[15,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"3.0","version":"190'586","reported_seq":1362,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272358+0000","last_change":"2024-09-21T18:12:47.641775+0000","last_active":"2024-09-21T18:13:10.272358+0000","last_peered":"2024-09-21T18:13:10.272358+0000","last_clean":"2024-09-21T18:13:10.272358+0000","last_became_active":"2024-09-21T18:07:01.404929+0000","last_became_peered":"2024-09-21T18:07:01.404929+0000","last_unstale":"2024-09-21T18:13:10.272358+0000","last_undegraded":"2024-09-21T18:13:10.272358+0000","last_fullsized":"2024-09-21T18:13:10.272358+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'586","last_scrub_stamp":"2024-09-21T18:12:47.641422+0000","last_deep_scrub":"190'586","last_deep_scrub_stamp":"2024-09-21T18:12:47.641422+0000","last_clean_scrub_stamp":"2024-09-21T18:12:47.641422+0000","objects_scrubbed":104,"log_size":586,"log_dups_size":0,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:59.043205+0000","scrub_duration":15755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287593026,"num_objects":104,"num_object_clones":0,"num_object_copies":208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104,"num_whiteouts":0,"num_read":529,"num_read_kb":117861,"num_write":555,"num_write_kb":414402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,9],"acting":[1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"188'3","reported_seq":112,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.234838+0000","last_change":"2024-09-21T18:12:33.731538+0000","last_active":"2024-09-21T18:13:10.234838+0000","last_peered":"2024-09-21T18:13:10.234838+0000","last_clean":"2024-09-21T18:13:10.234838+0000","last_became_active":"2024-09-21T18:11:47.579778+0000","last_became_peered":"2024-09-21T18:11:47.579778+0000","last_unstale":"2024-09-21T18:13:10.234838+0000","last_undegraded":"2024-09-21T18:13:10.234838+0000","last_fullsized":"2024-09-21T18:13:10.234838+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"188'3","last_scrub_stamp":"2024-09-21T18:12:33.731370+0000","last_deep_scrub":"188'3","last_deep_scrub_stamp":"2024-09-21T18:12:33.731370+0000","last_clean_scrub_stamp":"2024-09-21T18:12:33.731370+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:34.371870+0000","scrub_duration":203,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,12],"acting":[2,12],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"190'815","reported_seq":1869,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272908+0000","last_change":"2024-09-21T18:12:52.830439+0000","last_active":"2024-09-21T18:13:10.272908+0000","last_peered":"2024-09-21T18:13:10.272908+0000","last_clean":"2024-09-21T18:13:10.272908+0000","last_became_active":"2024-09-21T18:11:47.378720+0000","last_became_peered":"2024-09-21T18:11:47.378720+0000","last_unstale":"2024-09-21T18:13:10.272908+0000","last_undegraded":"2024-09-21T18:13:10.272908+0000","last_fullsized":"2024-09-21T18:13:10.272908+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'815","last_scrub_stamp":"2024-09-21T18:12:52.830195+0000","last_deep_scrub":"190'815","last_deep_scrub_stamp":"2024-09-21T18:12:52.830195+0000","last_clean_scrub_stamp":"2024-09-21T18:12:52.830195+0000","objects_scrubbed":97,"log_size":815,"log_dups_size":0,"ondisk_log_size":815,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:06.567595+0000","scrub_duration":16307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278985894,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":843,"num_read_kb":320014,"num_write":849,"num_write_kb":500474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":75643431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,2],"acting":[15,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"4.2","version":"190'1287","reported_seq":2833,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272872+0000","last_change":"2024-09-21T18:12:51.777380+0000","last_active":"2024-09-21T18:13:10.272872+0000","last_peered":"2024-09-21T18:13:10.272872+0000","last_clean":"2024-09-21T18:13:10.272872+0000","last_became_active":"2024-09-21T18:11:45.457230+0000","last_became_peered":"2024-09-21T18:11:45.457230+0000","last_unstale":"2024-09-21T18:13:10.272872+0000","last_undegraded":"2024-09-21T18:13:10.272872+0000","last_fullsized":"2024-09-21T18:13:10.272872+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1287","last_scrub_stamp":"2024-09-21T18:12:51.777116+0000","last_deep_scrub":"190'1287","last_deep_scrub_stamp":"2024-09-21T18:12:51.777116+0000","last_clean_scrub_stamp":"2024-09-21T18:12:51.777116+0000","objects_scrubbed":113,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:12.310648+0000","scrub_duration":14223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":241219334,"num_objects":113,"num_object_clones":0,"num_object_copies":226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":1583,"num_read_kb":1178376,"num_write":1159,"num_write_kb":430599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":37784802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":105,"num_flush_kb":266015,"num_evict":0,"num_evict_kb":0,"num_promote":141,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[15,10],"acting":[15,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":15,"acting_primary":15,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":369,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.599362+0000","last_change":"2024-09-21T18:12:30.615900+0000","last_active":"2024-09-21T18:13:10.599362+0000","last_peered":"2024-09-21T18:13:10.599362+0000","last_clean":"2024-09-21T18:13:10.599362+0000","last_became_active":"2024-09-21T18:06:45.078202+0000","last_became_peered":"2024-09-21T18:06:45.078202+0000","last_unstale":"2024-09-21T18:13:10.599362+0000","last_undegraded":"2024-09-21T18:13:10.599362+0000","last_fullsized":"2024-09-21T18:13:10.599362+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.615805+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.615805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:46.219982+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.10121905,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"190'972","reported_seq":2186,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272851+0000","last_change":"2024-09-21T18:12:56.959315+0000","last_active":"2024-09-21T18:13:10.272851+0000","last_peered":"2024-09-21T18:13:10.272851+0000","last_clean":"2024-09-21T18:13:10.272851+0000","last_became_active":"2024-09-21T18:07:04.453884+0000","last_became_peered":"2024-09-21T18:07:04.453884+0000","last_unstale":"2024-09-21T18:13:10.272851+0000","last_undegraded":"2024-09-21T18:13:10.272851+0000","last_fullsized":"2024-09-21T18:13:10.272851+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'972","last_scrub_stamp":"2024-09-21T18:12:56.959071+0000","last_deep_scrub":"190'972","last_deep_scrub_stamp":"2024-09-21T18:12:56.959071+0000","last_clean_scrub_stamp":"2024-09-21T18:12:56.959071+0000","objects_scrubbed":117,"log_size":972,"log_dups_size":0,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":17,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:59.950119+0000","scrub_duration":16523,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250293818,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":1223,"num_read_kb":966444,"num_write":803,"num_write_kb":330015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":288,"num_flush":71,"num_flush_kb":168882,"num_evict":0,"num_evict_kb":0,"num_promote":140,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5],"acting":[11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":395,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.290085+0000","last_change":"2024-09-21T18:12:32.137554+0000","last_active":"2024-09-21T18:13:10.290085+0000","last_peered":"2024-09-21T18:13:10.290085+0000","last_clean":"2024-09-21T18:13:10.290085+0000","last_became_active":"2024-09-21T18:06:45.073452+0000","last_became_peered":"2024-09-21T18:06:45.073452+0000","last_unstale":"2024-09-21T18:13:10.290085+0000","last_undegraded":"2024-09-21T18:13:10.290085+0000","last_fullsized":"2024-09-21T18:13:10.290085+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:32.137371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:32.137371+0000","last_clean_scrub_stamp":"2024-09-21T18:12:32.137371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:01.991717+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0.00079733400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4],"acting":[9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.0","version":"190'1282","reported_seq":2933,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.599373+0000","last_change":"2024-09-21T18:12:50.640184+0000","last_active":"2024-09-21T18:13:10.599373+0000","last_peered":"2024-09-21T18:13:10.599373+0000","last_clean":"2024-09-21T18:13:10.599373+0000","last_became_active":"2024-09-21T18:11:45.361317+0000","last_became_peered":"2024-09-21T18:11:45.361317+0000","last_unstale":"2024-09-21T18:13:10.599373+0000","last_undegraded":"2024-09-21T18:13:10.599373+0000","last_fullsized":"2024-09-21T18:13:10.599373+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1282","last_scrub_stamp":"2024-09-21T18:12:50.640001+0000","last_deep_scrub":"190'1282","last_deep_scrub_stamp":"2024-09-21T18:12:50.640001+0000","last_clean_scrub_stamp":"2024-09-21T18:12:50.640001+0000","objects_scrubbed":124,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":19,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:54.193704+0000","scrub_duration":18159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271235812,"num_objects":124,"num_object_clones":0,"num_object_copies":248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":31,"num_read":1845,"num_read_kb":1354861,"num_write":1118,"num_write_kb":456179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":48773635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":298,"num_flush":99,"num_flush_kb":223712,"num_evict":0,"num_evict_kb":0,"num_promote":154,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10],"acting":[4,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":374,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.373323+0000","last_change":"2024-09-21T18:12:30.533934+0000","last_active":"2024-09-21T18:13:10.373323+0000","last_peered":"2024-09-21T18:13:10.373323+0000","last_clean":"2024-09-21T18:13:10.373323+0000","last_became_active":"2024-09-21T18:06:44.964498+0000","last_became_peered":"2024-09-21T18:06:44.964498+0000","last_unstale":"2024-09-21T18:13:10.373323+0000","last_undegraded":"2024-09-21T18:13:10.373323+0000","last_fullsized":"2024-09-21T18:13:10.373323+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:30.533780+0000","last_clean_scrub_stamp":"2024-09-21T18:12:30.533780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:43.131707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011274029999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,13],"acting":[8,13],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1","version":"190'1008","reported_seq":2396,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.272666+0000","last_change":"2024-09-21T18:12:54.267271+0000","last_active":"2024-09-21T18:13:10.272666+0000","last_peered":"2024-09-21T18:13:10.272666+0000","last_clean":"2024-09-21T18:13:10.272666+0000","last_became_active":"2024-09-21T18:11:47.379826+0000","last_became_peered":"2024-09-21T18:11:47.379826+0000","last_unstale":"2024-09-21T18:13:10.272666+0000","last_undegraded":"2024-09-21T18:13:10.272666+0000","last_fullsized":"2024-09-21T18:13:10.272666+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'1008","last_scrub_stamp":"2024-09-21T18:12:54.267048+0000","last_deep_scrub":"190'1008","last_deep_scrub_stamp":"2024-09-21T18:12:54.267048+0000","last_clean_scrub_stamp":"2024-09-21T18:12:54.267048+0000","objects_scrubbed":117,"log_size":1008,"log_dups_size":0,"ondisk_log_size":1008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:14:19.465376+0000","scrub_duration":14819,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265491244,"num_objects":117,"num_object_clones":0,"num_object_copies":234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":28,"num_read":1350,"num_read_kb":1058667,"num_write":841,"num_write_kb":321569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":50922921,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":1,"num_bytes_hit_set_archive":292,"num_flush":76,"num_flush_kb":170692,"num_evict":0,"num_evict_kb":0,"num_promote":145,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2],"acting":[11,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":195,"state":"active+clean","last_fresh":"2024-09-21T18:13:10.597852+0000","last_change":"2024-09-21T18:12:33.939411+0000","last_active":"2024-09-21T18:13:10.597852+0000","last_peered":"2024-09-21T18:13:10.597852+0000","last_clean":"2024-09-21T18:13:10.597852+0000","last_became_active":"2024-09-21T18:11:49.948510+0000","last_became_peered":"2024-09-21T18:11:49.948510+0000","last_unstale":"2024-09-21T18:13:10.597852+0000","last_undegraded":"2024-09-21T18:13:10.597852+0000","last_fullsized":"2024-09-21T18:13:10.597852+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":25,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T18:12:33.939244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T18:12:33.939244+0000","last_clean_scrub_stamp":"2024-09-21T18:12:33.939244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T18:13:59.178929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.20102272199999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":4,"stat_sum":{"num_bytes":1028240208,"num_objects":471,"num_object_clones":0,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":283,"num_whiteouts":126,"num_read":6001,"num_read_kb":4558348,"num_write":3921,"num_write_kb":1538362,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":137481358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":3,"num_bytes_hit_set_archive":878,"num_flush":351,"num_flush_kb":829301,"num_evict":0,"num_evict_kb":0,"num_promote":580,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1181863936,"data_stored":1227723083,"data_compressed":44472695,"data_compressed_allocated":47341568,"data_compressed_original":96842266,"omap_allocated":0,"internal_metadata":0},"log_size":4549,"ondisk_log_size":4549,"up":8,"acting":8,"num_store_stats":7},{"poolid":3,"num_pg":4,"stat_sum":{"num_bytes":1153632024,"num_objects":409,"num_object_clones":0,"num_object_copies":818,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":409,"num_whiteouts":0,"num_read":2723,"num_read_kb":876176,"num_write":2816,"num_write_kb":1851182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":118251159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1254223872,"data_stored":1313499365,"data_compressed":56773892,"data_compressed_allocated":60456960,"data_compressed_original":123271142,"omap_allocated":0,"internal_metadata":0},"log_size":2822,"ondisk_log_size":2822,"up":8,"acting":8,"num_store_stats":9},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":57,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":13},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":145,"num_write_kb":2672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":65574,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":114,"ondisk_log_size":114,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":15,"up_from":23,"seq":98784247891,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":323308,"kb_used_data":283800,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94048532,"statfs":{"total":96636764160,"available":96305696768,"internally_reserved":0,"allocated":290611200,"data_stored":301148431,"data_compressed":10270986,"data_compressed_allocated":10932224,"data_compressed_original":22663064,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,4,5,6,9,10,11,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":13,"up_from":23,"seq":98784247889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31860,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339980,"statfs":{"total":96636764160,"available":96604139520,"internally_reserved":0,"allocated":688128,"data_stored":448042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":23,"seq":98784247890,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182672,"kb_used_data":151484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189168,"statfs":{"total":96636764160,"available":96449708032,"internally_reserved":0,"allocated":155119616,"data_stored":162756702,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,4,5,6,7,8,9,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":21,"seq":90194313299,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":350552,"kb_used_data":311044,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94021288,"statfs":{"total":96636764160,"available":96277798912,"internally_reserved":0,"allocated":318509056,"data_stored":331191567,"data_compressed":13084116,"data_compressed_allocated":13897728,"data_compressed_original":27867778,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,3,5,6,7,10,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":21,"seq":90194313300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31856,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339984,"statfs":{"total":96636764160,"available":96604143616,"internally_reserved":0,"allocated":684032,"data_stored":447727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,5,7,8,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":22,"seq":94489280595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":448473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,4,5,6,8,9,11,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":22,"seq":94489280593,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":338204,"kb_used_data":298696,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94033636,"statfs":{"total":96636764160,"available":96290443264,"internally_reserved":0,"allocated":305864704,"data_stored":319097747,"data_compressed":12087879,"data_compressed_allocated":12894208,"data_compressed_original":27209338,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,3,5,8,9,10,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":22,"seq":94489280594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27700,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344140,"statfs":{"total":96636764160,"available":96608399360,"internally_reserved":0,"allocated":688128,"data_stored":448129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,4,5,6,8,9,10,11,12],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":22,"seq":94489280595,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":327684,"kb_used_data":292336,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94044156,"statfs":{"total":96636764160,"available":96301215744,"internally_reserved":0,"allocated":299352064,"data_stored":314008964,"data_compressed":13601324,"data_compressed_allocated":14524416,"data_compressed_original":30174316,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[2,4,6,7,9,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":142,"seq":609885356055,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28656,"kb_used_data":668,"kb_used_omap":4,"kb_used_meta":27963,"kb_avail":94343184,"statfs":{"total":96636764160,"available":96607420416,"internally_reserved":0,"allocated":684032,"data_stored":447877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4427,"internal_metadata":28634805},"hb_peers":[0,6,8,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":22,"seq":94489280595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344136,"statfs":{"total":96636764160,"available":96608395264,"internally_reserved":0,"allocated":692224,"data_stored":448510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[3,6,7,9,10,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":22,"seq":94489280593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":182688,"kb_used_data":151500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94189152,"statfs":{"total":96636764160,"available":96449691648,"internally_reserved":0,"allocated":155136000,"data_stored":162757920,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":1599,"internal_metadata":31914433},"hb_peers":[3,4,6,7,8,10,11,12,13,14],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":22,"seq":94489280593,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313604,"kb_used_data":286576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94058236,"statfs":{"total":96636764160,"available":96315633664,"internally_reserved":0,"allocated":293453824,"data_stored":303370348,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,3,5,8,9,11,12,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":11,"up_from":22,"seq":94489280594,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":482344,"kb_used_data":442836,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93889496,"statfs":{"total":96636764160,"available":96142843904,"internally_reserved":0,"allocated":453464064,"data_stored":474497240,"data_compressed":19343601,"data_compressed_allocated":20647936,"data_compressed_original":43084518,"omap_allocated":1596,"internal_metadata":40434116},"hb_peers":[2,4,5,6,8,9,10,12,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":12,"up_from":22,"seq":94489280595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27692,"kb_used_data":664,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344148,"statfs":{"total":96636764160,"available":96608407552,"internally_reserved":0,"allocated":679936,"data_stored":445805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[2,5,6,7,8,10,11,13,14,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":14,"up_from":22,"seq":94489280595,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191924,"kb_used_data":160736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94179916,"statfs":{"total":96636764160,"available":96440233984,"internally_reserved":0,"allocated":164593664,"data_stored":170806133,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[3,4,5,6,8,9,11,12,13,15],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":32787,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":12,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":13,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880640,"data_stored":875631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":154431488,"data_stored":162308531,"data_compressed":7838921,"data_compressed_allocated":8335360,"data_compressed_original":16625664,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":157265920,"data_stored":165482081,"data_compressed":7600488,"data_compressed_allocated":8122368,"data_compressed_original":16723968,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":163901440,"data_stored":170359152,"data_compressed":6601935,"data_compressed_allocated":7000064,"data_compressed_original":13986803,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":151072768,"data_stored":158162103,"data_compressed":6345602,"data_compressed_allocated":6770688,"data_compressed_original":14299136,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":154103808,"data_stored":160487701,"data_compressed":5742277,"data_compressed_allocated":6123520,"data_compressed_original":12910202,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":153923584,"data_stored":160384580,"data_compressed":6482181,"data_compressed_allocated":6897664,"data_compressed_original":13880975,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":141344768,"data_stored":147550467,"data_compressed":5968049,"data_compressed_allocated":6361088,"data_compressed_original":12938348,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":292777984,"data_stored":302923186,"data_compressed":10407565,"data_compressed_allocated":11059200,"data_compressed_original":22244903,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":295448576,"data_stored":308038168,"data_compressed":11710326,"data_compressed_allocated":12484608,"data_compressed_original":25848550,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":14,"total":0,"available":0,"internally_reserved":0,"allocated":5410816,"data_stored":5800375,"data_compressed":236913,"data_compressed_allocated":253952,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":15,"total":0,"available":0,"internally_reserved":0,"allocated":138854400,"data_stored":142538606,"data_compressed":3925384,"data_compressed_allocated":4161536,"data_compressed_original":8363928,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T18:13:17.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T18:13:17.812 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T18:13:17.812 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T18:13:17.813 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T18:13:17.813 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:17.908 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T18:13:17.909 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-21T18:13:17.909 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:18.916 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-09-21T18:13:18.914+0000 7f3ce267c640 -1 osd.6 195 handle_osd_map injecting map crc failure 2024-09-21T18:13:20.056 INFO:tasks.ceph.osd.8.smithi029.stderr:2024-09-21T18:13:20.051+0000 7fb924fb6640 -1 osd.8 196 handle_osd_map injecting map crc failure 2024-09-21T18:13:24.011 INFO:tasks.ceph.osd.4:Stopped 2024-09-21T18:13:24.011 DEBUG:tasks.ceph.osd.8:waiting for process to exit 2024-09-21T18:13:24.011 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:26.327 INFO:tasks.ceph.osd.5.smithi040.stderr:2024-09-21T18:13:26.326+0000 7f4976023640 -1 osd.5 199 handle_osd_map injecting map crc failure 2024-09-21T18:13:26.735 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 is failed for ~0s 2024-09-21T18:13:29.963 INFO:tasks.ceph.osd.10.smithi070.stderr:2024-09-21T18:13:29.962+0000 7f9bb588f640 -1 osd.10 200 handle_osd_map injecting map crc failure 2024-09-21T18:13:30.113 INFO:tasks.ceph.osd.8:Stopped 2024-09-21T18:13:30.114 DEBUG:tasks.ceph.osd.12:waiting for process to exit 2024-09-21T18:13:30.114 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.180 INFO:tasks.ceph.osd.12:Stopped 2024-09-21T18:13:30.180 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T18:13:30.180 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.263 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T18:13:30.263 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T18:13:30.264 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.363 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T18:13:30.363 DEBUG:tasks.ceph.osd.9:waiting for process to exit 2024-09-21T18:13:30.363 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.462 INFO:tasks.ceph.osd.9:Stopped 2024-09-21T18:13:30.462 DEBUG:tasks.ceph.osd.13:waiting for process to exit 2024-09-21T18:13:30.463 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.536 INFO:tasks.ceph.osd.13:Stopped 2024-09-21T18:13:30.537 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T18:13:30.537 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.601 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T18:13:30.601 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-21T18:13:30.601 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.679 INFO:tasks.ceph.osd.6:Stopped 2024-09-21T18:13:30.679 DEBUG:tasks.ceph.osd.10:waiting for process to exit 2024-09-21T18:13:30.679 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.750 INFO:tasks.ceph.osd.10:Stopped 2024-09-21T18:13:30.751 DEBUG:tasks.ceph.osd.14:waiting for process to exit 2024-09-21T18:13:30.751 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.830 INFO:tasks.ceph.osd.14:Stopped 2024-09-21T18:13:30.830 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T18:13:30.830 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:30.928 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T18:13:30.928 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-21T18:13:30.928 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:31.013 INFO:tasks.ceph.osd.7:Stopped 2024-09-21T18:13:31.014 DEBUG:tasks.ceph.osd.11:waiting for process to exit 2024-09-21T18:13:31.014 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:32.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 is failed for ~0s 2024-09-21T18:13:32.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.8 has been restored 2024-09-21T18:13:37.115 INFO:tasks.ceph.osd.11:Stopped 2024-09-21T18:13:37.116 DEBUG:tasks.ceph.osd.15:waiting for process to exit 2024-09-21T18:13:37.116 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.186 INFO:tasks.ceph.osd.15:Stopped 2024-09-21T18:13:37.186 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T18:13:37.186 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-21T18:13:37.186 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.215 INFO:tasks.ceph.mgr.y:Stopped 2024-09-21T18:13:37.216 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T18:13:37.216 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.256 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T18:13:37.256 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T18:13:37.256 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T18:13:37.256 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.277 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T18:13:37.277 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T18:13:37.277 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.300 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T18:13:37.300 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T18:13:37.300 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T18:13:37.327 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T18:13:37.327 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T18:13:37.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v 'failed to encode map' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v SLOW_OPS | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | egrep -v 'must scrub before tier agent can activate' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-21T18:13:37.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.11 has been restored 2024-09-21T18:13:37.347 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:13:37.348 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T18:13:37.591 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:13:37.591 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-21T18:13:37.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-8 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:13:37.789 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-8 2024-09-21T18:13:37.980 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-12 on ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:13:37.980 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-12 2024-09-21T18:13:38.172 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:13:38.172 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T18:13:38.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:13:38.309 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-21T18:13:38.500 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-9 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:13:38.500 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-9 2024-09-21T18:13:38.674 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-13 on ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:13:38.675 DEBUG:teuthology.orchestra.run.smithi040:> sync && sudo umount -f /var/lib/ceph/osd/ceph-13 2024-09-21T18:13:38.851 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:13:38.851 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T18:13:39.073 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:13:39.074 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-21T18:13:39.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-10 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:13:39.251 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-10 2024-09-21T18:13:39.405 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-14 on ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:13:39.406 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-14 2024-09-21T18:13:39.536 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:13:39.536 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T18:13:39.800 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:13:39.800 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-21T18:13:40.020 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-11 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:13:40.021 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/ceph-11 2024-09-21T18:13:40.215 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-15 on ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:13:40.215 DEBUG:teuthology.orchestra.run.smithi160:> sync && sudo umount -f /var/lib/ceph/osd/ceph-15 2024-09-21T18:13:40.381 INFO:tasks.ceph:Archiving mon data... 2024-09-21T18:13:40.382 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/data/mon.a.tgz 2024-09-21T18:13:40.382 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-09-21T18:13:40.388 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.7AijvsGffL 2024-09-21T18:13:40.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.7AijvsGffL 2024-09-21T18:13:40.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.7AijvsGffL 2024-09-21T18:13:40.607 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.7AijvsGffL is 478KB 2024-09-21T18:13:40.667 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.7AijvsGffL 2024-09-21T18:13:40.672 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/data/mon.b.tgz 2024-09-21T18:13:40.672 DEBUG:teuthology.orchestra.run.smithi040:> mktemp 2024-09-21T18:13:40.678 INFO:teuthology.orchestra.run.smithi040.stdout:/tmp/tmp.begZTgdYaF 2024-09-21T18:13:40.679 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.begZTgdYaF 2024-09-21T18:13:40.828 DEBUG:teuthology.orchestra.run.smithi040:> sudo chmod 0666 /tmp/tmp.begZTgdYaF 2024-09-21T18:13:40.895 DEBUG:teuthology.orchestra.remote:smithi040:/tmp/tmp.begZTgdYaF is 478KB 2024-09-21T18:13:40.956 DEBUG:teuthology.orchestra.run.smithi040:> rm -fr /tmp/tmp.begZTgdYaF 2024-09-21T18:13:40.961 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/data/mon.c.tgz 2024-09-21T18:13:40.962 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-09-21T18:13:40.967 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.wzCc6P3l3h 2024-09-21T18:13:40.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wzCc6P3l3h 2024-09-21T18:13:41.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.wzCc6P3l3h 2024-09-21T18:13:41.183 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.wzCc6P3l3h is 478KB 2024-09-21T18:13:41.281 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.wzCc6P3l3h 2024-09-21T18:13:41.286 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T18:13:41.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T18:13:41.289 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T18:13:41.291 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T18:13:41.332 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T18:13:41.351 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T18:13:41.357 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T18:13:41.391 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T18:13:41.396 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T18:13:41.397 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T18:13:41.398 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi029/crash 2024-09-21T18:13:41.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T18:13:41.411 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi040/crash 2024-09-21T18:13:41.412 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T18:13:41.427 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi070/crash 2024-09-21T18:13:41.428 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T18:13:41.448 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/lib/ceph/crash to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi160/crash 2024-09-21T18:13:41.449 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T18:13:41.461 INFO:tasks.ceph:Compressing logs... 2024-09-21T18:13:41.461 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T18:13:41.464 DEBUG:teuthology.orchestra.run.smithi040:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T18:13:41.475 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T18:13:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40637.log 2024-09-21T18:13:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41093.log 2024-09-21T18:13:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40924.log 2024-09-21T18:13:41.486 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40637.log.gz 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32440.log 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41093.log.gz 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30828.log 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40924.log.gz 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32440.log.gz 2024-09-21T18:13:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38339.log 2024-09-21T18:13:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41522.log 2024-09-21T18:13:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30828.log: /var/log/ceph/ceph-client.admin.38339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-09-21T18:13:41.489 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30828.log.gz 2024-09-21T18:13:41.489 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38339.log.gz 2024-09-21T18:13:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41522.log.gz 2024-09-21T18:13:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32139.log 2024-09-21T18:13:41.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38049.log 2024-09-21T18:13:41.491 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-09-21T18:13:41.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36225.log 2024-09-21T18:13:41.491 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32139.log.gz 2024-09-21T18:13:41.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38049.log.gz 2024-09-21T18:13:41.492 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31642.log 2024-09-21T18:13:41.492 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43212.log 2024-09-21T18:13:41.492 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36225.log.gz 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36826.log 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31642.log.gz 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38990.log 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43212.log.gz 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33538.log 2024-09-21T18:13:41.494 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36826.log.gz 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31143.log 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38990.log.gz 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31715.log 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33538.log.gz 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31143.log.gz 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40976.log 2024-09-21T18:13:41.495 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31715.log.gz 2024-09-21T18:13:41.496 DEBUG:teuthology.orchestra.run.smithi160:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T18:13:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30391.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40976.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40976.log.gz 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40272.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30391.log.gz 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30431.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30315.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40272.log.gz 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31767.log/var/log/ceph/ceph-client.admin.30431.log: 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30431.log.gz 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33596.log 2024-09-21T18:13:41.501 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30315.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31767.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34737.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32494.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33596.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39663.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34737.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29952.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32494.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41886.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39663.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38957.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29952.log.gz 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29871.log 2024-09-21T18:13:41.502 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41886.log.gz 2024-09-21T18:13:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40898.log 2024-09-21T18:13:41.503 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38957.log.gz 2024-09-21T18:13:41.503 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29819.log 2024-09-21T18:13:41.503 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29871.log.gz 2024-09-21T18:13:41.506 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-21T18:13:41.506 INFO:teuthology.orchestra.run.smithi040.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T18:13:41.506 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T18:13:41.506 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T18:13:41.506 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.9.log 2024-09-21T18:13:41.507 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.13.log 2024-09-21T18:13:41.507 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-21T18:13:41.507 INFO:teuthology.orchestra.run.smithi040.stderr:/var/log/ceph/ceph-osd.13.log: /var/log/ceph/ceph-mon.b.log: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T18:13:41.507 INFO:teuthology.orchestra.run.smithi040.stderr: 89.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35612.log 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40898.log.gz 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29819.log.gz 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32305.log 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35612.log.gz 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33683.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T18:13:41.519 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32305.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21467.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32305.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36354.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.21467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36700.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21467.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41104.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36354.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36700.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36700.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29982.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36915.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41104.log.gz 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/ceph-client.admin.29982.log: --verbose -- /var/log/ceph/ceph-client.admin.35794.log 2024-09-21T18:13:41.520 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29982.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36915.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36915.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37728.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31337.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35794.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37728.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31337.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40246.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40455.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34925.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40246.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39638.log 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40455.log.gz 2024-09-21T18:13:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36489.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34925.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42432.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39638.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36489.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36489.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33513.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43134.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42432.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34476.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33513.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41519.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43134.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35297.log 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34476.log.gz 2024-09-21T18:13:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31851.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41519.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35297.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.35297.log.gz -- 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.34061.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32200.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31851.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42406.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34061.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32200.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31313.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30783.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42406.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34612.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31313.log.gz 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41078.log 2024-09-21T18:13:41.523 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30783.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39339.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34612.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39926.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41078.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39339.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39339.log.gz /var/log/ceph/ceph-client.admin.37956.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29983.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39926.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37552.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37956.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29983.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29975.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29983.log.gz 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43290.log 2024-09-21T18:13:41.524 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37552.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/ceph-client.admin.29975.log: --verbose -- /var/log/ceph/ceph-client.admin.34522.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29975.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43290.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41968.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30338.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34522.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41968.log: -- /var/log/ceph/ceph-client.admin.32841.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41968.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30338.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.42536.log -- replaced with /var/log/ceph/ceph-client.admin.30338.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34642.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32841.log.gz 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32733.log 2024-09-21T18:13:41.525 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-09-21T18:13:41.561 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T18:13:41.561 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-09-21T18:13:41.561 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.14.log 2024-09-21T18:13:41.562 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T18:13:41.562 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.14.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-21T18:13:41.562 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T18:13:41.562 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.10.log 2024-09-21T18:13:41.563 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-osd.10.log: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T18:13:41.564 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30176.log 2024-09-21T18:13:41.564 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26507.log 2024-09-21T18:13:41.564 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30176.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30176.log.gz 2024-09-21T18:13:41.564 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26284.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30399.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26507.log: /var/log/ceph/ceph-client.admin.26284.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-osd.7.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: -- replaced with /var/log/ceph/ceph-client.admin.26507.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26284.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30424.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30399.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30251.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30313.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30424.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30251.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30251.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30313.log.gz 2024-09-21T18:13:41.565 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.30472.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26236.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.0.30472.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.11.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.30472.log.gz 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26236.log.gz 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26411.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-osd.11.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.30449.log: /var/log/ceph/objectstore_tool.31928.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26411.log.gz 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.15.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26533.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/objectstore_tool.31928.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-osd.15.log: /var/log/ceph/ceph-client.admin.26533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30151.log 2024-09-21T18:13:41.566 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26533.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26557.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26332.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30151.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26557.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30780.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26557.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26332.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26356.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30276.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/objectstore_tool.30780.log: /var/log/ceph/ceph-client.admin.26356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26356.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26212.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30276.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26308.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26212.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26212.log.gz 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26435.log 2024-09-21T18:13:41.567 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26308.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26260.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr: 93.3% -- replaced with /var/log/ceph/objectstore_tool.30780.log.gz/var/log/ceph/ceph-client.admin.26435.log: 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26435.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26188.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26387.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26260.log: /var/log/ceph/ceph-client.admin.26188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26260.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26188.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30201.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26387.log: /var/log/ceph/ceph-client.admin.30362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26387.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.31172.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26459.log 2024-09-21T18:13:41.568 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30201.log: /var/log/ceph/objectstore_tool.31172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30201.log.gz 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30226.log 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26459.log.gz 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26483.log 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.30226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30226.log.gz 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30976.log 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/ceph-client.admin.26483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26483.log.gz 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr:/var/log/ceph/objectstore_tool.30976.log: 93.5% -- replaced with /var/log/ceph/objectstore_tool.31928.log.gz 2024-09-21T18:13:41.569 INFO:teuthology.orchestra.run.smithi160.stderr: 93.0% -- replaced with /var/log/ceph/objectstore_tool.30976.log.gz 2024-09-21T18:13:41.570 INFO:teuthology.orchestra.run.smithi070.stderr: 89.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T18:13:41.576 INFO:teuthology.orchestra.run.smithi160.stderr: 93.7% -- replaced with /var/log/ceph/objectstore_tool.31172.log.gz 2024-09-21T18:13:41.622 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34642.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.34642.log.gz -- 2024-09-21T18:13:41.622 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.42302.log 2024-09-21T18:13:41.622 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30831.log 2024-09-21T18:13:41.622 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32733.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42302.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.8.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42302.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30831.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.37022.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30831.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30479.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42770.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37022.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30479.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30479.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41970.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42770.log: /var/log/ceph/ceph-client.admin.35669.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42770.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41970.log.gz 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32654.log 2024-09-21T18:13:41.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26254.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35669.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32654.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33382.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.26254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38417.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26254.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33382.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33033.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.38417.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32140.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38417.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33033.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35524.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32140.log: /var/log/ceph/ceph-client.admin.36252.log 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32140.log.gz 2024-09-21T18:13:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35524.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29845.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.36252.log: -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36252.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29845.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43108.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42006.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.43108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43108.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42006.log: /var/log/ceph/ceph-client.admin.41982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42006.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39554.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41002.log 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41109.log: /var/log/ceph/ceph-client.admin.39554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz 2024-09-21T18:13:41.625 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39554.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30759.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37467.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41002.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30759.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35588.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30355.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37467.log: /var/log/ceph/ceph-client.admin.35588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37467.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35588.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36785.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30355.log: /var/log/ceph/ceph-client.admin.36785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30355.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33188.log 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36785.log.gz 2024-09-21T18:13:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42562.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33188.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39163.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42562.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31217.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39163.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32591.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31217.log: /var/log/ceph/ceph-client.admin.42510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31217.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34557.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.32591.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38708.log 2024-09-21T18:13:41.627 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32591.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29976.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34557.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38708.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29976.log: /var/log/ceph/ceph-client.admin.41495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29976.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30755.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39517.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30372.log 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30755.log.gz 2024-09-21T18:13:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37396.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36991.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30372.log: /var/log/ceph/ceph-client.admin.37396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30372.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37396.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30892.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40741.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36991.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30892.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40741.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36088.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29955.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36088.log.gz 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42978.log 2024-09-21T18:13:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29955.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34159.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35323.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42978.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34159.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.34119.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34159.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35323.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39952.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/ceph-client.admin.34119.log: --verbose -- /var/log/ceph/ceph-client.admin.33810.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34119.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39952.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41992.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39952.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33810.log.gz 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42380.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40715.log 2024-09-21T18:13:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41992.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42380.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41890.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40715.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.41890.log: -- /var/log/ceph/ceph-client.admin.40611.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41890.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41462.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40611.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42952.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31013.log 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41462.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41462.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42952.log.gz 2024-09-21T18:13:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35853.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31013.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34327.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43082.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35853.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34327.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34327.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30890.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43082.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43392.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37342.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30890.log.gz 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43392.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43392.log.gz -5 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37096.log 2024-09-21T18:13:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37342.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38607.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42900.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37096.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38607.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38662.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42900.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31513.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38662.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31513.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31513.log.gz -5 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35167.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37492.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37758.log 2024-09-21T18:13:41.633 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35167.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37492.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39900.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38395.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37758.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39900.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41948.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40663.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38395.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33271.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41948.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40663.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40006.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31265.log 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33271.log: /var/log/ceph/ceph-client.admin.40006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33271.log.gz 2024-09-21T18:13:41.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41988.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40006.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31265.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.31265.log.gz -- 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.37548.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33312.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41988.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37548.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33312.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34822.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31241.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29897.log 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34822.log.gz 2024-09-21T18:13:41.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31241.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31471.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29897.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31471.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31471.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38897.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40194.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41457.log: /var/log/ceph/ceph-client.admin.38897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38897.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29942.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36726.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40194.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42614.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29942.log.gz 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34847.log 2024-09-21T18:13:41.636 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36726.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.42614.log: /var/log/ceph/ceph-client.admin.40533.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42614.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34847.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31986.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31193.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40533.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40533.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38314.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31986.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29973.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31193.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31193.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38314.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38023.log 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29973.log.gz 2024-09-21T18:13:41.637 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42640.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42328.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38023.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40429.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42640.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42328.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42328.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40351.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36435.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40429.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40429.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32974.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40351.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35113.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36435.log.gz 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30891.log 2024-09-21T18:13:41.638 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32974.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31506.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35113.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41487.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30891.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26373.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31506.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40032.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41487.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.26373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33893.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26373.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40032.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40032.log.gz 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.39848.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2024-09-21T18:13:41.639 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33893.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39848.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38817.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38104.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38817.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.38817.log.gz -- 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.31045.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41490.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38104.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40220.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31045.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33296.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41490.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40220.log: /var/log/ceph/ceph-client.admin.32650.log 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40220.log.gz 2024-09-21T18:13:41.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29964.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33296.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42848.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32650.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38791.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29964.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41510.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42848.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38791.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34927.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41510.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30369.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32471.log 2024-09-21T18:13:41.641 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34927.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36524.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30369.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32471.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.32015.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32471.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31169.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36524.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32015.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32015.log.gz -5 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40299.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32874.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31169.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40299.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40299.log.gz 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.39082.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-09-21T18:13:41.642 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32874.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39082.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33867.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36358.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33867.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43056.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36358.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36358.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31855.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43056.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41080.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26493.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31855.log.gz 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31441.log 2024-09-21T18:13:41.643 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41080.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.26493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30857.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26493.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35378.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31441.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34214.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30857.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39218.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35378.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33247.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34214.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29960.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39218.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29963.log 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33247.log.gz 2024-09-21T18:13:41.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29960.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29963.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43368.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39373.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31363.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43368.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39373.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40950.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39373.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31363.log: 89.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32466.log 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31363.log.gz 2024-09-21T18:13:41.645 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40872.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40950.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41110.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32466.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31825.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40872.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34408.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41110.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34239.log 0.0% 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.31825.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41028.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34408.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35879.log 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34239.log.gz 2024-09-21T18:13:41.646 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32198.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41028.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31041.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35879.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43160.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32198.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32949.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31041.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43186.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43160.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39428.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32949.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30393.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43186.log.gz 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.39428.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-09-21T18:13:41.647 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39428.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42666.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30393.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35483.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32085.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37657.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42666.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41996.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32085.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39719.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37657.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41996.log.gz 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42354.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29980.log 2024-09-21T18:13:41.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39719.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32355.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42354.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43316.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29980.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32355.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42012.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43316.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43342.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42012.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43342.log.gz 2024-09-21T18:13:41.649 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30456.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.41523.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39113.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31289.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30456.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33722.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39113.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31289.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30356.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31289.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33722.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33722.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31761.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30356.log.gz 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29974.log 2024-09-21T18:13:41.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: /var/log/ceph/ceph-client.admin.31761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41103.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31761.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41485.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29974.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41103.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42016.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41103.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42796.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41485.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42016.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.42016.log.gz -5 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32309.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32760.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42796.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32309.log: gzip 89.0% -5 --verbose -- replaced with /var/log/ceph/ceph.audit.log.gz -- 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.40137.log 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32309.log.gz 2024-09-21T18:13:41.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32760.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37839.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30387.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39453.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37839.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/ceph-client.admin.30387.log: --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30387.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41524.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39453.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30451.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34766.log 2024-09-21T18:13:41.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41524.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41524.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40325.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30451.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30343.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34766.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43030.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40325.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30343.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43030.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43238.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.12.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33058.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41083.log 2024-09-21T18:13:41.653 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43238.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.12.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35960.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41083.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40793.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42588.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35960.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36169.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40793.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37163.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42588.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30362.log 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36169.log.gz 2024-09-21T18:13:41.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37163.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38562.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30362.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30362.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43264.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40767.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38562.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36603.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43264.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42011.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40767.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40377.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36603.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42011.log.gz 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43004.log 2024-09-21T18:13:41.655 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40377.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32408.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.43004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33976.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43004.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40559.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32408.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42007.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33976.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37813.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40559.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29768.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42007.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31389.log 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37813.log.gz 2024-09-21T18:13:41.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29768.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31040.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31389.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31040.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42458.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31515.log 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42458.log.gz 2024-09-21T18:13:41.657 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31117.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37894.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31515.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29951.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31117.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33620.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37894.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29951.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38129.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33620.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39978.log 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-09-21T18:13:41.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36047.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38129.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.39978.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39978.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40689.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36047.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36645.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30222.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40689.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42692.log 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36645.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.30222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30222.log.gz 2024-09-21T18:13:41.659 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-09-21T18:13:41.660 INFO:teuthology.orchestra.run.smithi160.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T18:13:41.672 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42822.log 2024-09-21T18:13:41.672 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42692.log.gz 2024-09-21T18:13:41.672 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41114.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz -5 2024-09-21T18:13:41.672 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29950.log 2024-09-21T18:13:41.672 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29793.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42822.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29950.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37283.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29793.log: /var/log/ceph/ceph-client.admin.41581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32279.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29793.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37283.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41100.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.32279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34299.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32279.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31938.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41100.log.gz 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42874.log 2024-09-21T18:13:41.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34299.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38872.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31938.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38223.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42874.log: /var/log/ceph/ceph-client.admin.38872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37201.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42874.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38872.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.38223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38223.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40585.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42926.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.37201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37201.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42484.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40585.log.gz 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40058.log 2024-09-21T18:13:41.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.42926.log: /var/log/ceph/ceph-client.admin.42484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29962.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42926.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42484.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40058.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.40058.log.gz -- 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.31934.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41497.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29962.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31934.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42718.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31934.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35696.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41497.log: /var/log/ceph/ceph-client.admin.42718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41894.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41497.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42718.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35696.log.gz 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31415.log 2024-09-21T18:13:41.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35032.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.41894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41894.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33950.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31415.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40111.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35032.log: /var/log/ceph/ceph-client.admin.33950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36144.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35032.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33950.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.40111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34442.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40111.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35934.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36144.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33415.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34442.log.gz 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34089.log 2024-09-21T18:13:41.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.35934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35934.log.gz 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.33415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31688.log 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33415.log.gz 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36910.log 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.34089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34089.log.gz 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.31688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29743.log 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31688.log.gz 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.36910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36910.log.gz 2024-09-21T18:13:41.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.29743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29743.log.gz 2024-09-21T18:13:41.721 INFO:teuthology.orchestra.run.smithi160.stderr: 90.9% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-09-21T18:13:41.757 INFO:teuthology.orchestra.run.smithi029.stderr: 90.3% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T18:13:41.817 INFO:teuthology.orchestra.run.smithi070.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-21T18:13:41.843 INFO:teuthology.orchestra.run.smithi040.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-21T18:13:41.879 INFO:teuthology.orchestra.run.smithi029.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.8.log.gz 2024-09-21T18:13:41.883 INFO:teuthology.orchestra.run.smithi029.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.12.log.gz 2024-09-21T18:13:41.896 INFO:teuthology.orchestra.run.smithi160.stderr: 90.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-21T18:13:42.119 INFO:teuthology.orchestra.run.smithi029.stderr: 90.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T18:13:42.127 INFO:teuthology.orchestra.run.smithi040.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.9.log.gz 2024-09-21T18:13:42.221 INFO:teuthology.orchestra.run.smithi040.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.13.log.gz 2024-09-21T18:13:42.274 INFO:teuthology.orchestra.run.smithi070.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-09-21T18:13:42.363 INFO:teuthology.orchestra.run.smithi070.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.14.log.gz 2024-09-21T18:13:42.782 INFO:teuthology.orchestra.run.smithi040.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T18:13:42.792 INFO:teuthology.orchestra.run.smithi040.stderr: 91.9% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-21T18:13:42.794 INFO:teuthology.orchestra.run.smithi040.stderr: 2024-09-21T18:13:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:real 0m1.317s 2024-09-21T18:13:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:user 0m4.182s 2024-09-21T18:13:42.794 INFO:teuthology.orchestra.run.smithi040.stderr:sys 0m0.114s 2024-09-21T18:13:43.248 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T18:13:43.584 INFO:teuthology.orchestra.run.smithi070.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.10.log.gz 2024-09-21T18:13:43.585 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-09-21T18:13:43.585 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m2.090s 2024-09-21T18:13:43.585 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m5.310s 2024-09-21T18:13:43.585 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.222s 2024-09-21T18:13:44.385 INFO:teuthology.orchestra.run.smithi160.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.15.log.gz 2024-09-21T18:13:44.701 INFO:teuthology.orchestra.run.smithi029.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-21T18:13:44.702 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-09-21T18:13:44.702 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m3.237s 2024-09-21T18:13:44.702 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m4.756s 2024-09-21T18:13:44.702 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.223s 2024-09-21T18:13:45.016 INFO:teuthology.orchestra.run.smithi160.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.11.log.gz 2024-09-21T18:13:45.017 INFO:teuthology.orchestra.run.smithi160.stderr: 2024-09-21T18:13:45.017 INFO:teuthology.orchestra.run.smithi160.stderr:real 0m3.505s 2024-09-21T18:13:45.017 INFO:teuthology.orchestra.run.smithi160.stderr:user 0m6.793s 2024-09-21T18:13:45.017 INFO:teuthology.orchestra.run.smithi160.stderr:sys 0m0.279s 2024-09-21T18:13:45.018 INFO:tasks.ceph:Archiving logs... 2024-09-21T18:13:45.019 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi029/log 2024-09-21T18:13:45.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T18:13:45.941 DEBUG:teuthology.misc:Transferring archived files from smithi040:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi040/log 2024-09-21T18:13:45.942 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T18:13:46.448 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi070/log 2024-09-21T18:13:46.449 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T18:13:47.149 DEBUG:teuthology.misc:Transferring archived files from smithi160:/var/log/ceph to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi160/log 2024-09-21T18:13:47.150 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T18:13:48.106 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T18:13:48.115 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T18:13:48.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T18:13:48.118 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T18:13:48.120 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T18:13:48.123 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T18:13:48.236 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T18:13:48.245 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T18:13:48.245 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:13:48.248 DEBUG:teuthology.orchestra.run.smithi040:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:13:48.250 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:13:48.252 DEBUG:teuthology.orchestra.run.smithi160:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================== 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout:+hv01.front.sepi 67.205.162.81 3 u 57 64 377 0.071 -0.077 1.065 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout:+hv02.front.sepi 63.231.80.2 3 u 46 64 377 0.049 -1.250 1.001 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout:*hv03.front.sepi 74.6.168.72 3 u 60 64 377 0.103 -0.708 0.601 2024-09-21T18:13:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout:============================================================================== 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout:*hv01.front.sepi 67.205.162.81 3 u 51 64 377 0.091 -0.847 0.476 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout:+hv02.front.sepi 63.231.80.2 3 u 58 64 377 0.106 -1.554 0.477 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout:+hv03.front.sepi 74.6.168.72 3 u 55 64 377 0.070 -0.541 0.439 2024-09-21T18:13:48.262 INFO:teuthology.orchestra.run.smithi040.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-21T18:13:48.263 INFO:teuthology.orchestra.run.smithi070.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:13:48.263 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================== 2024-09-21T18:13:48.263 INFO:teuthology.orchestra.run.smithi070.stdout:+hv01.front.sepi 67.205.162.81 3 u 51 64 377 0.113 -0.880 1.057 2024-09-21T18:13:48.263 INFO:teuthology.orchestra.run.smithi070.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.106 -4.095 1.654 2024-09-21T18:13:48.264 INFO:teuthology.orchestra.run.smithi070.stdout:*hv03.front.sepi 74.6.168.72 3 u 56 64 377 0.078 -0.687 1.024 2024-09-21T18:13:48.264 INFO:teuthology.orchestra.run.smithi070.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout:============================================================================== 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout:*hv01.front.sepi 67.205.162.81 3 u 58 64 377 0.071 -0.439 0.564 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout:+hv02.front.sepi 63.231.80.2 3 u 60 64 377 0.077 -1.216 0.731 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout:+hv03.front.sepi 74.6.168.72 3 u 61 64 377 0.095 +0.552 1.054 2024-09-21T18:13:48.267 INFO:teuthology.orchestra.run.smithi160.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-21T18:13:48.268 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T18:13:48.277 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T18:13:48.278 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T18:13:48.298 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T18:13:48.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T18:13:48.356 INFO:teuthology.task.internal:Duration was 1122.800492 seconds 2024-09-21T18:13:48.356 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T18:13:48.391 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T18:13:48.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T18:13:48.393 DEBUG:teuthology.orchestra.run.smithi040:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T18:13:48.396 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T18:13:48.398 DEBUG:teuthology.orchestra.run.smithi160:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T18:13:48.436 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T18:13:48.437 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-09-21T18:13:48.437 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T18:13:48.487 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi040.front.sepia.ceph.com 2024-09-21T18:13:48.487 DEBUG:teuthology.orchestra.run.smithi040:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T18:13:48.498 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-09-21T18:13:48.498 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T18:13:48.508 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi160.front.sepia.ceph.com 2024-09-21T18:13:48.508 DEBUG:teuthology.orchestra.run.smithi160:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T18:13:48.518 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T18:13:48.518 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T18:13:48.530 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T18:13:48.543 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T18:13:48.552 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T18:13:48.591 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T18:13:48.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T18:13:48.602 DEBUG:teuthology.orchestra.run.smithi040:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T18:13:48.615 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T18:13:48.623 DEBUG:teuthology.orchestra.run.smithi160:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T18:13:48.700 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T18:13:48.711 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T18:13:48.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T18:13:48.722 DEBUG:teuthology.orchestra.run.smithi040:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T18:13:48.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T18:13:48.749 DEBUG:teuthology.orchestra.run.smithi160:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T18:13:48.761 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T18:13:48.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.774 DEBUG:teuthology.orchestra.run.smithi040:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.779 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.784 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-09-21T18:13:48.788 INFO:teuthology.orchestra.run.smithi040.stdout:kernel.core_pattern = core 2024-09-21T18:13:48.792 DEBUG:teuthology.orchestra.run.smithi160:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.800 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-09-21T18:13:48.814 INFO:teuthology.orchestra.run.smithi160.stdout:kernel.core_pattern = core 2024-09-21T18:13:48.897 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:13:48.903 DEBUG:teuthology.orchestra.run.smithi040:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:13:48.908 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:13:48.913 DEBUG:teuthology.orchestra.run.smithi160:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T18:13:48.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T18:13:48.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T18:13:48.955 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T18:13:48.955 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi029 2024-09-21T18:13:48.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T18:13:48.978 DEBUG:teuthology.misc:Transferring archived files from smithi040:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi040 2024-09-21T18:13:48.978 DEBUG:teuthology.orchestra.run.smithi040:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T18:13:49.000 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi070 2024-09-21T18:13:49.000 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T18:13:49.021 DEBUG:teuthology.misc:Transferring archived files from smithi160:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rfriedma-2024-09-21_17:35:20-rados-wip-rf-qprio-w-qa-distro-default-smithi/7914219/remote/smithi160 2024-09-21T18:13:49.021 DEBUG:teuthology.orchestra.run.smithi160:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T18:13:49.047 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T18:13:49.047 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T18:13:49.050 DEBUG:teuthology.orchestra.run.smithi040:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T18:13:49.053 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T18:13:49.059 DEBUG:teuthology.orchestra.run.smithi160:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T18:13:49.089 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T18:13:49.101 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T18:13:49.101 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T18:13:49.116 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T18:13:49.116 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T18:13:49.119 DEBUG:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T18:13:49.121 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T18:13:49.122 INFO:teuthology.orchestra.run.smithi029.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 18:13 /home/ubuntu/cephtest 2024-09-21T18:13:49.123 DEBUG:teuthology.orchestra.run.smithi160:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T18:13:49.124 INFO:teuthology.orchestra.run.smithi040.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 18:13 /home/ubuntu/cephtest 2024-09-21T18:13:49.126 INFO:teuthology.orchestra.run.smithi070.stdout: 658633 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 18:13 /home/ubuntu/cephtest 2024-09-21T18:13:49.137 INFO:teuthology.orchestra.run.smithi160.stdout: 655429 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 18:13 /home/ubuntu/cephtest 2024-09-21T18:13:49.139 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T18:13:49.154 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T18:13:49.221 INFO:teuthology.run:Summary data: description: rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/mapgap_host thrashosds-health workloads/cache} duration: 1122.8004915714264 flavor: default owner: scheduled_rfriedma@teuthology success: true 2024-09-21T18:13:49.221 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T18:13:49.307 INFO:teuthology.run:pass